truf-server/docs/worker-operator-experience-live-trace-2026-09-25.md

Worker Operator Experience Live Trace - 2026-09-25

Result

The accepted Windows package and Linux image completed a fresh, concurrent live cohort against the sec runtime. All 295 assignments were acknowledged, ingested, projected, and settled. There were no unresolved assignments, pre-commit bundles, expiries, pre-bundle failures, quarantine rows, append failures, or publication-outbox rows at the final cut.

The worker transport and server pipeline acceptance result is pass. Four product defects and two operational warnings were found. The defects did not invalidate the one-authoritative-acceptance, ingestion, projection, recovery, or shutdown guarantees demonstrated by this cohort, but they remain release inputs and are listed below.

This report is sanitized. It intentionally excludes authentication values, private routes, worker command lines, raw targets, raw findings, secrets, and runtime configuration bodies. The protected evidence files referenced below contain sensitive material and must not be published.

Validated artifacts

The run used the previously accepted reproducible artifacts without modifying their product code:

Platform	Accepted identity
Windows x86-64	package manifest 78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6
Linux x86-64	package identity 45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42
Linux image	sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3

Both trusted manifests remained registered on the server. Each validation worker ran one slot with local parallelism 1. The measured combined concurrency reached exactly 2, proving concurrent accepted Windows and Linux execution without increasing either worker's local parallelism.

Final cohort

Platform and source distribution

Worker	DockerHub	GitLab	Hugging Face	Total
Windows	58	57	51	166
Linux	50	30	49	129
Total	108	87	100	295

Every history row ended with bundle_accepted and mapped to exactly one server reservation and receipt. The Windows and Linux reservation sets were disjoint and their union exactly matched the 295-row server cohort.

Scan and queue outcomes

Scan outcome	Count
clean	143
degraded	73
error	75
found	4

Queue disposition	Count
done	220
deferred	74
failed	1

These are scanner/queue outcomes, not transport failures. All corresponding result bundles were accepted and projected. In particular, timeout and scanner error results remained normal uploadable terminal results.

Persisted detail

The stable capture contains:

• 295 admission intents, reservations, bundles, target scans, compatibility rows, and completed scan projection jobs;
• 2,905 ordered progress events;
• 75 structured diagnostics;
• 129 normalized scan errors;
• 4 normalized findings, 4 stable UID mappings, and 4 bounded compatibility payloads;
• 4 pending keycheck candidates linked to 2 normalized credentials;
• 374 appended projection records across 3 streams;
• 956 pipeline artifacts, all deleted by the final snapshot; and
• 19 successful typed runtime operations with 38 chained audit events.

The diagnostic aggregate was:

Worker	Scanner result errors	Stage timeouts	Total diagnostics
Windows	55	8	63
Linux	1	11	12

Of the Windows scanner-result errors, 54 were retryable and one was non-retryable. The Linux scanner-result error was retryable. Complete safe diagnostic envelopes, exception identities, bounded process-log representations, occurrence times, receipt authority, and scan links were retained and checked.

End-to-end integrity checks

build/operator-experience-validation/analyze-live-trace-final-20260925.py executed 65,675 checks with zero failures. It verified, row by row:

• admission, reservation, queue, bundle, scan, compatibility, and projection foreign-key relationships;
• receipt, payload, bundle, event, device, and deadline identities;
• canonical SHA-256 values for execution snapshots, progress events, diagnostics, compatibility metadata, plans, projection events, and finding payloads;
• exact bounded reconstruction of the four compatibility findings, including their original numeric detector identity and explicit null mapped fields;
• every normalized error, finding, keycheck candidate, credential reference, projection append, capacity release, and deleted artifact;
• all 19 operation-to-audit pairs; and
• the complete 38-event audit parent/hash chain.

build/operator-experience-validation/analyze-worker-states-final-20260925.py executed a further 28,686 checks with zero failures. It parsed every retained worker JSON/JSONL record and verified:

• 166 Windows and 129 Linux history rows against the server receipts;
• 2,338 contiguous Windows events and 1,672 contiguous Linux events;
• receipt payload, bundle, event, acceptance-time, and reservation identities;
• clean local shutdown with drained=true, exit_code=0, and empty progress outboxes; and
• no active work root in either final worker snapshot.

The two analyzers therefore executed 94,361 deterministic checks without a failure.

Recovery and shutdown

The validation exercised durable recovery rather than only clean executions:

• transient server 502 responses were retained in both local worker logs and recovered without duplicate authoritative acceptance;
• one Linux assignment survived a worker stop in the persistent volume, resumed after restart, produced one accepted result, and released all capacity;
• a transient assignment-status network failure retried the same durable assignment without rescanning or data loss; and
• both workers then drained and stopped cleanly.

The final local states were:

Worker	State	Drained	Exit	Pending outbox
Windows	stopped	true	0	0
Linux container	exited	true	0	0

Retained work/abandoned roots are inactive evidence governed by normal worker retention. They are not active assignments.

Worker API validation

Authenticated worker API validation covered:

• device identity, package-manifest trust, and assignment-cap enforcement;
• claim, reservation replay, assignment status, and immutable execution snapshot;
• monotonic progress submission and latest-progress readback;
• bounded diagnostic body and process-log payloads;
• durable bundle upload, idempotent receipt replay, and accepted resolution;
• local restart recovery and terminal history;
• server ingestion, normalized scan authority, compatibility reconstruction, and projection completion; and
• terminal capacity release and zero unresolved work.

The API preserved receipt, payload, scan-event, diagnostic, and reservation identities throughout the cohort. A separate terminal readback defect affecting only scan_deadline_at is documented below.

Admin UI and operator workflow

The authenticated admin UI was exercised through a browser across the complete operator surface:

• Workers / Dispatch: control state, users, devices, assignment caps, enable, disable, revoke, unrevoke, and bounded worker detail;
• Overview: runtime health, producer lifecycle, pipeline workers, leases, queue counts, capacity, controls, recent operations, and duration groups;
• Search: bounded assignment, scan, finding, error, diagnostic, and progress lookup with safe empty and populated states;
• Supervisor: source start, restart, stop, lifecycle, and safe error rendering;
• Logs: bounded source/component/level/time filters and empty-result handling;
• Config and Secrets: active identities, stale-candidate warning, redacted projections, validation, and non-secret operation results;
• Files: bounded listing, file identity/hash verification, and safe download behavior;
• Operations: accepted/running/succeeded projections and filters; and
• Audit: accepted/succeeded event pairs, pagination, actor/action filters, and parent/hash continuity.

The final UI snapshot showed:

• Supervisor ACTIVE and PostgreSQL READY;
• result ingester, JSONL projector, janitor, and worker API all running;
• ingester and projector leases ready;
• control revision 126, discovery and dispatch open, drain state normal;
• zero active assignments and zero pre-commit bundles; and
• zero quarantined queue rows.

The current DockerHub producer safe state remained runtime_error; it is the known retry-coalescing defect plus unavailable credential pool described below, not an unclassified new failure.

Server and pipeline final state

The final server snapshot at 2026-09-25 14:59 UTC confirmed:

• 295 issued, 295 accepted, 295 ingested, 295 projected, and 295 settled;
• 0 unresolved, expired, pre-bundle-failed, pre-commit, quarantine, and drain blockers;
• bundle, projection, and quarantine capacity at zero;
• keycheck capacity at 137 items / 27,262,866 bytes, representing real pending work rather than leaked assignment or projection capacity;
• runtime container healthy with zero restarts and no OOM;
• edge container running with zero restarts and no OOM; and
• dashboard health endpoint returning 200 ok.

Defects found

1. Windows scanner timestamps lose their UTC offset

Status: open in the accepted Windows package.

Naive local scanner timestamps are relabeled as UTC, producing approximately a three-hour future displacement on the validation host. Progress transport and monotonic durations remain correct, but diagnostic ordering, time filters, and scan start/end instants are wrong.

Detailed evidence and correction: docs/defect-windows-scan-timestamps-utc-2026-09-25.md.

2. DockerHub retry coalescing fails with a null retry time

Status: open in the live runtime; fixed locally but not deployed.

PostgreSQL cannot infer the type of a nullable retry placeholder while coalescing an existing row, raising SQLSTATE 42P18. The managed producer masks that exception as a generic delegation failure. A minimal local correction casts the placeholder to text, and regression coverage now exercises same-row null-time coalescing.

Detailed evidence and local fix: docs/defect-dockerhub-discovery-retry-null-type-2026-09-25.md.

3. Terminal status can lose the durable scan deadline

Status: open in the live runtime.

A later progress event with a null scan deadline can replace the durable receipt's concrete immutable deadline in authenticated terminal status readback. The persisted receipt and all other identities remain correct.

Detailed evidence: docs/defect-terminal-status-scan-deadline-readback-2026-09-25.md.

4. Network OSError is mislabeled as local I/O

Status: open in the accepted workers.

The broad safe-summary branch classifies socket/transport OSError as local I/O operation failed. Recovery worked and no data was lost, but the operator message incorrectly points toward local storage.

Detailed evidence: docs/defect-worker-network-oserror-mislabeled-local-io-2026-09-25.md.

Operational warnings

• The server root filesystem was approximately 90% used with roughly 1 GiB free. Containers remained healthy, but capacity should be reclaimed or expanded.
• The DockerHub credential pool was independently unavailable: ten credentials were invalid and the remaining entry was rate-limited. Deploying the SQL fix preserves retries correctly but cannot make an unavailable credential pool healthy.

Tests and specification gates

The retained gates are:

• worker/operator focused matrix: 355 passed, 3 skipped;
• admin/runtime focused matrix: 124 passed, 2 warnings;
• DockerHub incremental discovery matrix: 27 passed;
• SQLite retry lifecycle regression: 1 passed;
• corrected PostgreSQL statement verified transactionally against the live schema and rolled back; and
• OpenSpec strict validation passed with all 27 implementation tasks complete.

The disposable PostgreSQL integration test remains skipped locally because no disposable DSN was configured. The live transactional SQL verification did not persist a change.

Evidence manifest

Artifact	Bytes	SHA-256
build/live-trace-20260925/raw-evidence-final.json	10,030,750	4071e38a1dec540663bc6febd9538ff54bedafa1627250933045beb8f7d09ec5
build/live-trace-20260925/monitor-final.ndjson	1,260,087	e07507b0b8f0f86d1a1c7aade7186297c372b1ff177067bea19dfd219f5027a9
build/live-trace-20260925/summary-final.json	3,669	19e5ec40cf354c6095a478b68a69f8e51fb3b8ea1c6f278c1d9c90bdaab9ebe2
build/live-trace-20260925/final-analysis-summary.json	946	2d17605ba7b730ad78a48bcc70e40e78f90637e3fd59004ebf5eb4a08241ba42
build/live-trace-20260925/final-worker-state-analysis-summary.json	1,376	b0ab428eb08587f13f59a1763f835125216f769713e259d85989ea8b7f8af95c
build/live-trace-20260925/linux-worker-state-final.tar.gz	134,380	b64e81c90b232f46b400a63ed08f5660f46e34fedb1f67b64afba079d8d36364

The final Windows state is retained under build/live-trace-20260925/windows-localappdata/TRUF/RemoteWorker.

Deliberately retained live state

The user requested that validation state not be restored. The following changes therefore remain deliberate:

• accepted Windows and Linux trusted manifests remain installed;
• the keycheck queue maximum remains increased from 4,096 to 8,192 items;
• the Linux validation user remains enabled at assignment cap 0;
• the Windows validation user remains enabled at assignment cap 1;
• both validation workers themselves are stopped;
• normal global controls remain open at revision 126; and
• the dashboard remains running.

The config editor still contains an intentionally stale candidate based on the pre-validation active hash. It must not be applied without first rebasing it onto the current active configuration.

Release conclusion

The accepted worker artifacts passed live cross-platform execution, concurrent dispatch, bounded progress/diagnostics, durable recovery, one-authoritative receipt handling, normalized ingestion, projection, audit, local shutdown, and operator UI validation. The complete cohort settled without leaked worker, bundle, projection, or quarantine capacity.

Before broad rollout, deploy and revalidate the DockerHub SQL correction, decide release treatment for the Windows timestamp and terminal-deadline defects, fix network error classification, and address server disk pressure and DockerHub credential health. The OpenSpec change is complete but remains unarchived until explicitly requested.