# Worker Operator Experience Live Trace - 2026-09-25

## Result

The accepted Windows package and Linux image completed a fresh, concurrent live
cohort against the `sec` runtime. All 295 assignments were acknowledged,
ingested, projected, and settled. There were no unresolved assignments,
pre-commit bundles, expiries, pre-bundle failures, quarantine rows, append
failures, or publication-outbox rows at the final cut.

The worker transport and server pipeline acceptance result is **pass**. Four
product defects and two operational warnings were found. The defects did not
invalidate the one-authoritative-acceptance, ingestion, projection, recovery, or
shutdown guarantees demonstrated by this cohort, but they remain release inputs
and are listed below.

This report is sanitized. It intentionally excludes authentication values,
private routes, worker command lines, raw targets, raw findings, secrets, and
runtime configuration bodies. The protected evidence files referenced below
contain sensitive material and must not be published.

## Validated artifacts

The run used the previously accepted reproducible artifacts without modifying
their product code:

| Platform | Accepted identity |
| --- | --- |
| Windows x86-64 | package manifest `78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6` |
| Linux x86-64 | package identity `45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42` |
| Linux image | `sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3` |

Both trusted manifests remained registered on the server. Each validation worker
ran one slot with local parallelism `1`. The measured combined concurrency
reached exactly `2`, proving concurrent accepted Windows and Linux execution
without increasing either worker's local parallelism.

## Final cohort

### Platform and source distribution

| Worker | DockerHub | GitLab | Hugging Face | Total |
| --- | ---: | ---: | ---: | ---: |
| Windows | 58 | 57 | 51 | 166 |
| Linux | 50 | 30 | 49 | 129 |
| Total | 108 | 87 | 100 | 295 |

Every history row ended with `bundle_accepted` and mapped to exactly one server
reservation and receipt. The Windows and Linux reservation sets were disjoint and
their union exactly matched the 295-row server cohort.

### Scan and queue outcomes

| Scan outcome | Count |
| --- | ---: |
| clean | 143 |
| degraded | 73 |
| error | 75 |
| found | 4 |

| Queue disposition | Count |
| --- | ---: |
| done | 220 |
| deferred | 74 |
| failed | 1 |

These are scanner/queue outcomes, not transport failures. All corresponding
result bundles were accepted and projected. In particular, timeout and scanner
error results remained normal uploadable terminal results.

### Persisted detail

The stable capture contains:

- 295 admission intents, reservations, bundles, target scans, compatibility rows,
  and completed scan projection jobs;
- 2,905 ordered progress events;
- 75 structured diagnostics;
- 129 normalized scan errors;
- 4 normalized findings, 4 stable UID mappings, and 4 bounded compatibility
  payloads;
- 4 pending keycheck candidates linked to 2 normalized credentials;
- 374 appended projection records across 3 streams;
- 956 pipeline artifacts, all deleted by the final snapshot; and
- 19 successful typed runtime operations with 38 chained audit events.

The diagnostic aggregate was:

| Worker | Scanner result errors | Stage timeouts | Total diagnostics |
| --- | ---: | ---: | ---: |
| Windows | 55 | 8 | 63 |
| Linux | 1 | 11 | 12 |

Of the Windows scanner-result errors, 54 were retryable and one was
non-retryable. The Linux scanner-result error was retryable. Complete safe
diagnostic envelopes, exception identities, bounded process-log representations,
occurrence times, receipt authority, and scan links were retained and checked.

## End-to-end integrity checks

`build/operator-experience-validation/analyze-live-trace-final-20260925.py`
executed 65,675 checks with zero failures. It verified, row by row:

- admission, reservation, queue, bundle, scan, compatibility, and projection
  foreign-key relationships;
- receipt, payload, bundle, event, device, and deadline identities;
- canonical SHA-256 values for execution snapshots, progress events,
  diagnostics, compatibility metadata, plans, projection events, and finding
  payloads;
- exact bounded reconstruction of the four compatibility findings, including
  their original numeric detector identity and explicit null mapped fields;
- every normalized error, finding, keycheck candidate, credential reference,
  projection append, capacity release, and deleted artifact;
- all 19 operation-to-audit pairs; and
- the complete 38-event audit parent/hash chain.

`build/operator-experience-validation/analyze-worker-states-final-20260925.py`
executed a further 28,686 checks with zero failures. It parsed every retained
worker JSON/JSONL record and verified:

- 166 Windows and 129 Linux history rows against the server receipts;
- 2,338 contiguous Windows events and 1,672 contiguous Linux events;
- receipt payload, bundle, event, acceptance-time, and reservation identities;
- clean local shutdown with `drained=true`, `exit_code=0`, and empty progress
  outboxes; and
- no active work root in either final worker snapshot.

The two analyzers therefore executed 94,361 deterministic checks without a
failure.

## Recovery and shutdown

The validation exercised durable recovery rather than only clean executions:

- transient server `502` responses were retained in both local worker logs and
  recovered without duplicate authoritative acceptance;
- one Linux assignment survived a worker stop in the persistent volume, resumed
  after restart, produced one accepted result, and released all capacity;
- a transient assignment-status network failure retried the same durable
  assignment without rescanning or data loss; and
- both workers then drained and stopped cleanly.

The final local states were:

| Worker | State | Drained | Exit | Pending outbox |
| --- | --- | --- | ---: | ---: |
| Windows | stopped | true | 0 | 0 |
| Linux container | exited | true | 0 | 0 |

Retained `work/abandoned` roots are inactive evidence governed by normal worker
retention. They are not active assignments.

## Worker API validation

Authenticated worker API validation covered:

- device identity, package-manifest trust, and assignment-cap enforcement;
- claim, reservation replay, assignment status, and immutable execution snapshot;
- monotonic progress submission and latest-progress readback;
- bounded diagnostic body and process-log payloads;
- durable bundle upload, idempotent receipt replay, and accepted resolution;
- local restart recovery and terminal history;
- server ingestion, normalized scan authority, compatibility reconstruction, and
  projection completion; and
- terminal capacity release and zero unresolved work.

The API preserved receipt, payload, scan-event, diagnostic, and reservation
identities throughout the cohort. A separate terminal readback defect affecting
only `scan_deadline_at` is documented below.

## Admin UI and operator workflow

The authenticated admin UI was exercised through a browser across the complete
operator surface:

- Workers / Dispatch: control state, users, devices, assignment caps, enable,
  disable, revoke, unrevoke, and bounded worker detail;
- Overview: runtime health, producer lifecycle, pipeline workers, leases, queue
  counts, capacity, controls, recent operations, and duration groups;
- Search: bounded assignment, scan, finding, error, diagnostic, and progress
  lookup with safe empty and populated states;
- Supervisor: source start, restart, stop, lifecycle, and safe error rendering;
- Logs: bounded source/component/level/time filters and empty-result handling;
- Config and Secrets: active identities, stale-candidate warning, redacted
  projections, validation, and non-secret operation results;
- Files: bounded listing, file identity/hash verification, and safe download
  behavior;
- Operations: accepted/running/succeeded projections and filters; and
- Audit: accepted/succeeded event pairs, pagination, actor/action filters, and
  parent/hash continuity.

The final UI snapshot showed:

- Supervisor `ACTIVE` and PostgreSQL `READY`;
- result ingester, JSONL projector, janitor, and worker API all running;
- ingester and projector leases ready;
- control revision `126`, discovery and dispatch open, drain state normal;
- zero active assignments and zero pre-commit bundles; and
- zero quarantined queue rows.

The current DockerHub producer safe state remained `runtime_error`; it is the
known retry-coalescing defect plus unavailable credential pool described below,
not an unclassified new failure.

## Server and pipeline final state

The final server snapshot at 2026-09-25 14:59 UTC confirmed:

- 295 issued, 295 accepted, 295 ingested, 295 projected, and 295 settled;
- 0 unresolved, expired, pre-bundle-failed, pre-commit, quarantine, and drain
  blockers;
- bundle, projection, and quarantine capacity at zero;
- keycheck capacity at 137 items / 27,262,866 bytes, representing real pending
  work rather than leaked assignment or projection capacity;
- runtime container healthy with zero restarts and no OOM;
- edge container running with zero restarts and no OOM; and
- dashboard health endpoint returning `200 ok`.

## Defects found

### 1. Windows scanner timestamps lose their UTC offset

Status: open in the accepted Windows package.

Naive local scanner timestamps are relabeled as UTC, producing approximately a
three-hour future displacement on the validation host. Progress transport and
monotonic durations remain correct, but diagnostic ordering, time filters, and
scan start/end instants are wrong.

Detailed evidence and correction:
`docs/defect-windows-scan-timestamps-utc-2026-09-25.md`.

### 2. DockerHub retry coalescing fails with a null retry time

Status: open in the live runtime; fixed locally but not deployed.

PostgreSQL cannot infer the type of a nullable retry placeholder while
coalescing an existing row, raising SQLSTATE `42P18`. The managed producer masks
that exception as a generic delegation failure. A minimal local correction casts
the placeholder to text, and regression coverage now exercises same-row null-time
coalescing.

Detailed evidence and local fix:
`docs/defect-dockerhub-discovery-retry-null-type-2026-09-25.md`.

### 3. Terminal status can lose the durable scan deadline

Status: open in the live runtime.

A later progress event with a null scan deadline can replace the durable
receipt's concrete immutable deadline in authenticated terminal status readback.
The persisted receipt and all other identities remain correct.

Detailed evidence:
`docs/defect-terminal-status-scan-deadline-readback-2026-09-25.md`.

### 4. Network `OSError` is mislabeled as local I/O

Status: open in the accepted workers.

The broad safe-summary branch classifies socket/transport `OSError` as
`local I/O operation failed`. Recovery worked and no data was lost, but the
operator message incorrectly points toward local storage.

Detailed evidence:
`docs/defect-worker-network-oserror-mislabeled-local-io-2026-09-25.md`.

## Operational warnings

- The server root filesystem was approximately 90% used with roughly 1 GiB free.
  Containers remained healthy, but capacity should be reclaimed or expanded.
- The DockerHub credential pool was independently unavailable: ten credentials
  were invalid and the remaining entry was rate-limited. Deploying the SQL fix
  preserves retries correctly but cannot make an unavailable credential pool
  healthy.

## Tests and specification gates

The retained gates are:

- worker/operator focused matrix: `355 passed, 3 skipped`;
- admin/runtime focused matrix: `124 passed, 2 warnings`;
- DockerHub incremental discovery matrix: `27 passed`;
- SQLite retry lifecycle regression: `1 passed`;
- corrected PostgreSQL statement verified transactionally against the live schema
  and rolled back; and
- OpenSpec strict validation passed with all 27 implementation tasks complete.

The disposable PostgreSQL integration test remains skipped locally because no
disposable DSN was configured. The live transactional SQL verification did not
persist a change.

## Evidence manifest

| Artifact | Bytes | SHA-256 |
| --- | ---: | --- |
| `build/live-trace-20260925/raw-evidence-final.json` | 10,030,750 | `4071e38a1dec540663bc6febd9538ff54bedafa1627250933045beb8f7d09ec5` |
| `build/live-trace-20260925/monitor-final.ndjson` | 1,260,087 | `e07507b0b8f0f86d1a1c7aade7186297c372b1ff177067bea19dfd219f5027a9` |
| `build/live-trace-20260925/summary-final.json` | 3,669 | `19e5ec40cf354c6095a478b68a69f8e51fb3b8ea1c6f278c1d9c90bdaab9ebe2` |
| `build/live-trace-20260925/final-analysis-summary.json` | 946 | `2d17605ba7b730ad78a48bcc70e40e78f90637e3fd59004ebf5eb4a08241ba42` |
| `build/live-trace-20260925/final-worker-state-analysis-summary.json` | 1,376 | `b0ab428eb08587f13f59a1763f835125216f769713e259d85989ea8b7f8af95c` |
| `build/live-trace-20260925/linux-worker-state-final.tar.gz` | 134,380 | `b64e81c90b232f46b400a63ed08f5660f46e34fedb1f67b64afba079d8d36364` |

The final Windows state is retained under
`build/live-trace-20260925/windows-localappdata/TRUF/RemoteWorker`.

## Deliberately retained live state

The user requested that validation state not be restored. The following changes
therefore remain deliberate:

- accepted Windows and Linux trusted manifests remain installed;
- the keycheck queue maximum remains increased from 4,096 to 8,192 items;
- the Linux validation user remains enabled at assignment cap `0`;
- the Windows validation user remains enabled at assignment cap `1`;
- both validation workers themselves are stopped;
- normal global controls remain open at revision `126`; and
- the dashboard remains running.

The config editor still contains an intentionally stale candidate based on the
pre-validation active hash. It must not be applied without first rebasing it onto
the current active configuration.

## Release conclusion

The accepted worker artifacts passed live cross-platform execution, concurrent
dispatch, bounded progress/diagnostics, durable recovery, one-authoritative
receipt handling, normalized ingestion, projection, audit, local shutdown, and
operator UI validation. The complete cohort settled without leaked worker,
bundle, projection, or quarantine capacity.

Before broad rollout, deploy and revalidate the DockerHub SQL correction, decide
release treatment for the Windows timestamp and terminal-deadline defects, fix
network error classification, and address server disk pressure and DockerHub
credential health. The OpenSpec change is complete but remains unarchived until
explicitly requested.