# Worker Operator Experience Validation - 2026-09-24 ## Scope and acceptance This report closes the release and production-proof work for OpenSpec change `add-worker-operator-experience`. Validation covered the shared worker event contract, local supervisor and contained runner, progress and diagnostics APIs, admin projections, reproducible Windows and Linux packages, packaged cross-platform operation, bounded production behavior, and final restoration. Acceptance required: - the focused unit, integration, protocol, and package matrix to pass; - independently reproducible Windows and Linux artifacts with documented and registered package manifests; - packaged Windows/Linux evidence for multi-slot operation, outage and restart recovery, durable bundles and receipts, shutdown, and local cleanup; - bounded production evidence for progress, a full scan-stage timeout, diagnostics, reconciliation, and restoration; and - a from-zero operator runbook covering acquisition through removal. No production assignment was repeated to prepare this report. All production facts below are derived from the retained validation snapshots. Raw targets, findings, credentials, private routes, runtime configuration, and authenticated worker command lines are intentionally excluded. ## Focused test matrix The final 14-file worker-operator matrix completed on 2026-09-25: ```text 355 passed, 3 skipped in 49.54s ``` The matrix includes worker API/runtime, assignment and contained-runner, CLI/contracts/local state, observability persistence, package, Linux handoff, supervisor, remote database, scan execution, and admin API coverage. The three independent-watchdog timing regressions also passed after their test setup bounds were stabilized. The timing change did not alter product deadlines or watchdog behavior. An unrestricted repository-wide test run is not a release gate for this change: the checkout has unrelated missing private/generated assets and platform assumptions. The focused matrix, packaged E2E, production evidence, and strict OpenSpec validation are the scoped gates. ## Reproducible artifacts ### Windows portable package Final independently built archives: - `build/operator-experience-validation/windows-i.zip` - `build/operator-experience-validation/windows-j.zip` Both archives have the following identical identities: | Identity | Value | | --- | --- | | Archive bytes | `134850988` | | Archive SHA-256 | `6ea9290736a059f1e17d8e89d9cf83506fa4abe2ba2f3731a7422a7b0f386e97` | | Package manifest identity | `78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6` | | Build-input identity | `6991ebbce6ae758c2bdd19a6ae934335aa585a50f86b18ccde8d88bca40ce436` | | Raw `worker-package.json` SHA-256 | `e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c` | Acceptance used the fresh extraction at `build/pwe-final-i-extracted`. Its own `prepare-worker.ps1` established protected explicit ACLs before direct package verification. Older G/H Windows archives are excluded because their preparation script could leave packaged executables inaccessible. ### Linux worker image Final independently built local tags: - `truf-worker-test:operator-experience-final-3g` - `truf-worker-test:operator-experience-final-3h` Both provenance-disabled builds have the following identical identities: | Identity | Value | | --- | --- | | Worker package identity | `45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42` | | Image manifest / accepted image ID | `sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3` | | Config SHA-256 | `sha256:687a1c4c51c1b962c7fa7ea0cc4b04d159e7ba4f94ef347940c9fb225f7cb87d` | | Raw `worker-package.json` SHA-256 | `ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60` | The retained manifest snapshot is `build/operator-experience-validation/linux-worker-package-g.json`. ### Trusted manifest registration The accepted Linux and Windows manifests were registered after packaged E2E acceptance. Their remote SHA-256 values match the raw manifest hashes above. Both files are owned by `root:root` with mode `0644`; pre-change backups remain intact and upload temporary files were removed. Registration required no runtime restart or configuration mutation, and canonical health remained successful. ## Packaged Windows/Linux E2E Run `35f3f52e232067c1` passed with the freshly extracted/prepared Windows I package and Linux G image. The safe summary is `build/pwe-35f3f52e232067c1/summary.json`. The gate confirmed: - real packaged Windows and Linux operation at two slots; - server outage handling and restart recovery; - durable and direct assignment bundle paths; - authoritative receipt handling; - graceful shutdown receipts; - no active local work after completion while intentionally retained abandoned roots remained inactive; - matching normalized cross-platform evidence; and - complete cleanup of owned resources with foreign Docker state unchanged. ## Production evidence ### Reconciliation The retained snapshot records 34 issued assignments: 33 accepted and one intentional expected expiry. All 33 accepted bundles were ingested, settled, and projected. Final unresolved, precommit, quarantine, and drain-blocker counts were zero. Evidence sources: - `build/operator-experience-validation/final-evidence.json` - `build/operator-experience-validation/progress-v3-evidence.json` - `build/operator-experience-validation/timeout-evidence.json` - `build/operator-experience-validation/server-baseline.json` ### Duration percentiles The table reports every retained end-to-end metric group. Values are seconds. `Sufficient` means the server-side minimum sample count of five was met. Rows below that minimum are retained observations, not statistically sufficient percentile estimates. | Platform | Source | Outcome | Samples | p50 | p95 | p99 | Sufficient | | --- | --- | --- | ---: | ---: | ---: | ---: | --- | | Linux | DockerHub | degraded | 1 | 305 | 305 | 305 | no | | Linux | DockerHub | error | 1 | 19 | 19 | 19 | no | | Linux | GitLab | error | 1 | 5371 | 5371 | 5371 | no | | Linux | HuggingFace | expired | 1 | 7219 | 7219 | 7219 | no | | Windows | DockerHub | clean | 2 | 31 | 31.9 | 31.98 | no | | Windows | DockerHub | degraded | 4 | 275.5 | 443.45 | 462.29 | no | | Windows | DockerHub | error | 7 | 619 | 1679.5 | 1731.1 | yes | | Windows | GitLab | clean | 3 | 27 | 873 | 948.2 | no | | Windows | GitLab | error | 6 | 342.5 | 1723.75 | 1916.75 | yes | | Windows | HuggingFace | clean | 1 | 1019 | 1019 | 1019 | no | | Windows | HuggingFace | error | 7 | 971 | 3092.6 | 3452.12 | yes | The snapshot contains 11 Linux and 21 Windows phase/outcome metric groups in total. Three Windows end-to-end error groups met the minimum; the other 29 phase/outcome groups did not. Rollout decisions must therefore preserve the sample-count qualification rather than treating all reported percentiles as stable capacity estimates. ### Progress and watchdog evidence Reservation `1455` is the retained complete-stage progress reference. It was acknowledged with an accepted bundle and persisted ten monotonic events spanning `assigned`, `preparing`, `waiting_permit`, `scanning`, `filtering`, `cleaning`, `bundling`, `uploading`, and `awaiting_receipt`. This demonstrates one coherent server-visible sequence across the complete local execution and upload boundary. Independent watchdog fault-injection coverage passed for blocked state persistence, startup-gate persistence, and event draining. The contained runner tests verify bounded process-tree termination rather than relying on scanner cooperation. Packaged E2E additionally passed its watchdog, restart, durable bundle, and cleanup gates. ### Natural full-stage timeout Reservation `1453` is the retained natural timeout reference. The scan ended with one `timeout / scan.stage_timeout / scanning` diagnostic after 603.367 seconds. The diagnostic was current and available, with no body or process-log payload fabricated for the exception. The end-to-end assignment-resolution duration was 619 seconds. The scan outcome was `error`, while the transport outcome was independently accepted: the bundle was acknowledged, the projection completed, and the diagnostic was attached to the authoritative result. This confirms that a hard scan-stage timeout remains a normal, uploadable terminal result and does not collapse scan, transport, and projection outcomes into one status. ### Diagnostic and admin snapshots The final diagnostic aggregate contains five grouped rows and 23 occurrences: | Category | Code | Phase | Occurrences | | --- | --- | --- | ---: | | scanner | `scan.result_error` | `scanning` | 20 | | timeout | `scan.stage_timeout` | `scanning` | 1 | | assignment expiry | `assignment.deadline_expired` | `assigned` | 1 | | network | `scan.result_error` | `scanning` | 1 | The retained snapshots also confirm separate assignment and scan outcomes, ordered progress, current diagnostic availability, accepted receipt state, ingestion/settlement/projection completion, duration metrics, and zero unresolved or precommit work. This is the durable machine-readable substitute for copying private admin pages or unbounded diagnostic bodies into the report. ## Sanitized operator transcript The release and validation sequence was: 1. Build the Windows package twice from the same reviewed inputs and compare the archive, package-manifest, build-input, and raw-manifest identities. 2. Extract Windows I into a fresh directory, run its packaged `prepare-worker.ps1`, and run direct package verification. 3. Build Linux G and H independently with provenance disabled and compare image, config, package, and raw-manifest identities. 4. Run `docker/verify_packaged_workers.py` with the accepted Windows extraction, Linux image, and isolated test image; retain only its safe summary and owned evidence directory. 5. Register the two accepted trusted manifests through the reviewed deployment path and recheck canonical runtime health. 6. Use typed operations to bound production dispatch, start at assignment cap `1`, observe status/attach/history and server progress, exercise normal, timeout, outage, and restart paths, and reconcile accepted, ingested, settled, and projected counts. 7. Restore standard identities and normal/open controls, disable/revoke temporary validation identities, and recheck runtime and edge health. 8. Run the exact focused pytest matrix recorded above. Authentication values, worker argv, raw targets/findings, private route names, and runtime configuration are omitted by design. ## Known limits - There is no public ZIP download, image registry, installer, or automatic updater. Release artifacts must move through a trusted channel and match a server-registered manifest. - Completed runner roots move under top-level `work/abandoned` and are retained for at least 60 seconds; normal retention maintenance runs every 300 seconds. They are inactive evidence, not live work. - Most retained percentile groups have fewer than five samples. Their values are useful validation observations but not stable performance baselines. - Ownership fencing guarantees one authoritative acceptance, not exactly-once physical execution across a long partition and server-side expiry/reissue. - Local state remains recovery authority until the server resolves the slot. Operators must not remove pending bundles or work trees to clear an alert. ## Rollout, rollback, and restoration Rollout uses the exact accepted package identities, begins with one user/device at server cap `1` and local parallelism `1`, and requires one accepted, ingested, settled, and projected assignment before expansion. Caps and client count should increase in stages while unresolved/precommit counts, diagnostic availability, duration sample counts, and authenticated contact remain observable. Rollback first sets the affected cap to `0`, allows pending uploads to resolve, and obtains a graceful shutdown receipt. The operator then returns to the previous exact artifact while preserving the same private state tree or volume. Additive server progress and diagnostic records do not require schema rollback. Final restored production state: - operations controls normal/open at revision `126`; - standard WSL production worker user enabled at assignment cap `1`; - standard production device enabled and not revoked; - temporary validation identities disabled/revoked; - unresolved, precommit, quarantine, and drain blockers at zero; - canonical runtime healthy; and - edge service remained available. Strict OpenSpec validation passed. The operationally validated change is ready for archival.