truf-server/docs/worker-operator-experience-validation-2026-09-24.md

# Worker Operator Experience Validation - 2026-09-24

## Scope and acceptance

This report closes the release and production-proof work for OpenSpec change
`add-worker-operator-experience`. Validation covered the shared worker event
contract, local supervisor and contained runner, progress and diagnostics APIs,
admin projections, reproducible Windows and Linux packages, packaged
cross-platform operation, bounded production behavior, and final restoration.

Acceptance required:

- the focused unit, integration, protocol, and package matrix to pass;
- independently reproducible Windows and Linux artifacts with documented and
  registered package manifests;
- packaged Windows/Linux evidence for multi-slot operation, outage and restart
  recovery, durable bundles and receipts, shutdown, and local cleanup;
- bounded production evidence for progress, a full scan-stage timeout,
  diagnostics, reconciliation, and restoration; and
- a from-zero operator runbook covering acquisition through removal.

No production assignment was repeated to prepare this report. All production
facts below are derived from the retained validation snapshots. Raw targets,
findings, credentials, private routes, runtime configuration, and authenticated
worker command lines are intentionally excluded.

## Focused test matrix

The final 14-file worker-operator matrix completed on 2026-09-25:

```text
355 passed, 3 skipped in 49.54s
```

The matrix includes worker API/runtime, assignment and contained-runner,
CLI/contracts/local state, observability persistence, package, Linux handoff,
supervisor, remote database, scan execution, and admin API coverage. The three
independent-watchdog timing regressions also passed after their test setup bounds
were stabilized. The timing change did not alter product deadlines or watchdog
behavior.

An unrestricted repository-wide test run is not a release gate for this change:
the checkout has unrelated missing private/generated assets and platform
assumptions. The focused matrix, packaged E2E, production evidence, and strict
OpenSpec validation are the scoped gates.

## Reproducible artifacts

### Windows portable package

Final independently built archives:

- `build/operator-experience-validation/windows-i.zip`
- `build/operator-experience-validation/windows-j.zip`

Both archives have the following identical identities:

| Identity | Value |
| --- | --- |
| Archive bytes | `134850988` |
| Archive SHA-256 | `6ea9290736a059f1e17d8e89d9cf83506fa4abe2ba2f3731a7422a7b0f386e97` |
| Package manifest identity | `78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6` |
| Build-input identity | `6991ebbce6ae758c2bdd19a6ae934335aa585a50f86b18ccde8d88bca40ce436` |
| Raw `worker-package.json` SHA-256 | `e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c` |

Acceptance used the fresh extraction at `build/pwe-final-i-extracted`. Its own
`prepare-worker.ps1` established protected explicit ACLs before direct package
verification. Older G/H Windows archives are excluded because their preparation
script could leave packaged executables inaccessible.

### Linux worker image

Final independently built local tags:

- `truf-worker-test:operator-experience-final-3g`
- `truf-worker-test:operator-experience-final-3h`

Both provenance-disabled builds have the following identical identities:

| Identity | Value |
| --- | --- |
| Worker package identity | `45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42` |
| Image manifest / accepted image ID | `sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3` |
| Config SHA-256 | `sha256:687a1c4c51c1b962c7fa7ea0cc4b04d159e7ba4f94ef347940c9fb225f7cb87d` |
| Raw `worker-package.json` SHA-256 | `ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60` |

The retained manifest snapshot is
`build/operator-experience-validation/linux-worker-package-g.json`.

### Trusted manifest registration

The accepted Linux and Windows manifests were registered after packaged E2E
acceptance. Their remote SHA-256 values match the raw manifest hashes above.
Both files are owned by `root:root` with mode `0644`; pre-change backups remain
intact and upload temporary files were removed. Registration required no runtime
restart or configuration mutation, and canonical health remained successful.

## Packaged Windows/Linux E2E

Run `35f3f52e232067c1` passed with the freshly extracted/prepared Windows I
package and Linux G image. The safe summary is
`build/pwe-35f3f52e232067c1/summary.json`.

The gate confirmed:

- real packaged Windows and Linux operation at two slots;
- server outage handling and restart recovery;
- durable and direct assignment bundle paths;
- authoritative receipt handling;
- graceful shutdown receipts;
- no active local work after completion while intentionally retained abandoned
  roots remained inactive;
- matching normalized cross-platform evidence; and
- complete cleanup of owned resources with foreign Docker state unchanged.

## Production evidence

### Reconciliation

The retained snapshot records 34 issued assignments: 33 accepted and one
intentional expected expiry. All 33 accepted bundles were ingested, settled, and
projected. Final unresolved, precommit, quarantine, and drain-blocker counts were
zero.

Evidence sources:

- `build/operator-experience-validation/final-evidence.json`
- `build/operator-experience-validation/progress-v3-evidence.json`
- `build/operator-experience-validation/timeout-evidence.json`
- `build/operator-experience-validation/server-baseline.json`

### Duration percentiles

The table reports every retained end-to-end metric group. Values are seconds.
`Sufficient` means the server-side minimum sample count of five was met. Rows
below that minimum are retained observations, not statistically sufficient
percentile estimates.

| Platform | Source | Outcome | Samples | p50 | p95 | p99 | Sufficient |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- |
| Linux | DockerHub | degraded | 1 | 305 | 305 | 305 | no |
| Linux | DockerHub | error | 1 | 19 | 19 | 19 | no |
| Linux | GitLab | error | 1 | 5371 | 5371 | 5371 | no |
| Linux | HuggingFace | expired | 1 | 7219 | 7219 | 7219 | no |
| Windows | DockerHub | clean | 2 | 31 | 31.9 | 31.98 | no |
| Windows | DockerHub | degraded | 4 | 275.5 | 443.45 | 462.29 | no |
| Windows | DockerHub | error | 7 | 619 | 1679.5 | 1731.1 | yes |
| Windows | GitLab | clean | 3 | 27 | 873 | 948.2 | no |
| Windows | GitLab | error | 6 | 342.5 | 1723.75 | 1916.75 | yes |
| Windows | HuggingFace | clean | 1 | 1019 | 1019 | 1019 | no |
| Windows | HuggingFace | error | 7 | 971 | 3092.6 | 3452.12 | yes |

The snapshot contains 11 Linux and 21 Windows phase/outcome metric groups in
total. Three Windows end-to-end error groups met the minimum; the other 29
phase/outcome groups did not. Rollout decisions must therefore preserve the
sample-count qualification rather than treating all reported percentiles as
stable capacity estimates.

### Progress and watchdog evidence

Reservation `1455` is the retained complete-stage progress reference. It was
acknowledged with an accepted bundle and persisted ten monotonic events spanning
`assigned`, `preparing`, `waiting_permit`, `scanning`, `filtering`, `cleaning`,
`bundling`, `uploading`, and `awaiting_receipt`. This demonstrates one coherent
server-visible sequence across the complete local execution and upload boundary.

Independent watchdog fault-injection coverage passed for blocked state
persistence, startup-gate persistence, and event draining. The contained runner
tests verify bounded process-tree termination rather than relying on scanner
cooperation. Packaged E2E additionally passed its watchdog, restart, durable
bundle, and cleanup gates.

### Natural full-stage timeout

Reservation `1453` is the retained natural timeout reference. The scan ended
with one `timeout / scan.stage_timeout / scanning` diagnostic after 603.367
seconds. The diagnostic was current and available, with no body or process-log
payload fabricated for the exception. The end-to-end assignment-resolution
duration was 619 seconds.

The scan outcome was `error`, while the transport outcome was independently
accepted: the bundle was acknowledged, the projection completed, and the
diagnostic was attached to the authoritative result. This confirms that a hard
scan-stage timeout remains a normal, uploadable terminal result and does not
collapse scan, transport, and projection outcomes into one status.

### Diagnostic and admin snapshots

The final diagnostic aggregate contains five grouped rows and 23 occurrences:

| Category | Code | Phase | Occurrences |
| --- | --- | --- | ---: |
| scanner | `scan.result_error` | `scanning` | 20 |
| timeout | `scan.stage_timeout` | `scanning` | 1 |
| assignment expiry | `assignment.deadline_expired` | `assigned` | 1 |
| network | `scan.result_error` | `scanning` | 1 |

The retained snapshots also confirm separate assignment and scan outcomes,
ordered progress, current diagnostic availability, accepted receipt state,
ingestion/settlement/projection completion, duration metrics, and zero unresolved
or precommit work. This is the durable machine-readable substitute for copying
private admin pages or unbounded diagnostic bodies into the report.

## Sanitized operator transcript

The release and validation sequence was:

1. Build the Windows package twice from the same reviewed inputs and compare the
   archive, package-manifest, build-input, and raw-manifest identities.
2. Extract Windows I into a fresh directory, run its packaged
   `prepare-worker.ps1`, and run direct package verification.
3. Build Linux G and H independently with provenance disabled and compare image,
   config, package, and raw-manifest identities.
4. Run `docker/verify_packaged_workers.py` with the accepted Windows extraction,
   Linux image, and isolated test image; retain only its safe summary and owned
   evidence directory.
5. Register the two accepted trusted manifests through the reviewed deployment
   path and recheck canonical runtime health.
6. Use typed operations to bound production dispatch, start at assignment cap
   `1`, observe status/attach/history and server progress, exercise normal,
   timeout, outage, and restart paths, and reconcile accepted, ingested, settled,
   and projected counts.
7. Restore standard identities and normal/open controls, disable/revoke temporary
   validation identities, and recheck runtime and edge health.
8. Run the exact focused pytest matrix recorded above.

Authentication values, worker argv, raw targets/findings, private route names,
and runtime configuration are omitted by design.

## Known limits

- There is no public ZIP download, image registry, installer, or automatic
  updater. Release artifacts must move through a trusted channel and match a
  server-registered manifest.
- Completed runner roots move under top-level `work/abandoned` and are retained
  for at least 60 seconds; normal retention maintenance runs every 300 seconds.
  They are inactive evidence, not live work.
- Most retained percentile groups have fewer than five samples. Their values are
  useful validation observations but not stable performance baselines.
- Ownership fencing guarantees one authoritative acceptance, not exactly-once
  physical execution across a long partition and server-side expiry/reissue.
- Local state remains recovery authority until the server resolves the slot.
  Operators must not remove pending bundles or work trees to clear an alert.

## Rollout, rollback, and restoration

Rollout uses the exact accepted package identities, begins with one user/device at
server cap `1` and local parallelism `1`, and requires one accepted, ingested,
settled, and projected assignment before expansion. Caps and client count should
increase in stages while unresolved/precommit counts, diagnostic availability,
duration sample counts, and authenticated contact remain observable.

Rollback first sets the affected cap to `0`, allows pending uploads to resolve,
and obtains a graceful shutdown receipt. The operator then returns to the
previous exact artifact while preserving the same private state tree or volume.
Additive server progress and diagnostic records do not require schema rollback.

Final restored production state:

- operations controls normal/open at revision `126`;
- standard WSL production worker user enabled at assignment cap `1`;
- standard production device enabled and not revoked;
- temporary validation identities disabled/revoked;
- unresolved, precommit, quarantine, and drain blockers at zero;
- canonical runtime healthy; and
- edge service remained available.

Strict OpenSpec validation passed. The operationally validated change is ready
for archival.