Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,269 @@
|
||||
# Worker Operator Experience Validation - 2026-09-24
|
||||
|
||||
## Scope and acceptance
|
||||
|
||||
This report closes the release and production-proof work for OpenSpec change
|
||||
`add-worker-operator-experience`. Validation covered the shared worker event
|
||||
contract, local supervisor and contained runner, progress and diagnostics APIs,
|
||||
admin projections, reproducible Windows and Linux packages, packaged
|
||||
cross-platform operation, bounded production behavior, and final restoration.
|
||||
|
||||
Acceptance required:
|
||||
|
||||
- the focused unit, integration, protocol, and package matrix to pass;
|
||||
- independently reproducible Windows and Linux artifacts with documented and
|
||||
registered package manifests;
|
||||
- packaged Windows/Linux evidence for multi-slot operation, outage and restart
|
||||
recovery, durable bundles and receipts, shutdown, and local cleanup;
|
||||
- bounded production evidence for progress, a full scan-stage timeout,
|
||||
diagnostics, reconciliation, and restoration; and
|
||||
- a from-zero operator runbook covering acquisition through removal.
|
||||
|
||||
No production assignment was repeated to prepare this report. All production
|
||||
facts below are derived from the retained validation snapshots. Raw targets,
|
||||
findings, credentials, private routes, runtime configuration, and authenticated
|
||||
worker command lines are intentionally excluded.
|
||||
|
||||
## Focused test matrix
|
||||
|
||||
The final 14-file worker-operator matrix completed on 2026-09-25:
|
||||
|
||||
```text
|
||||
355 passed, 3 skipped in 49.54s
|
||||
```
|
||||
|
||||
The matrix includes worker API/runtime, assignment and contained-runner,
|
||||
CLI/contracts/local state, observability persistence, package, Linux handoff,
|
||||
supervisor, remote database, scan execution, and admin API coverage. The three
|
||||
independent-watchdog timing regressions also passed after their test setup bounds
|
||||
were stabilized. The timing change did not alter product deadlines or watchdog
|
||||
behavior.
|
||||
|
||||
An unrestricted repository-wide test run is not a release gate for this change:
|
||||
the checkout has unrelated missing private/generated assets and platform
|
||||
assumptions. The focused matrix, packaged E2E, production evidence, and strict
|
||||
OpenSpec validation are the scoped gates.
|
||||
|
||||
## Reproducible artifacts
|
||||
|
||||
### Windows portable package
|
||||
|
||||
Final independently built archives:
|
||||
|
||||
- `build/operator-experience-validation/windows-i.zip`
|
||||
- `build/operator-experience-validation/windows-j.zip`
|
||||
|
||||
Both archives have the following identical identities:
|
||||
|
||||
| Identity | Value |
|
||||
| --- | --- |
|
||||
| Archive bytes | `134850988` |
|
||||
| Archive SHA-256 | `6ea9290736a059f1e17d8e89d9cf83506fa4abe2ba2f3731a7422a7b0f386e97` |
|
||||
| Package manifest identity | `78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6` |
|
||||
| Build-input identity | `6991ebbce6ae758c2bdd19a6ae934335aa585a50f86b18ccde8d88bca40ce436` |
|
||||
| Raw `worker-package.json` SHA-256 | `e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c` |
|
||||
|
||||
Acceptance used the fresh extraction at `build/pwe-final-i-extracted`. Its own
|
||||
`prepare-worker.ps1` established protected explicit ACLs before direct package
|
||||
verification. Older G/H Windows archives are excluded because their preparation
|
||||
script could leave packaged executables inaccessible.
|
||||
|
||||
### Linux worker image
|
||||
|
||||
Final independently built local tags:
|
||||
|
||||
- `truf-worker-test:operator-experience-final-3g`
|
||||
- `truf-worker-test:operator-experience-final-3h`
|
||||
|
||||
Both provenance-disabled builds have the following identical identities:
|
||||
|
||||
| Identity | Value |
|
||||
| --- | --- |
|
||||
| Worker package identity | `45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42` |
|
||||
| Image manifest / accepted image ID | `sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3` |
|
||||
| Config SHA-256 | `sha256:687a1c4c51c1b962c7fa7ea0cc4b04d159e7ba4f94ef347940c9fb225f7cb87d` |
|
||||
| Raw `worker-package.json` SHA-256 | `ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60` |
|
||||
|
||||
The retained manifest snapshot is
|
||||
`build/operator-experience-validation/linux-worker-package-g.json`.
|
||||
|
||||
### Trusted manifest registration
|
||||
|
||||
The accepted Linux and Windows manifests were registered after packaged E2E
|
||||
acceptance. Their remote SHA-256 values match the raw manifest hashes above.
|
||||
Both files are owned by `root:root` with mode `0644`; pre-change backups remain
|
||||
intact and upload temporary files were removed. Registration required no runtime
|
||||
restart or configuration mutation, and canonical health remained successful.
|
||||
|
||||
## Packaged Windows/Linux E2E
|
||||
|
||||
Run `35f3f52e232067c1` passed with the freshly extracted/prepared Windows I
|
||||
package and Linux G image. The safe summary is
|
||||
`build/pwe-35f3f52e232067c1/summary.json`.
|
||||
|
||||
The gate confirmed:
|
||||
|
||||
- real packaged Windows and Linux operation at two slots;
|
||||
- server outage handling and restart recovery;
|
||||
- durable and direct assignment bundle paths;
|
||||
- authoritative receipt handling;
|
||||
- graceful shutdown receipts;
|
||||
- no active local work after completion while intentionally retained abandoned
|
||||
roots remained inactive;
|
||||
- matching normalized cross-platform evidence; and
|
||||
- complete cleanup of owned resources with foreign Docker state unchanged.
|
||||
|
||||
## Production evidence
|
||||
|
||||
### Reconciliation
|
||||
|
||||
The retained snapshot records 34 issued assignments: 33 accepted and one
|
||||
intentional expected expiry. All 33 accepted bundles were ingested, settled, and
|
||||
projected. Final unresolved, precommit, quarantine, and drain-blocker counts were
|
||||
zero.
|
||||
|
||||
Evidence sources:
|
||||
|
||||
- `build/operator-experience-validation/final-evidence.json`
|
||||
- `build/operator-experience-validation/progress-v3-evidence.json`
|
||||
- `build/operator-experience-validation/timeout-evidence.json`
|
||||
- `build/operator-experience-validation/server-baseline.json`
|
||||
|
||||
### Duration percentiles
|
||||
|
||||
The table reports every retained end-to-end metric group. Values are seconds.
|
||||
`Sufficient` means the server-side minimum sample count of five was met. Rows
|
||||
below that minimum are retained observations, not statistically sufficient
|
||||
percentile estimates.
|
||||
|
||||
| Platform | Source | Outcome | Samples | p50 | p95 | p99 | Sufficient |
|
||||
| --- | --- | --- | ---: | ---: | ---: | ---: | --- |
|
||||
| Linux | DockerHub | degraded | 1 | 305 | 305 | 305 | no |
|
||||
| Linux | DockerHub | error | 1 | 19 | 19 | 19 | no |
|
||||
| Linux | GitLab | error | 1 | 5371 | 5371 | 5371 | no |
|
||||
| Linux | HuggingFace | expired | 1 | 7219 | 7219 | 7219 | no |
|
||||
| Windows | DockerHub | clean | 2 | 31 | 31.9 | 31.98 | no |
|
||||
| Windows | DockerHub | degraded | 4 | 275.5 | 443.45 | 462.29 | no |
|
||||
| Windows | DockerHub | error | 7 | 619 | 1679.5 | 1731.1 | yes |
|
||||
| Windows | GitLab | clean | 3 | 27 | 873 | 948.2 | no |
|
||||
| Windows | GitLab | error | 6 | 342.5 | 1723.75 | 1916.75 | yes |
|
||||
| Windows | HuggingFace | clean | 1 | 1019 | 1019 | 1019 | no |
|
||||
| Windows | HuggingFace | error | 7 | 971 | 3092.6 | 3452.12 | yes |
|
||||
|
||||
The snapshot contains 11 Linux and 21 Windows phase/outcome metric groups in
|
||||
total. Three Windows end-to-end error groups met the minimum; the other 29
|
||||
phase/outcome groups did not. Rollout decisions must therefore preserve the
|
||||
sample-count qualification rather than treating all reported percentiles as
|
||||
stable capacity estimates.
|
||||
|
||||
### Progress and watchdog evidence
|
||||
|
||||
Reservation `1455` is the retained complete-stage progress reference. It was
|
||||
acknowledged with an accepted bundle and persisted ten monotonic events spanning
|
||||
`assigned`, `preparing`, `waiting_permit`, `scanning`, `filtering`, `cleaning`,
|
||||
`bundling`, `uploading`, and `awaiting_receipt`. This demonstrates one coherent
|
||||
server-visible sequence across the complete local execution and upload boundary.
|
||||
|
||||
Independent watchdog fault-injection coverage passed for blocked state
|
||||
persistence, startup-gate persistence, and event draining. The contained runner
|
||||
tests verify bounded process-tree termination rather than relying on scanner
|
||||
cooperation. Packaged E2E additionally passed its watchdog, restart, durable
|
||||
bundle, and cleanup gates.
|
||||
|
||||
### Natural full-stage timeout
|
||||
|
||||
Reservation `1453` is the retained natural timeout reference. The scan ended
|
||||
with one `timeout / scan.stage_timeout / scanning` diagnostic after 603.367
|
||||
seconds. The diagnostic was current and available, with no body or process-log
|
||||
payload fabricated for the exception. The end-to-end assignment-resolution
|
||||
duration was 619 seconds.
|
||||
|
||||
The scan outcome was `error`, while the transport outcome was independently
|
||||
accepted: the bundle was acknowledged, the projection completed, and the
|
||||
diagnostic was attached to the authoritative result. This confirms that a hard
|
||||
scan-stage timeout remains a normal, uploadable terminal result and does not
|
||||
collapse scan, transport, and projection outcomes into one status.
|
||||
|
||||
### Diagnostic and admin snapshots
|
||||
|
||||
The final diagnostic aggregate contains five grouped rows and 23 occurrences:
|
||||
|
||||
| Category | Code | Phase | Occurrences |
|
||||
| --- | --- | --- | ---: |
|
||||
| scanner | `scan.result_error` | `scanning` | 20 |
|
||||
| timeout | `scan.stage_timeout` | `scanning` | 1 |
|
||||
| assignment expiry | `assignment.deadline_expired` | `assigned` | 1 |
|
||||
| network | `scan.result_error` | `scanning` | 1 |
|
||||
|
||||
The retained snapshots also confirm separate assignment and scan outcomes,
|
||||
ordered progress, current diagnostic availability, accepted receipt state,
|
||||
ingestion/settlement/projection completion, duration metrics, and zero unresolved
|
||||
or precommit work. This is the durable machine-readable substitute for copying
|
||||
private admin pages or unbounded diagnostic bodies into the report.
|
||||
|
||||
## Sanitized operator transcript
|
||||
|
||||
The release and validation sequence was:
|
||||
|
||||
1. Build the Windows package twice from the same reviewed inputs and compare the
|
||||
archive, package-manifest, build-input, and raw-manifest identities.
|
||||
2. Extract Windows I into a fresh directory, run its packaged
|
||||
`prepare-worker.ps1`, and run direct package verification.
|
||||
3. Build Linux G and H independently with provenance disabled and compare image,
|
||||
config, package, and raw-manifest identities.
|
||||
4. Run `docker/verify_packaged_workers.py` with the accepted Windows extraction,
|
||||
Linux image, and isolated test image; retain only its safe summary and owned
|
||||
evidence directory.
|
||||
5. Register the two accepted trusted manifests through the reviewed deployment
|
||||
path and recheck canonical runtime health.
|
||||
6. Use typed operations to bound production dispatch, start at assignment cap
|
||||
`1`, observe status/attach/history and server progress, exercise normal,
|
||||
timeout, outage, and restart paths, and reconcile accepted, ingested, settled,
|
||||
and projected counts.
|
||||
7. Restore standard identities and normal/open controls, disable/revoke temporary
|
||||
validation identities, and recheck runtime and edge health.
|
||||
8. Run the exact focused pytest matrix recorded above.
|
||||
|
||||
Authentication values, worker argv, raw targets/findings, private route names,
|
||||
and runtime configuration are omitted by design.
|
||||
|
||||
## Known limits
|
||||
|
||||
- There is no public ZIP download, image registry, installer, or automatic
|
||||
updater. Release artifacts must move through a trusted channel and match a
|
||||
server-registered manifest.
|
||||
- Completed runner roots move under top-level `work/abandoned` and are retained
|
||||
for at least 60 seconds; normal retention maintenance runs every 300 seconds.
|
||||
They are inactive evidence, not live work.
|
||||
- Most retained percentile groups have fewer than five samples. Their values are
|
||||
useful validation observations but not stable performance baselines.
|
||||
- Ownership fencing guarantees one authoritative acceptance, not exactly-once
|
||||
physical execution across a long partition and server-side expiry/reissue.
|
||||
- Local state remains recovery authority until the server resolves the slot.
|
||||
Operators must not remove pending bundles or work trees to clear an alert.
|
||||
|
||||
## Rollout, rollback, and restoration
|
||||
|
||||
Rollout uses the exact accepted package identities, begins with one user/device at
|
||||
server cap `1` and local parallelism `1`, and requires one accepted, ingested,
|
||||
settled, and projected assignment before expansion. Caps and client count should
|
||||
increase in stages while unresolved/precommit counts, diagnostic availability,
|
||||
duration sample counts, and authenticated contact remain observable.
|
||||
|
||||
Rollback first sets the affected cap to `0`, allows pending uploads to resolve,
|
||||
and obtains a graceful shutdown receipt. The operator then returns to the
|
||||
previous exact artifact while preserving the same private state tree or volume.
|
||||
Additive server progress and diagnostic records do not require schema rollback.
|
||||
|
||||
Final restored production state:
|
||||
|
||||
- operations controls normal/open at revision `126`;
|
||||
- standard WSL production worker user enabled at assignment cap `1`;
|
||||
- standard production device enabled and not revoked;
|
||||
- temporary validation identities disabled/revoked;
|
||||
- unresolved, precommit, quarantine, and drain blockers at zero;
|
||||
- canonical runtime healthy; and
|
||||
- edge service remained available.
|
||||
|
||||
Strict OpenSpec validation passed. The operationally validated change is ready
|
||||
for archival.
|
||||
Reference in New Issue
Block a user