truf-server/WORKER_OPERATOR_EXPERIENCE_HANDOFF.md

Worker Operator Experience Handoff

Historical handoff. The current continuation entry point is docs/session-handoff/README.md. This file retains implementation and artifact provenance, but its stop point and immediate-next-actions section are obsolete.

Last updated: 2026-09-25

This is the historical implementation record for the OpenSpec change add-worker-operator-experience. For current continuation instructions, read docs/session-handoff/README.md. Do not repeat completed production validation or rebuild accepted artifacts unless a current verification fails.

User intent and constraints

• Continue autonomously from this handoff and finish the change end to end.
• The user explicitly requested a file handoff because invoking conversation compression appears to stop or destabilize all OpenCode sessions. Avoid proactively invoking the compression tool in the continuation session.
• Workspace: D:\truf-workers.
• Use the configured SSH server named sec only. Never call or connect through the configured server named prod.
• Never print or record tokens, credentials, the private admin prefix, raw targets/findings, runtime YAML, or worker command lines containing auth data.
• Do not add a new masking, redaction, credential-sandbox, or other security scope without explicit approval and an OpenSpec requirement.
• Do not remove or revert unrelated workspace files. The repository baseline is entirely untracked (git status --short shows the whole tree as ??), so Git cannot provide a meaningful task-specific diff.
• Do not archive the OpenSpec change unless the user explicitly asks. Completing tasks and reporting "ready to archive" is expected.

Current OpenSpec state

• Change: add-worker-operator-experience
• Schema: spec-driven
• Artifact status: proposal, design, specs, and tasks are complete.
• Apply progress before final closure: 23/27 tasks complete.
• File: openspec/changes/add-worker-operator-experience/tasks.md
• Tasks 1.1 through 5.5 are checked.
• Remaining unchecked tasks:
  • 6.1: validate the canonical from-zero operator guide.
  • 6.2: complete test matrix, reproducible packages, manifest registration, and documented identities.
  • 6.3: bounded Windows and WSL/Docker production validation and restoration.
  • 6.4: durable dated report with timings, watchdog evidence, snapshots, transcript, known limits, rollout, and rollback.

Do not check 6.1-6.4 until the remaining focused matrix, report, and strict OpenSpec validation have passed.

Implemented scope

The change now includes:

• Versioned worker phase/events and canonical transitions.
• Monotonic sequence handling and JSON/NDJSON contracts.
• Unified bounded diagnostics with deterministic identities.
• PostgreSQL progress/diagnostic persistence and admin queries.
• Server-owned global/per-source assignment deadline policy.
• Private local state, logs, history, diagnostic artifacts, and retention.
• Status, attach, logs/history, JSON/NDJSON, bounded follow/tail CLI behavior.
• Cross-platform worker supervisor, control protocol, drain/stop, shutdown receipts, stale instance handling, and recovery slots.
• Per-assignment contained runner, controller protocol, watchdog, timeout bundle publication, restart adoption, and abandoned-root cleanup.
• Authenticated progress endpoint and diagnostic ingestion.
• Admin assignment/progress/diagnostic experience.
• Windows portable and Linux image packaging for the supervisor runtime.
• Canonical operator runbook in docs/remote-worker-operations.md.

Important implementation files include:

• app/worker_contracts.py
• app/worker_local_state.py
• app/worker_supervisor.py
• app/worker_cli.py
• app/worker_assignment_runner.py
• app/remote_worker_client.py
• app/worker_api.py
• app/scanner_db.py
• app/admin_api.py
• app/worker_package.py
• app/worker_package_builder.py
• docker/verify_packaged_workers.py
• Worker-related tests under tests/

Final correctness fixes

Recovered ready-bundle transition

WorkerSlot could recover a published ready bundle while its persisted event phase was still assigned. Upload code emitted uploading only from bundling/backoff, then attempted the invalid transition assigned -> awaiting_receipt.

Fix in app/remote_worker_client.py:

• Emit UPLOADING when the current event phase is ASSIGNED, as well as the existing bundling/backoff cases.
• Regression in tests/test_worker_api.py validates the event sequence assigned -> uploading -> awaiting_receipt.

The focused worker API/local-state/supervisor suite passed 100 tests after this fix.

Packaged E2E abandoned work invariant

Completed runner roots are intentionally retained under work/abandoned for at least 60 seconds; retention maintenance normally runs every 300 seconds. The E2E harness incorrectly required the total work file count to be zero, causing a false linux_direct_claims_timeout after Linux had correctly claimed both direct assignments.

Fix in docker/verify_packaged_workers.py:

• Linux and Windows work-tree identities now include active_entries.
• Files/directories beneath top-level abandoned are retained but not active.
• Direct-assignment and final-cleanup predicates require zero active entries, while preserving strict state and bundle identity checks.
• Outage marker waits also check worker liveness, so an exited worker fails immediately rather than timing out after four minutes.

Windows prepare-worker.ps1 ACL defect

Testing a freshly extracted ZIP exposed a real release bug. The old generated script ran icacls ... /grant:r ... /T; on descendants this produced inheritance-only ACEs, returned success, and made packaged python.exe inaccessible.

Final fix in app/worker_package_builder.py:

1. Set private inheritable full-control ACEs for the current user, SYSTEM, and Administrators on the package root only.
2. Run icacls (Join-Path $root '*') /inheritance:d /T /C so each descendant converts inherited ACLs to explicit protected ACLs with the correct file or directory flags.

Regression in tests/test_worker_package.py checks the generated script and, on Windows, executes it and verifies private_directory_ready(root) plus private_file_ready(child). tests/test_worker_package.py passes 20 tests.

Do not use the earlier /reset /T idea: inherited ACLs are not accepted because runtime trust requires protected explicit ACLs.

Watchdog test timing stabilization

The broad focused suite exposed two false failures because three tests created a 100 ms absolute watchdog deadline before runner protocol-root/state setup. Under the complete Windows suite that setup could consume the deadline, exercising the startup-deadline branch instead of the intended blocked-operation watchdog.

Test-only changes in tests/test_worker_assignment_runner.py:

• Affected tests:
  • test_watchdog_kills_while_state_persistence_is_blocked
  • test_blocked_startup_gate_write_enters_preparing_timeout_result_path
  • test_watchdog_kills_while_event_drain_is_blocked
• Scan deadline: 1 second -> 2 seconds.
• Watchdog deadline: 0.1 second -> 1 second.
• Injected block: 0.4 second -> 1.4 seconds.
• Kill bound: 0.3 second -> 1.3 seconds.

This preserves the independent watchdog assertion and does not weaken product code. The exact three-test rerun passed: 3 passed in 5.17s.

Accepted reproducible artifacts

Windows final pair: I and J

Paths:

• build/operator-experience-validation/windows-i.zip
• build/operator-experience-validation/windows-i.zip.json
• build/operator-experience-validation/windows-j.zip
• build/operator-experience-validation/windows-j.zip.json

Both independently built archives are identical:

• Bytes: 134850988
• Archive SHA-256: 6ea9290736a059f1e17d8e89d9cf83506fa4abe2ba2f3731a7422a7b0f386e97
• Package manifest identity: 78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6
• Build-input identity: 6991ebbce6ae758c2bdd19a6ae934335aa585a50f86b18ccde8d88bca40ce436
• Raw worker-package.json SHA-256: e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c

Acceptance used a fresh extraction, not the builder output:

• build/pwe-final-i-extracted
• The package's own corrected prepare-worker.ps1 was run once.
• Direct package verification then passed.

The older Windows G/H archives are obsolete for acceptance because they contain the broken preparation script. Their package manifest identity happens to be the same because the support script is outside that manifest, but their archive identity is not accepted. Do not publish or register G/H as final Windows ZIPs.

Linux final pair: G and H

Tags:

• truf-worker-test:operator-experience-final-3g
• truf-worker-test:operator-experience-final-3h

Both were built with provenance disabled and are reproducible:

• Worker package identity: 45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42
• Image manifest / accepted image ID: sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3
• Config: sha256:687a1c4c51c1b962c7fa7ea0cc4b04d159e7ba4f94ef347940c9fb225f7cb87d
• Raw worker-package.json SHA-256: ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60

Extracted final manifest:

• build/operator-experience-validation/linux-worker-package-g.json

Test image:

• Tag: truf-worker-test:operator-experience-final-3
• ID: sha256:1a22c396dbf329e20caf77f88b7c7a310bda86befbcf3b917f10ded3720ee712

Final packaged E2E

Passed run:

• Run ID: 35f3f52e232067c1
• Safe summary: build/pwe-35f3f52e232067c1/summary.json
• Windows input: freshly extracted and prepared Windows I.
• Linux input: Linux G.
• Status: passed.
• Cleanup: complete.
• Foreign Docker state: unchanged.
• Windows and Linux normalized evidence matched.
• Restart, outage, durable bundle, direct assignment, direct bundle, receipt, shutdown, local cleanup, and cross-platform evidence gates all passed.

Do not copy raw target values from the summary into reports or chat. Only the safe aggregate facts above are needed.

All Docker resources from final and diagnosed failed runs were cleaned by exact owned IDs/names. Some local build/pwe-* failure evidence directories remain and are safe to leave. build/pwe-final-g may still have unusable ACLs after running the old broken preparation script; do not use it. build/pwe-final-i-extracted is the accepted extracted Windows directory.

Production validation evidence

Bounded production validation was completed before final package acceptance and production was restored afterward.

Safe aggregate results:

• Assignments issued: 34.
• Accepted: 33.
• One intentional expected expiry.
• Accepted assignments ingested, settled, and projected: 33.
• Unresolved, precommit, and quarantine counts: zero.
• Natural timeout evidence reservation: 1453.
• Full-stage progress/watchdog evidence reservation: 1455.

Evidence files:

• build/operator-experience-validation/final-evidence.json
• build/operator-experience-validation/progress-v3-evidence.json
• build/operator-experience-validation/timeout-evidence.json
• build/operator-experience-validation/server-baseline.json

These files are the source for duration percentiles, phase/watchdog evidence, diagnostic/admin snapshots, and reconciled counts in the final report. Derive only aggregate/sanitized facts. Do not reproduce raw targets, findings, secrets, or private route names.

Final production state after restoration:

• Operations controls: normal/open, revision 126.
• Standard WSL production worker user: enabled, assignment cap 1.
• Standard production device: enabled and not revoked.
• Temporary validation identities: disabled/revoked.
• Runtime canonical health: healthy.
• Edge remained up.

Do not repeat production assignments merely to write the report. Existing evidence is sufficient.

Registered trusted manifests

Registration was completed only after the final packaged E2E passed, using SSH server sec only.

Remote paths:

• /etc/truf/worker-packages/linux-worker-package-v2.json
• /etc/truf/worker-packages/windows-worker-package-v3.json

Final remote SHA-256 values match the accepted manifests:

• Linux: ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60
• Windows: e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c

Both are root:root mode 0644. Existing .pre-operator-experience backups were preserved unchanged. Upload temp files were removed. After registration, canonical runtime health succeeded and Docker reported truf-docker-runtime-1 healthy. No restart or config mutation was needed.

Test state

Completed checks:

• Worker API/local-state/supervisor focused suite: 100 passed.
• Worker package tests after ACL fix: 20 passed.
• Exact three watchdog timing tests after stabilization: 3 passed.
• Full packaged Windows/Linux E2E: passed, run 35f3f52e232067c1.
• Production health after final manifest registration: passed.

The broad focused matrix was run before the watchdog test timing patch:

python -B -m pytest tests/test_worker_api.py tests/test_worker_api_runtime.py tests/test_worker_assignment.py tests/test_worker_assignment_runner.py tests/test_worker_cli.py tests/test_worker_contracts.py tests/test_worker_local_state.py tests/test_worker_observability_db.py tests/test_worker_package.py tests/test_worker_runner_handoff_linux.py tests/test_worker_supervisor.py tests/test_remote_worker_db.py tests/test_scan_execution.py tests/test_admin_api.py -q

Result before the timing-only patch:

• 353 passed.
• 3 skipped.
• 2 false timing failures described above.

The two failures and the nearby equivalent test pass after the patch, but the complete 14-file command has not yet been rerun. This is the exact current stop point.

An unrestricted repository-wide pytest run is not a useful release gate in this checkout because unrelated private/generated assets and platform assumptions are absent. Its known baseline was 3031 passed, 134 skipped, 68 failed. Do not try to fix unrelated failures as part of this change. The focused change matrix, packaged E2E, production proof, and strict OpenSpec validation are the gates.

Immediate next actions

1. Rerun the exact 14-file focused matrix shown above. Expected result after the timing patch is 355 passed and 3 skipped. If it fails, diagnose only genuine worker-operator regressions; do not broaden scope.

2. Create the durable report: docs/worker-operator-experience-validation-2026-09-24.md.

3. In the report, include only sanitized aggregate evidence:

   • Scope and acceptance criteria.
   • Final Windows I/J and Linux G/H identities from this handoff.
   • Packaged E2E run 35f3f52e232067c1 and cleanup/foreign-state result.
   • Production issued/accepted/reconciled counts.
   • Duration percentiles derived from final-evidence.json.
   • Watchdog/full-stage evidence from progress-v3-evidence.json.
   • Natural timeout evidence from timeout-evidence.json.
   • Diagnostic/admin snapshot facts without private content.
   • Sanitized operator command transcript.
   • Known limits, especially no public registry/auto-updater and intentional abandoned-root retention.
   • Rollout and rollback/restoration facts, controls revision 126, and final healthy state.

4. Re-read docs/remote-worker-operations.md against task 6.1. It already covers package acquisition/build, Windows preparation, install/first run, lifecycle, status/attach/logs/history, phases/deadlines, diagnostics, drain/stop, recovery, update, and removal. Make only a minimal correction if the final artifact/report facts expose an actual gap.

5. Run strict validation:

   openspec validate add-worker-operator-experience --strict
   

6. If the focused matrix, report, runbook review, and strict validation pass, change only task checkboxes 6.1-6.4 in openspec/changes/add-worker-operator-experience/tasks.md from [ ] to [x].

7. Re-run openspec instructions apply --change "add-worker-operator-experience" --json and confirm progress 27/27 with state all_done.

8. Give the user a concise completion result and say the change is ready to archive. Do not archive it without an explicit request.

Report safety checklist

Before saving or quoting the final report, verify it contains none of:

• Tokens or credentials.
• Raw worker targets or findings.
• Runtime YAML or secret environment values.
• The private admin route prefix.
• Worker argv/auth command lines.
• Unbounded log or diagnostic bodies.

Allowed report content includes hashes, aggregate counts, reservation numeric IDs used as evidence references, phase names, durations/percentiles, safe test counts, generic command names, and public artifact paths within this workspace.