Initial server source import

openspec/changes/add-worker-operator-experience/design.md

@@ -0,0 +1,206 @@
+## Context
+
+The protocol-2 remote worker has proven its authoritative data path under real production load, including native Windows concurrency 3 and simultaneous Windows/WSL execution. Its operator experience has not reached the same level: the package exposes only `--server`, `--token`, and `--parallelism`; successful work is mostly silent; scanner output is captured out of view; slot state remains `assigned` during synchronous execution; and the server sees no phase progress between claim and result.
+
+Two production DockerHub assignments remained unresolved until the fixed 7,200-second assignment deadline even though the configured Docker target timeout was 600 seconds. The current watchdog terminates the owned TruffleHog process tree, but permit acquisition and surrounding Python work such as cleanup, filtering, result serialization, bundle staging, and handoff are not one hard-preemptible unit. Current evidence cannot identify which phase stalled.
+
+The administration page compounds the problem by labeling `result_reservations.last_error_code` as `Error category`. That field describes assignment transport failures and expiry, while accepted scan errors live in `target_scans`, `errors`, and result metadata and are not shown as assignment diagnostics.
+
+Legacy `supervisor.py --attach` demonstrates a useful interaction model: a verified background instance, an authenticated local control channel, a live status table, bounded logs, and detach without shutdown. The remote-worker package does not contain that supervisor and needs a smaller worker-specific implementation.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Deliver one coherent operator product rather than temporary UI over incomplete fields.
+- Give owners a cross-platform lifecycle CLI with detached operation, attach, status, logs, history, diagnostics, and doctor commands.
+- Define one phase/event model used locally, over the Worker API, in PostgreSQL, and in the admin UI.
+- Apply a hard local scan-stage deadline to the complete assignment execution unit, not only its scanner child process.
+- Preserve exact diagnostic material when available and explicitly describe size truncation or other transformations.
+- Separate assignment transport outcome, scan outcome, and diagnostics in storage and UI.
+- Make global and per-source assignment deadline policy visible and measurable with phase duration percentiles.
+- Ship a from-zero operator guide and fault-injection coverage as part of the same change.
+
+**Non-Goals:**
+
+- Replacing PostgreSQL assignment authority, immutable assignment expiry, durable receipts, or bundle ingestion.
+- Renewing assignment ownership from progress events.
+- Reporting fabricated percentage completion when the scanner has no reliable denominator.
+- Rebuilding the full server runtime supervisor inside the worker package.
+- Adding a temporary compatibility-only error page that will be removed after diagnostics land.
+- Adding a new diagnostic masking or redaction subsystem.
+
+## Decisions
+
+### 1. One worker supervisor owns lifecycle and authoritative local projections
+
+The package will expose a `truf-worker` command with `install`, `run`, `start`, `stop`, `status`, `attach`, `logs`, `history`, and `doctor` subcommands. `run` executes the supervisor in the foreground; `start` launches the same supervisor detached and waits for a startup handshake. Docker continues to run the supervisor in the foreground under Tini while Docker supplies detachment.
+
+The supervisor owns the singleton lock, slot controllers, local control endpoint, rotating human log, event JSONL, current status snapshot, and terminal local history. A verified instance record binds PID, process creation identity, executable, package identity, local control endpoint, and lifecycle state. `attach` reads status/events through the local control channel; it does not attach to arbitrary stdout and detaching never stops the worker.
+
+The current three-flag invocation remains a foreground-run migration alias for already shipped package launch definitions, but all new documentation and generated launchers use explicit subcommands.
+
+Alternative considered: add more prints to `remote_worker_client.py`. Rejected because prints cannot provide detached lifecycle control, reliable concurrent-slot rendering, machine output, history, or process identity.
+
+### 2. A single append-only event model drives every projection
+
+Each state transition emits a versioned event with a monotonic local sequence:
+
+```json
+{
+  "schema": 1,
+  "sequence": 42,
+  "timestamp": "2026-09-23T12:00:00Z",
+  "instance_id": "...",
+  "slot_id": 0,
+  "reservation_id": 123,
+  "source": "dockerhub",
+  "type": "slot.phase",
+  "phase": "scanning",
+  "phase_started_at": "...",
+  "scan_deadline_at": "...",
+  "assignment_deadline_at": "...",
+  "progress": {}
+}
+```
+
+Canonical phases are `idle`, `claiming`, `assigned`, `waiting_permit`, `preparing`, `resolving`, `downloading`, `cloning`, `scanning`, `filtering`, `cleaning`, `bundling`, `uploading`, `awaiting_receipt`, `backoff`, `draining`, and `stopped`.
+
+The local status JSON is a rebuildable projection of the event stream, not a second independently written state machine. Server progress uses the same event schema with a server-assigned receive timestamp. Terminal history records authoritative receipt/prebundle/stale outcomes plus durations.
+
+Alternative considered: define separate local, API, and admin status shapes. Rejected because they would drift and recreate the current ambiguity.
+
+### 3. Execute each scan stage in a supervised assignment runner process
+
+Threads in the controller continue to own claim/recovery/upload state, but scanner execution moves into a package-local assignment runner subprocess. The controller writes one validated input record and starts the runner in a contained process tree with a dedicated work/output directory.
+
+The hard scan-stage deadline starts before permit acquisition and covers:
+
+- permit wait;
+- assignment preparation and source resolution performed locally;
+- downloads/clones;
+- scanner subprocess execution;
+- filtering and result conversion;
+- cleanup;
+- result and bundle staging.
+
+The runner emits phase events over a local pipe/file protocol. At the deadline the controller terminates the runner process tree, atomically detaches its work directory for later janitor handling, and creates a normal timeout result bundle with the phase and diagnostic envelope. The controller then uploads that result while the immutable assignment deadline still permits it.
+
+Normal cleanup remains cooperative. Cleanup that exceeds the deadline cannot keep the slot occupied; abandoned work is moved into a janitor-owned tree and reported in status.
+
+Alternative considered: add deadline checks around existing Python calls. Rejected because blocking Python/filesystem/library calls cannot be hard-preempted reliably in the controller process.
+
+### 4. Progress events are informational and never renew authority
+
+The Worker API gains an authenticated progress endpoint accepting ordered phase events for the worker's current reservation. It validates reservation/device ownership and monotonically advances the latest accepted event sequence. Duplicate events are idempotent.
+
+Progress updates do not change `remote_expires_at`, queue leases, bundle capacity, or receipt authority. Failure to send progress does not abort a healthy local scan; events remain in the local journal and retry with bounded backoff. The server can therefore display `last phase` and `last progress` without turning progress into a lease heartbeat.
+
+Alternative considered: renewable leases. Rejected because a stuck client could retain work indefinitely and because the fixed-expiry fencing model is already proven.
+
+### 5. Server-owned deadlines support global fallback and per-source policy
+
+`supervisor.worker_api.assignment_ttl_seconds` remains the global fallback. A managed per-source override map adds GitLab, DockerHub, and HuggingFace assignment TTL values. The server selects and commits the immutable deadline when issuing an assignment and includes the effective scan and assignment deadlines in the response.
+
+The editor labels these separately as `Target scan timeout`, `Assignment deadline (end-to-end)`, and `Result upload body deadline`. Validation retains absolute bounds and checks that each effective assignment deadline covers its source scan timeout, upload deadline, and handoff margin.
+
+The server aggregates phase and end-to-end durations by source and outcome as p50, p95, and p99. Configuration remains explicit; metrics inform changes but do not silently rewrite policy.
+
+Alternative considered: let each worker select or renew its TTL. Rejected because workload policy belongs to the assigning server and must be consistent for queue fencing.
+
+### 6. One diagnostic envelope spans scan and prebundle failures
+
+Diagnostics use one versioned envelope with indexed dimensions and exact optional payloads:
+
+- diagnostic ID and schema;
+- reservation, scan event, slot, attempt, and source;
+- phase, kind, category, stable code, summary, and retryable disposition;
+- provider operation and HTTP status/content type/request ID;
+- process name, exit code, signal, and timeout state;
+- exception type/message/fingerprint;
+- raw body material;
+- log stream head/tail material;
+- occurred/captured/received timestamps;
+- original byte count, stored byte count, content hash, and truncation state.
+
+Categories are broad query dimensions such as `authorization`, `rate_limit`, `not_found`, `network`, `timeout`, `scanner`, `storage`, `protocol`, and `assignment_expired`. Stable codes express the concrete cause, such as `docker.manifest_http_403` or `trufflehog.exit_nonzero`. Phase is independent of category.
+
+Text/bytes are preserved as captured. Non-text bodies use an explicit encoding field. Storage bounds are deterministic: body excerpt 16 KiB, combined log head/tail 32 KiB, one transmitted envelope 64 KiB, at most 32 diagnostics and 256 KiB per assignment. Metadata records every truncation; no truncation is presented as a complete body.
+
+Accepted result bundles gain a diagnostic frame. Prebundle reports carry the same envelope under the existing JSON body limit. The old E-frame error string remains ingestible during rollout but is projected into the new model exactly once.
+
+Alternative considered: keep scanner error strings, reservation error codes, and source metadata as separate taxonomies. Rejected because operators cannot correlate or filter them consistently.
+
+### 7. Local diagnostics retain complete operator evidence when available
+
+The supervisor writes:
+
+```text
+control/worker.instance.json
+control/worker.status.json
+events/worker-events.jsonl
+history/worker-history.jsonl
+diagnostics/YYYY-MM-DD/<reservation>/<diagnostic>.json
+diagnostics/YYYY-MM-DD/<reservation>/<diagnostic>.body
+diagnostics/YYYY-MM-DD/<reservation>/<diagnostic>.log
+logs/worker.log
+```
+
+The JSON envelope points to optional body/log files and records their hashes and sizes. Local retention is configurable by age and total bytes and is reported by `status` and `doctor`. Rotation never mutates terminal history entries; it changes attached-artifact availability explicitly.
+
+Alternative considered: store every raw artifact directly in one JSONL. Rejected because large multiline/process output makes append recovery and bounded tailing expensive.
+
+### 8. PostgreSQL stores diagnostics as first-class records
+
+Add `worker_diagnostics` with an idempotent diagnostic UID, reservation FK, optional target-scan FK, indexed phase/category/code/kind/retryable columns, summary, canonical envelope JSON, received timestamp, and optional bounded body/log payload columns. Bundle ingestion writes diagnostics in the same transaction as the target scan and error projection. Prebundle diagnostics attach to the reservation before a target scan exists.
+
+Existing `errors` rows remain the compatibility scan-error projection. Existing `last_error_code` is retained as assignment failure code but is no longer labeled as the complete error category.
+
+Alternative considered: put all envelopes only into `metadata_json`. Rejected because filtering, detail lookup, idempotency, and prebundle diagnostics require first-class rows.
+
+### 9. Admin views assignment outcome, scan outcome, progress, and diagnostics separately
+
+The worker assignment table presents:
+
+- assignment outcome: unfinished, accepted, prebundle failed, expired;
+- scan outcome: clean, found, degraded, error, skipped, or not available;
+- diagnostic count and highest-priority category/code;
+- current/latest phase, phase age, assignment deadline, and last progress age;
+- worker/device/package identity and slot where available.
+
+Each row links to a detail page with an event timeline, duration breakdown, transport/receipt data, scan summary, diagnostics, raw body/log tabs, canonical JSON copy/download, and explicit truncation metadata. Filters operate independently on assignment outcome, scan outcome, source, phase, category, code, retryability, and time.
+
+Alternative considered: make the existing `Error category` cell open a modal. Rejected because the list model itself conflates transport and scan semantics.
+
+### 10. Human output and machine output are equal product contracts
+
+Human status/attach uses a stable table and event stream. It displays elapsed time, configured scan deadline, assignment time remaining, last progress age, child state, and trustworthy counters. It never fabricates completion percentages.
+
+`--json` commands emit one versioned JSON document. Follow modes emit NDJSON with one event per line and no decorative output. Tests treat both output forms as contracts.
+
+## Risks / Trade-offs
+
+- [Runner process split touches scanner lifecycle and recovery paths] -> Introduce it behind the same `execute_protocol2_remote_claim` contract, preserve deterministic bundle validation, and fault-test every boundary before replacing in-process execution.
+- [Progress traffic increases database writes] -> Persist only monotonic phase transitions and coarse progress changes, deduplicate by reservation/sequence, and keep high-frequency local samples local.
+- [Diagnostic payloads increase bundle and database volume] -> Enforce deterministic per-item/per-assignment byte and count limits, expose truncation metadata, and track storage usage.
+- [One broad change can take too long] -> Implement as large vertical chunks that each finish a final architecture slice; do not ship throwaway status/error models.
+- [Per-source policy adds configuration complexity] -> Keep one global fallback, explicit source overrides, editor-derived effective values, and validation based on existing scan/upload settings.
+- [Local full-stage termination can leave work trees] -> Atomically detach them to janitor ownership and surface retained bytes/counts in status and doctor.
+- [Old packages do not emit progress/diagnostics] -> Admin renders legacy records from existing fields and marks phase/diagnostic availability explicitly until packages are upgraded.
+
+## Migration Plan
+
+1. Add schema/event/diagnostic libraries, PostgreSQL tables, and read paths without changing current assignment execution.
+2. Add the worker supervisor CLI and local projections while the current foreground invocation remains a migration alias.
+3. Add assignment runner execution, full-stage watchdog, local phase events, and fault-injection tests.
+4. Add Worker API progress and diagnostic transport, then enable server persistence and detail queries.
+5. Replace the worker admin list/detail presentation and add percentile/deadline editor views.
+6. Rebuild Windows/Linux packages, run protocol compatibility tests, then run bounded Windows and WSL production validation.
+7. Update generated launchers and the from-zero operator guide; migrate the production worker launch definition to explicit `run`.
+8. Remove the migration alias only in a separately declared breaking change after all known deployments use subcommands.
+
+Rollback disables progress ingestion and runner selection while retaining additive event/diagnostic tables. Existing immutable assignment, bundle, receipt, and legacy E-frame paths remain authoritative throughout rollout.
+
+## Open Questions
+
+No blocking product questions remain. Exact local retention defaults and percentile windows can be selected from implementation benchmarks without changing the external contracts above.