Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,78 @@
+# Current State
+
+Updated: 2026-09-26
+Workspace: `D:\truf-workers`
+
+## STATUS: Primary Objective Complete
+
+Extended production validation with exactly one native Windows worker slot and
+one WSL/Docker worker slot is complete. Run
+`e6ac8aec-ec20-4ba4-a924-abe5ee95d82c` exercised discovery, assignment, worker
+execution, progress, diagnostics, bundle upload and ingestion, projection,
+capacity release, and scheduled keycheck.
+
+The authoritative public-safe result is
+`docs/extended-live-validation-2026-09-26.md`. The machine-readable aggregate
+manifest is
+`build/extended-live-validation/runs/e6ac8aec-ec20-4ba4-a924-abe5ee95d82c/verification-manifest.json`.
+
+Conclusion: `pass_with_documented_deviations`.
+
+## STATUS: Terminal Validation Cut
+
+- Controller revision 143 was sealed with dispatch and discovery paused and the
+  validation drain complete.
+- All 314 reservations were terminal: 309 acknowledged and 5 refunded.
+- All 309 accepted bundles were ingested, projected, settled, and released.
+- No unresolved reservation, live assignment, pre-commit bundle, capacity use,
+  publication outbox item, quarantine row, waiting lock, or blocker remained.
+- All 348 projection jobs completed and released in one attempt without error.
+- The captured 39-candidate keycheck cohort completed, linked, projected, and
+  released; the global queue had no pending, leased, or deferred candidate.
+- Server and worker hash chains and all referenced objects verified.
+
+## STATUS: Production Restored
+
+Production restore completed through compare-and-swap revisions 143 to 146:
+
+1. Cancel the validation drain: 143 to 144.
+2. Resume discovery: 144 to 145.
+3. Resume dispatch: 145 to 146.
+
+Last verified state:
+
+- Controller revision 146, actor `validation-final-restore`.
+- Dispatch open, discovery open, drain normal.
+- Exactly one live assignment on Windows and one on WSL/Docker.
+- Runtime healthy with zero validation-run restarts.
+- Edge running with zero validation-run restarts.
+- Core services and discovery producers healthy.
+
+This is a recorded final snapshot, not a substitute for a fresh query in a
+future operational session.
+
+## STATUS: Documented Deviation
+
+The authenticated `recheck all` operation was accidentally used instead of
+rechecking only the two pending candidates, causing 35 broad provider checks
+contrary to `KEYCHECK-001`. All 35 were processed, none was skipped, the command
+returned success, and all effects settled. Do not repeat this probe.
+
+## STATUS: Remaining Defects
+
+- Windows GitLab filename-too-long checkout recovery is local and not deployed.
+- Invalid API-key classification is fixed locally and not deployed.
+- A roughly 20-second WSL clock-domain monotonic failure remains open.
+- Generic `WorkerContractError` diagnostics lose structured field detail.
+- Monitor aggregate queries produced six recovered statement timeouts under
+  load.
+
+Focused regression tests for the local checkout, long-path, and credential
+changes passed: 145 tests in 155.26 seconds.
+
+## NEXT: No Validation Work Pending
+
+Do not repeat the completed live validation or broad keycheck. Future work, if
+requested, is limited to reviewing and deploying the documented local fixes,
+investigating the remaining defects, and archiving the completed OpenSpec
+change. Do not archive OpenSpec without explicit user instruction.
@@ -0,0 +1,94 @@
+# Durable Decisions
+
+Updated: 2026-09-25
+
+## DECISION: RAW-001 - Raw Analysis Is Mandatory
+
+For analysis and live testing, it is forbidden to design or add token hiders,
+maskers, redactors, secret-scrubbers, post-hoc sanitizers, lossy summaries, or
+similar mechanisms that conceal the values being validated.
+
+- Inspect and retain the original raw payloads, targets, findings, credentials,
+  provider responses, diagnostics, logs, database rows, and worker records needed
+  to prove correctness.
+- The monitor must not replace, hash-only, truncate, partially reveal, or rewrite
+  a value before the analysis has consumed and preserved the raw record.
+- If a UI already has a reveal control, use the revealed raw value for the test;
+  do not build another concealment layer.
+- Raw evidence must remain in private local/protected storage and must not be
+  published to a public report, issue, commit, or chat excerpt. This storage rule
+  is not permission to hide data from the analysis itself.
+- A later public/operator report may reference counts and hashes, but it must be
+  derived only after raw correctness has been checked.
+
+This decision supersedes any old handoff wording that instructed the testing
+session to analyze only sanitized aggregates. Historical sanitized reports remain
+valid as reports; they are not sufficient evidence for the new run.
+
+## DECISION: ENV-001 - Production Target
+
+- Use the configured SSH server named `sec` only.
+- Never call, connect to, or mutate the configured server named `prod`.
+- Workspace is `D:\truf-workers`.
+- Do not run the inherited native runtime launchers in this source-only workspace.
+- Do not mount or mutate unrelated `D:\truf` runtime data.
+
+## DECISION: RUN-001 - Dual Worker Bounds
+
+- Native Windows: exactly one worker slot/thread.
+- Docker under WSL: exactly one worker slot/thread.
+- Expected maximum combined worker concurrency: two.
+- Do not increase caps or parallelism to accelerate the observation window.
+- Waits of up to ten minutes are allowed; several hours of observation are
+  explicitly authorized.
+
+## DECISION: KEYCHECK-001 - Scheduled Validation
+
+- Keycheck may be enabled for the run every 30 minutes (`1800` seconds).
+- Verify candidate leases, provider execution, append-only results, current-state
+  selection, projection jobs/appends, and capacity release from raw records.
+- Provider probes may have real external effects or cost; do not silently widen
+  service args or recheck policy beyond the active configuration.
+
+## DECISION: CONFIG-001 - Stale Candidate Must Not Be Applied
+
+Do not apply the stale config candidate with SHA-256
+`c0966cac4f7f0610a813fa8732e91953f2e3e838ad880f91fd1a9437096925c7`.
+It was based on an older active hash and would reduce
+`global.keycheck_queue_max_items` from the retained `8192` to `4096`.
+
+Always fetch the current active config identity and use the authenticated
+fresh-hash/CAS workflow for any 1800-second keycheck edit.
+
+## DECISION: ARCH-001 - Worker Authority
+
+- The worker is final authority for real provider access.
+- Server planning may bind immutable Git/Docker identity but must not add
+  per-target preflight/provider-access proof machinery.
+- Do not add credential sandboxes, environment rewriting, durable access proofs,
+  or security-specific infrastructure without a separate explicit user decision
+  and OpenSpec requirement.
+- Prefer bounded direct error classification. Authentication/access/not-found is
+  permanent when target-scoped; rate limits, network failures, and provider 5xx
+  are retryable.
+
+The complete engineering decision is in `AGENTS.md`.
+
+## DECISION: CHANGE-001 - Repository and OpenSpec
+
+- This repository has no baseline commit; the full tree appears untracked.
+  Never use Git to revert or clean files and never treat `git diff` as complete.
+- Preserve unrelated files and evidence directories.
+- `add-worker-operator-experience` is complete but must not be archived without
+  an explicit request.
+- Do not repeat the already completed 295-assignment production validation unless
+  a fresh verification proves its retained evidence invalid.
+
+## DECISION: CONTEXT-001 - Session Continuity
+
+- Use these files for continuation instead of recursive DCP summaries.
+- Do not proactively invoke conversation compression in the new session.
+- Batch searches and process large evidence in tools; avoid injecting raw
+  multi-megabyte files into the conversation context.
+- The prohibition on injecting large evidence into chat does not permit masking
+  or omitting it from the private analysis artifact.
@@ -0,0 +1,140 @@
+# Evidence Index
+
+Updated: 2026-09-26
+
+This file maps facts to their existing source. Do not duplicate the underlying
+evidence in handoff prose.
+
+## STATUS: Authoritative Reports
+
+- `docs/extended-live-validation-2026-09-26.md`
+  - Final public-safe report for the two-slot extended live run, including
+    settlement, keycheck, evidence integrity, deviation, restore, and open
+    defects.
+- `build/extended-live-validation/runs/e6ac8aec-ec20-4ba4-a924-abe5ee95d82c/verification-manifest.json`
+  - Machine-readable derived aggregates, classifications, evidence hashes,
+    terminal state, post-restore state, and test result.
+- `docs/worker-operator-experience-live-trace-2026-09-25.md`
+  - 295-assignment Windows/Linux cohort, aggregate outcomes, recovery, admin UI,
+    pipeline state, tests, and evidence hashes.
+- `docs/worker-operator-experience-validation-2026-09-24.md`
+  - Earlier bounded production/package acceptance and operator validation.
+- `docs/remote-worker-operations.md`
+  - Canonical worker install, lifecycle, diagnostics, drain, update, and removal.
+- `WORKER_OPERATOR_EXPERIENCE_HANDOFF.md`
+  - Historical implementation and reproducible artifact detail. Its stop point
+    predates the final live trace and defect-fix rollout.
+- `openspec/changes/add-worker-operator-experience/tasks.md`
+  - Current completion authority: all 27 tasks checked.
+
+## RAW: Private Live Evidence
+
+- `build/extended-live-validation/runs/e6ac8aec-ec20-4ba4-a924-abe5ee95d82c`
+  - Local run root with compact server artifacts and complete worker evidence.
+  - Server NDJSON SHA-256:
+    `a6beb1864c540fc5f22b2b647730a39e45dfddb10cf5ceff5e0181eb1a8f0bd8`.
+  - Worker NDJSON SHA-256:
+    `03a04a0da5a2db6bd02f2aaed41c191554b135ec287fbc1e2d9998d77da59898`.
+  - Server run SHA-256:
+    `5ae58df5f67a8d2a8d4e84d73262a6e7009e03dcbcc9ea176537b384d4e693c3`.
+- Private production server evidence root:
+  `/var/lib/docker/volumes/truf-remote-server-data/_data/extended-live-validation/e6ac8aec-ec20-4ba4-a924-abe5ee95d82c`
+  - Complete server evidence, approximately 1.2 GiB. Analyze in place and do not
+    copy raw values into public documents.
+- `build/live-trace-20260925/raw-evidence-final.json`
+  - Complete server cohort evidence. Historical recorded SHA-256:
+    `4071e38a1dec540663bc6febd9538ff54bedafa1627250933045beb8f7d09ec5`.
+- `build/live-trace-20260925/raw-evidence-expanded.json`
+  - Expanded unrestricted evidence used for defect diagnosis.
+- `build/live-trace-20260925/monitor-final.ndjson`
+  - Time-series server monitor. Historical SHA-256:
+    `e07507b0b8f0f86d1a1c7aade7186297c372b1ff177067bea19dfd219f5027a9`.
+- `build/live-trace-20260925/windows-localappdata/TRUF/RemoteWorker`
+  - Final retained Windows worker state, events, history, logs, and inactive
+    abandoned roots.
+- `build/live-trace-20260925/linux-worker-state-final.tar.gz`
+  - Final retained Linux worker state. Historical SHA-256:
+    `b64e81c90b232f46b400a63ed08f5660f46e34fedb1f67b64afba079d8d36364`.
+- `build/live-trace-20260925/dockerhub-discovery-db-raw.json`
+- `build/live-trace-20260925/dockerhub-unmasked-cycle-failure.json`
+- `build/live-trace-20260925/dockerhub-discovery.log`
+  - Raw DockerHub retry defect evidence.
+
+These files may contain sensitive raw values. Analyze them in place; do not copy
+their contents into a public document.
+
+## STATUS: Defects and Current Treatment
+
+- Extended-run open items:
+  - Windows GitLab filename-too-long checkout recovery is local and not
+    deployed.
+  - Invalid API-key classification is fixed locally and not deployed.
+  - A roughly 20-second WSL clock-domain monotonic failure remains open.
+  - Generic `WorkerContractError` diagnostics lose structured field detail.
+  - Six monitor aggregate-query statement timeouts recovered during the run.
+- `KEYCHECK-001` deviation:
+  - An unintended authenticated broad recheck processed 35 provider checks
+    instead of only the two pending candidates. All effects settled; do not
+    repeat this probe.
+- `docs/defect-windows-scan-timestamps-utc-2026-09-25.md`
+  - Still open. Local `app/scanner.py` continues to create naive timestamps.
+- `docs/defect-dockerhub-discovery-retry-null-type-2026-09-25.md`
+  - Original report says local-only. Current source has the PostgreSQL
+    `CAST(? AS TEXT)` correction and the defect-fix rollout included it. Reverify
+    live retry coalescing during the extended run.
+- `docs/defect-terminal-status-scan-deadline-readback-2026-09-25.md`
+  - Original report says open. Current source preserves durable deadlines with
+    `result.setdefault('deadlines', observability['deadlines'])`; included in the
+    defect-fix rollout. Reverify terminal readback.
+- `docs/defect-worker-network-oserror-mislabeled-local-io-2026-09-25.md`
+  - Original report says open. Current source introduces `WorkerNetworkError`
+    before broad `OSError` classification; included in the defect-fix worker
+    artifacts. Reverify under a real transport failure.
+
+## STATUS: Defect-Fix Rollout Artifacts
+
+- `build/runtime-defect-fixes-v1/deploy-runtime.sh`
+  - Atomic runtime/manifests cutover and rollback logic.
+- `build/runtime-defect-fixes-v1/windows-b.zip`
+- `build/runtime-defect-fixes-v1/windows-b.zip.json`
+- `build/runtime-defect-fixes-v1/linux-worker-package.json`
+- `build/runtime-defect-fixes-v1/verify-production-worker.py`
+- `build/runtime-defect-fixes-v1/restore-production-worker.py`
+
+Recorded identities:
+
+- Deployed runtime image:
+  `sha256:7d84fdf57a1cb9e6d38a571fbd3566b7549f1cda04ae02c4864cac70f74f2aaa`.
+- Current WSL worker image:
+  `sha256:491b3a2343571072209a7f92e83399fe206006dcccf247a0c551c50fc9f35e30`.
+- Rollback tag: `truf-local:runtime-pre-defect-fixes-v1`.
+- New registered manifest file hashes from the deploy script:
+  - Linux: `b9d3594e4846a21ca12de5fc6973c04d9eea6f61fd0ecda83875426aa48c7b4b`.
+  - Windows: `4cc97d17c34f7d89150927719f131f426f1068ab99af7f3f2f156d8642ae539e`.
+
+## STATUS: Historical Accepted Artifacts
+
+The pre-defect-fix live trace used:
+
+- Windows package manifest:
+  `78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6`.
+- Linux package identity:
+  `45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42`.
+- Linux image:
+  `sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3`.
+
+These remain valid historical evidence but are not the preferred artifacts for
+the new defect-fix observation run.
+
+## VERIFY: Fast Orientation Commands
+
+Run from `D:\truf-workers`:
+
+```powershell
+openspec list --json
+git status --short --branch
+python -B -m pytest tests/test_worker_api.py tests/test_worker_api_runtime.py tests/test_worker_assignment.py tests/test_worker_assignment_runner.py tests/test_worker_cli.py tests/test_worker_contracts.py tests/test_worker_local_state.py tests/test_worker_observability_db.py tests/test_worker_package.py tests/test_worker_runner_handoff_linux.py tests/test_worker_supervisor.py tests/test_remote_worker_db.py tests/test_scan_execution.py tests/test_admin_api.py -q
+```
+
+Do not use an unrestricted repository-wide pytest run as the release gate. Do
+not use Git clean/reset/checkout in this uncommitted snapshot.
@@ -0,0 +1,35 @@
+# Session Handoff Index
+
+Updated: 2026-09-26
+
+This directory is the authoritative entry point for a new session working on
+the live remote-worker validation. Read only these files first:
+
+1. `CURRENT_STATE.md` - where work stopped and the exact next actions.
+2. `DECISIONS.md` - binding user decisions and operational constraints.
+3. `EVIDENCE_INDEX.md` - existing reports, raw captures, artifacts, and hashes.
+
+The older root `WORKER_OPERATOR_EXPERIENCE_HANDOFF.md` is historical background.
+It remains useful for implementation detail and artifact provenance, but its
+"Immediate next actions" section is obsolete.
+
+## Knowledge Layout
+
+- A current fact has exactly one owner: `CURRENT_STATE.md`.
+- A durable rule has exactly one owner: `DECISIONS.md`.
+- Evidence is not copied into handoff prose; `EVIDENCE_INDEX.md` points to it.
+- Dated reports are immutable history. Record later corrections here instead of
+  rewriting the original report.
+- Replace stale current-state statements rather than appending contradictory
+  status paragraphs.
+
+Useful grep tags are `STATUS:`, `NEXT:`, `BLOCKER:`, `DECISION:`, `VERIFY:`, and
+`RAW:`.
+
+## New Session Start
+
+Use this prompt:
+
+> Read `docs/session-handoff/README.md` and its three linked files. Continue the
+> `NEXT:` work in `CURRENT_STATE.md` autonomously. Do not repeat completed live
+> validation. Follow every `DECISION:` literally, especially RAW-001 and ENV-001.