Initial server source import

openspec/changes/add-minimal-remote-scan-workers/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-17

openspec/changes/add-minimal-remote-scan-workers/baseline-evidence.md

@@ -0,0 +1,128 @@
+# Task 1.3 Baseline Evidence
+
+## Status
+
+This file records the best available historical baseline for task 1.3 and its
+reproducibility limits. It does not claim that a pre-change image was rerun during
+this change, and it does not convert current tests into pre-change evidence.
+
+An immutable rerun of the exact pre-change source and image is unavailable. On
+2026-09-18, the reviewer explicitly accepted this documentary baseline and waived
+that rerun requirement. Task 1.3 therefore relies on the historical record below;
+current regression results remain separately identified as post-change evidence.
+
+## Historical pre-change record
+
+The repository's [`DOCKER_MIGRATION.md`](../../../DOCKER_MIGRATION.md), under
+"Current Verified Evidence," records a final run on 2026-09-15. It reports:
+
+- The selected container regression suite completed with **578 passed** and
+  **7 expected Windows-only tests skipped on Linux**, with no failures recorded.
+- The selected suite used test image
+  `sha256:4ce11325728ba3e58e6643c1c8e800f317179d5c7c50e7e80568b58f62dbdfd0`.
+- The separate six-test `docker/test_verify.py` suite passed on Windows and WSL
+  Linux; those six tests were not included in the 578 count.
+- The recorded offline E2E passed its result/check gates and removed its owned
+  resources. This is contextual historical evidence, not a rerun for task 1.3.
+
+The `add-minimal-remote-scan-workers` OpenSpec change was created on 2026-09-17,
+after that recorded run. The preserved records contain no pre-existing failure
+for the cited 578-test selection to list separately. "No recorded failure" means
+only that no failure record was found; it is not proof that no unrecorded attempt
+failed.
+
+## Reproducibility limits
+
+- The old test image is unavailable and was not retrieved, rebuilt, or executed.
+  Current Docker runs use new, dedicated test image identities.
+- This workspace has no commit history. On 2026-09-18, `git status` reported
+  `No commits yet on master` and every repository path as untracked; `git log`
+  failed because the branch has no commits.
+- `WORKSPACE.md` names a source-side provenance commit, but also states that this
+  source-only snapshot includes modified and selected untracked files and did not
+  copy source Git history. It is not an immutable tree for either 2026-09-15 or
+  the moment immediately before the 2026-09-17 OpenSpec change.
+- The image digest and prose result are therefore useful historical evidence but
+  cannot independently reconstruct or rerun the claimed chronology from this
+  repository.
+
+## Current expanded selection
+
+The current reviewed allowlist is the `SELECTION` mapping in
+`tests/container_unit.py`. On 2026-09-18, its stdlib-only selection check reported
+**33 modules and 649 test definitions**, with no runner skips at declaration time;
+pytest parametrization may expand the executed count.
+
+The current mapping selects:
+
+```text
+test_docker_foundation.py
+test_container_runtime.py
+test_owned_process.py
+test_owned_process_linux.py
+test_owned_process_boundary.py
+test_runtime_bootstrap_authority.py
+test_supervisor_foreground_shutdown.py
+test_supervisor_startup_rollback.py
+test_supervisor_managed_postgres_gate.py
+test_observer_only_coordinated_shutdown.py
+test_supervisor_safety.py
+test_postgres_runtime.py
+test_container_security.py
+test_runtime_security.py
+test_postgres_empty_initialization.py
+test_container_migration_paths.py
+test_container_provider_portability.py
+test_container_e2e_helpers.py
+test_container_import.py
+test_container_import_config.py
+test_container_projection_recovery.py
+test_result_bundle_v2.py
+test_pipeline_cutover_invariants.py
+test_custom_provider_detector_compatibility.py
+test_scan_execution.py
+test_synthetic_llm_pipeline.py
+test_worker_api.py
+test_worker_api_runtime.py
+test_worker_assignment.py
+test_worker_package.py
+test_remote_worker_db.py
+test_admin_api.py
+test_edge_deployment.py
+```
+
+The remote-worker, package, administration, edge, synthetic-pipeline, and bundle
+modules in this current selection postdate the historical baseline. Their
+presence demonstrates current review scope, not pre-change execution.
+
+## Current post-change evidence
+
+The expanded selection and isolated end-to-end gates were rerun on 2026-09-19.
+They validate the completed implementation but do not replace the historical
+pre-change baseline:
+
+```text
+python -I -S -B tests/container_unit.py --check-selection
+container-unit: AST OK; 33 modules, 650 test definitions, no runner skips (parametrizations expand in pytest)
+
+isolated Linux container selection
+867 passed, 7 expected platform skips
+
+wsl.exe -d Ubuntu-24.04 -- python3 docker/verify.py
+E2E passed; project truf-worker-test-99b193a64f46a0002e34da9c5ff8029a; artifacts removed
+
+python -B docker/verify_packaged_workers.py ...
+Windows/Linux packaged-worker E2E passed; run 348d24046fb7b8d4; cleanup complete; foreign Docker state unchanged
+Windows package manifest sha256 f0bbbbf79d9f5f3f17440a60561b307fa7bafd312ab6110b14098ff90755f8e6
+Linux worker image manifest list sha256 888bffc5519fd3a27cb52d20eaea1143f89ac9779bb2e4b0d998e450583c57a0
+
+python -B docker/verify_edge_e2e.py --timeout-seconds 1200
+Edge/fail2ban E2E passed; run 597b3ff7826055e1; cleanup complete; foreign Docker state unchanged
+
+openspec validate add-minimal-remote-scan-workers --strict --no-interactive
+Change 'add-minimal-remote-scan-workers' is valid
+```
+
+The current runs used only random or dedicated test-owned identities and verified
+that foreign container and volume metadata remained unchanged. No current result
+is represented as historical or pre-change evidence.

openspec/changes/add-minimal-remote-scan-workers/design.md

@@ -0,0 +1,147 @@
+## Context
+
+The current PostgreSQL pipeline already has single-target admission, capacity reservation, immutable Git/Docker plans, scan/error classification, canonical v2 `.trb` staging, transactional ingestion, projection, and detailed keychecks. The execution seam is the claim-to-`stage_claim` path in `app/console_runner.py`, not a replacement scheduler.
+
+`scan_target_result()` in `app/scanner.py` dispatches existing source implementations. Bundle staging extracts candidates, including structured Postman evidence; ingestion persists findings/candidates and separately schedules projection and keycheck. JSONL projection is not a prerequisite for keycheck, and a TruffleHog `Verified` field is not a completed detailed keycheck.
+
+Current execution is not automatically portable to a DB-free client: process launch authority, exact Git/Docker plans, reservation accounting, and producer recovery depend on the local runtime. Recovery can inspect local PID/executable identity and local files, which cannot establish remote worker liveness. The design adapts these boundaries while keeping scanner/provider/error behavior intact.
+
+Development takes place in `D:\truf-workers`, a source-only snapshot of the current `D:\truf-docker` working tree, including uncommitted fixes. Production data and Git history were not copied. Existing unarchived specifications are references; `openspec/specs/` has no canonical baseline. Old three-global-permit/`S:` handoff assumptions apply to their historical local deployment, not this remote boundary. PostgreSQL remains authoritative despite older status-file accounting descriptions.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Move expensive download/scan work to trusted Windows/Linux clients with minimum new code and persistent state.
+- Preserve scanner results, attribution, exact-plan coverage, error classification, retry decisions, candidate routing, and detailed server-side keychecks.
+- Keep source/provider/scanner configuration centralized; support client-selected N slots capped by the server across a user's devices.
+- Recover through a fixed 24-hour default assignment deadline, durable retries, and existing identity fencing, without heartbeat traffic.
+- Secure the small public surface and test locally against empty storage and synthetic inputs.
+
+**Non-Goals:**
+- Client detailed keychecks, a second broker/queue/result format, generic job infrastructure, batch claims, worker affinity, or worker blacklists.
+- New scan retry counts, a three-attempt/dead-letter rule, exactly-once physical execution, or heartbeat/lease renewal protocols.
+- Client disk encryption, mTLS, public/untrusted workers, auto-update, complex roles, multiple server replicas, or PostgreSQL container extraction.
+- Production database import, production deployment, changes to the active runtime, or archiving unrelated OpenSpec changes.
+
+## Decisions
+
+### 1. Keep one runtime and add a thin remote boundary
+
+```text
+Internet -> Caddy :443
+               |-- /api/v1/worker/*   [device token] -> Worker API
+               `-- /<random-admin>/* [login/password] -> micro-admin
+
+Server runtime: PostgreSQL + existing queue/reservations
+                discovery/scheduling -> admission
+                receiver -> existing ingester -> projection
+                                             `-> detailed keycheck
+
+Client slot: claim -> existing download/scan -> canonical .trb -> upload/ack
+```
+
+Run Worker API and micro-admin as runtime-managed processes, not independent database-owning services. Keep the current dashboard backend private; any dashboard information in the admin area shares its authentication boundary. PostgreSQL, supervisor control, Caddy's control API, and raw backend ports have no public host bindings.
+
+Reuse `reserve_and_claim_target`, ambiguous-admission reconciliation, `scan_target_result`, bundle staging/validation, `mark_result_bundle_ready`, and ingester/projector/keycheck paths. Extract only the code needed to run one already-planned scan and build its complete bundle without PostgreSQL. Adapt DB-bound plan inputs and node-local process authority rather than giving a client a DSN or a supervisor credential. Preserve `OwnedProcess` containment and cleanup; the client must launch only the known scanner tools, not arbitrary server-supplied commands.
+
+Alternative rejected: copying `run_cycle_v2` unchanged or implementing a second scanner/queue. Both either retain server authority on clients or create divergent behavior.
+
+### 2. Central configuration, minimal device identity, compatible jobs
+
+Client-authored operational configuration has only server URL, opaque device token, and desired positive slot count N. Generated local pending-work state and workspace paths are not independent source configuration. The server resolves source/scanner settings, immutable plan, limits, and only the credentials required for this task. It must not send the whole config, discovery credential pool, database credentials, admin secrets, or control authority.
+
+Bind each device token to its owning user and a server-issued/stable worker identity; store token hashes and support revocation. Keep only the identity/quota metadata required for administration, bound to current reservations rather than creating `remote_jobs` or another scheduler. Server configuration supplies default limits; admin changes the per-user cap across all devices.
+
+Send protocol/build/policy compatibility metadata with normal claim traffic, not a background registration/liveness service. Validate the pinned scanner/custom detector policy and supported source execution on the client's OS/architecture before consuming a target. Pass a config snapshot or stable effective-config identity so the result stays attributable even if the server config changes mid-task. A client rejects an incompatible job instead of silently using its own settings.
+
+Alternative rejected: worker-owned provider settings/tokens or another config management system. Trusted clients receive the task-specific secrets they need over HTTPS; this is not a sandbox against a compromised client.
+
+### 3. One claim per free slot with existing backpressure
+
+Each free client slot requests one assignment. Effective admission is bounded by N locally, the atomic per-user server cap across devices, and existing source/global/output-capacity rules. A client advertising N=8 does not override an admin cap of 3. Concurrent claims from different devices must not overshoot a shared cap. Reducing a cap stops new admissions until usage falls below it; it does not invent cancellation semantics for already-issued work.
+
+Keep the current node-local scan-permit mechanics. A client work slot holds its assignment through durable result acknowledgement, which bounds pending uploads when the server is unavailable. Release the native scan permit at its existing handoff point; do not conflate that permit with remote user quota. A durably accepted bundle, acknowledged pre-bundle terminal disposition, or completed expiry recovery releases remote assignment quota exactly once; server spool credits remain governed by existing ingestion accounting.
+
+Persist the existing admission/request identity before the first claim request. Retry ambiguous requests with the same identity and reconcile the same reservation; do not allocate another task because a claim response was lost. Scope recovery to the authenticated device. Empty queues or denied capacity return a bounded polling delay, not a scan error or a new queue item.
+
+Alternative rejected: batches and client backlogs. Per-free-slot claims reuse current scheduling and avoid another recovery structure.
+
+### 4. Fixed expiry, no remote heartbeat
+
+Set `expires_at` from the server clock at assignment commit, with a configurable duration defaulting to 24 hours. The deadline includes download, scan, and upload. Ordinary API contact, retries, and client restarts do not renew it. Internal scanner timeouts and server discovery leases keep their existing meanings; local slot bookkeeping is not a worker heartbeat.
+
+Use one periodic runtime recovery pass for expired remote assignments, initially once per minute and configurable. Extend existing recovery ownership/state checks instead of inventing a remote liveness monitor. Reconcile reservations, target claims, output credits, and dependent exact-plan/blob leases together. Local producer-PID recovery must not release remote claims. No associated lease may silently expire earlier and allow a second owner while the remote assignment remains valid.
+
+Requeue unfinished expired work using the existing pre-handoff infrastructure-loss/refund path where applicable, not a fabricated scanner/provider failure or a new retry limit. The same worker may claim the task again. New issuance has a distinct current reservation/attempt identity. Both periodic recovery and result acceptance use the same atomic ownership/deadline boundary.
+
+Reject an unaccepted old result once its assignment expires or is superseded; it must not finish newer work, update plan coverage, or return somebody else's credit. A retry of an already accepted identical result still receives its original acknowledgement even after the deadline. Ready/accepted bundles awaiting ingestion are not unfinished worker assignments and must not be expired back into the queue.
+
+Trade-off accepted: a crashed client can delay work for about a day, and a partitioned client can keep physically scanning after reissue. Fencing gives one authoritative acceptance, not exactly-once physical execution.
+
+### 5. Reuse the full bundle and durable handoff
+
+The API needs only claim/reconcile, upload/acknowledge, and the existing pre-bundle failure/release outcome where necessary. Exact URL naming is implementation detail under `/api/v1/worker/`; do not add a heartbeat endpoint, remote shell, separate keycheck jobs, or a general command API.
+
+Pre-bundle terminal reports use the same issuance fencing and replay rules as results. Persist/retry their identity until acknowledged; a lost reply resolves to the original disposition without charging retries, updating counters, or releasing quota/credits again. A delayed report from an expired/superseded issuance cannot mutate a new issuance even when the same device owns both. Authoritative acknowledgement resolves that client slot exactly once.
+
+Send canonical v2 `.trb` bytes containing the existing findings, detector identities, source/origin/context, errors, metadata, candidate evidence, and exact-plan identity. Preserve structured Postman candidate extraction before ingestion. Do not send only provider names or reconstruct a reduced result JSON. Keep detailed keychecks entirely server-side after ingestion through current candidate/dedup/cache/routing logic. Retained `runtime/keychecks` outputs keep their existing location and retention rules.
+
+Upload a binary stream, not base64 JSON, into a bounded server-owned partial file. Enforce the existing bundle limit/capacity reservation (currently 192 MiB where configured), time limits, identity, version, and hash. Validate all paths/archive members with the existing codec; clients cannot choose arbitrary server paths. Publish durably using the existing same-filesystem atomic handoff and mark the exact reservation ready before acknowledging server custody.
+
+Acknowledgement means the validated bundle and its ready/recovery state survive a server restart; it does not mean projection or detailed keycheck has finished. An identical retry returns the same receipt without double ingestion/accounting. A conflicting body for an accepted identity is rejected. Interrupted uploads never count as successful scans; full-bundle retry is sufficient for the MVP, with no resumable-upload protocol.
+
+Keep the accepted identity, digest, and reconciliation outcome in existing authoritative records independently of the spool file. Ordinary ingestion and bundle cleanup must not erase the information needed to recover a receipt: accept-with-lost-reply, ingest, clean up, restart, and retry after the deadline must still return the original acceptance for identical bytes and reject conflicting bytes. No second receipt queue or result store is required.
+
+Keep the pending bundle and its assignment identity locally until durable acceptance is confirmed. After acknowledgement, existing cleanup may delete local task artifacts. A definitive fenced rejection transitions local work to an explicit stale/discard outcome with bounded cleanup, not an infinite retry or a false success. Scan errors still use the existing disposition path; HTTP retry/backoff does not consume scanner/provider retry budgets.
+
+Crash recovery must cover publication-before-ready and ready-before-response windows using exact current ownership and existing spool reconciliation. Never infer success from the mere presence of an unvalidated file or expire already accepted work because a producer PID is absent.
+
+### 6. Restricted edge and separate admin login bans
+
+Use direct DNS to Caddy for the initial deployment and HTTPS for all worker/admin transport. The public admin prefix contains at least 128 bits of randomness, but login/password remains mandatory for every administrative route and asset. Caddy password authentication with a supported password hash is the minimal starting point; no plaintext password configuration or public signup. Protect typed mutating actions against CSRF with validated Origin/CSRF handling. No generic supervisor-command passthrough.
+
+Only authenticated admin pages show operational summaries or typed user/device-token/quota and existing queue controls. Keep the standalone `/dashboard` route closed. Unknown paths return 404. Add no-store/same-origin-referrer, a restrictive compatible CSP, and production HSTS; avoid third-party assets and secrets/raw findings in diagnostics.
+
+Run fail2ban on the host, reading redacted Caddy authentication-failure events. Two actual bad-credential submissions to the admin boundary within ten minutes produce a 24-hour IP ban. Do not count an ordinary initial Basic-auth challenge without credentials, unrelated 404s, or Worker API authentication failures. With shared HTTPS ingress, the ban action must update an ADMIN-ONLY Caddy IP deny matcher, with validated configuration reload, not globally drop that address on port 443. Persist ban state and provide SSH unban/recovery.
+
+Use the direct connection address as client IP. Ignore arbitrary forwarded IP headers; a future trusted proxy requires explicit trust configuration and new tests. If network-level fail2ban actions are ever chosen instead, prove Docker forwarding-chain enforcement and preserve worker availability; a blanket host INPUT rule does not satisfy the contract.
+
+Alternative rejected: a secret URL as sole protection, separate dashboard exposure, or a shared admin/worker IP jail. No extra auth service, mTLS, Cloudflare dependency, or client-disk encryption is required.
+
+### 7. Observability from existing state
+
+Correlate source/target, user/worker, reservation/issuance, issue/finish time, duration, outcome, and safe error category. Expose per-worker unfinished, completed, failed, and expired counts and last authenticated API contact. Distinguish bundle acceptance from later ingestion/keycheck completion and define counters from existing authoritative transitions so duplicate uploads do not inflate them.
+
+Record issuance, acceptance, expiry/requeue, duplicate upload, and stale-result rejection events using existing logs/records. Never label a worker online/offline merely from silence: there is no heartbeat. Do not introduce a telemetry database or monitoring stack. Raw findings belong in the existing protected result storage, not transport/auth/debug logs.
+
+### 8. Empty and isolated local validation first
+
+Keep production checkout, processes, Docker containers, and volumes untouched. Reuse the existing stdlib checks, reviewed unit harness, and synthetic E2E driver. Before any runtime test, make test project names, image tags, volumes, ports, and config independent of inherited deployment defaults. The copied `compose.yaml`/legacy import overrides are not safe test launch instructions.
+
+Initialize PostgreSQL empty in new test-owned storage; never restore a production dump or mount production PGDATA. Scrub inherited DSNs, provider secrets, runtime overrides, and proxies before application imports. Disable live discovery/keycheck autostart; explicitly feed synthetic tasks and mock detailed provider transports. No production credentials, paid calls, discovered public targets, or uncontrolled TruffleHog verification.
+
+Exercise existing local and new remote scan execution on the same fixtures and compare normalized bundle/DB results while ignoring only transport identity/timing. Include Git/Docker immutable-plan and Postman-candidate parity, both custom detector directions, and all current source error dispositions. Test Windows and Linux clients, N>1 and multi-device caps, fixed-clock expiry/reissue races, lost claim/result replies, restarts, malformed/conflicting bundles, and auth isolation. Use controlled clocks instead of waiting a real day.
+
+Use an isolated local TLS/internal network for Caddy/API tests and deny external egress. Validate admin bans through the actual edge, including a worker sharing the banned admin IP. Package a pinned Windows portable client and Linux client/container using current tool dependencies; no installer service or updater is necessary. Do not claim cross-platform readiness from mocked scans alone.
+
+## Risks / Trade-offs
+
+- [Long assignment lifetime] Slow recovery and duplicate physical work are deliberate trade-offs; fencing and idempotent acceptance protect authoritative state.
+- [DB-bound execution seams] Git/Docker plans and process authority require adaptation; parity tests are a release gate, not permission to rewrite provider logic.
+- [Trusted client access] Clients can see task credentials and findings; issue least-needed credentials, redact logs, revoke device tokens, and use HTTPS.
+- [Shared NAT and aggressive admin bans] Two mistakes can lock out an operator; scope bans to admin and retain tested SSH recovery.
+- [Existing active OpenSpec deltas] Avoid mixing unrelated changes or reviving superseded local-only assumptions; archive/sync is a separate request.
+- [Unsafe inherited launch defaults] Static checks can run now; runtime/E2E execution waits for explicit test isolation and image/config provenance checks.
+
+## Migration Plan
+
+1. Prepare the separate source-only Git workspace and complete this plan; do not start runtime services.
+2. Establish neutral isolated tests, then implement the narrow execution/admission/transport boundary against empty PostgreSQL with synthetic fixtures.
+3. Run local parity, failure/recovery, cross-platform, and edge-auth gates before enabling any real remote claims.
+4. Keep remote admission disabled by default until explicitly configured. Preserve the local execution path using shared scanner logic; do not require a remote worker for existing local operation.
+5. Production deployment and any additive identity/reservation schema migration need a separate reviewed rollout. No production data migration or rebuild is performed during this planning task.
+6. To roll back a later rollout, stop new remote claims, retain/reconcile accepted bundles, and drain or expire outstanding assignments through current recovery before returning to local-only execution. Do not blindly drop reservation metadata or switch binaries underneath unfinished remote work.
+
+## Open Questions
+
+No blocking product decisions remain. Implementation must verify the smallest DB-free Git/Docker execution seam, the exact existing reservation fields to extend, and the host-specific fail2ban/Caddy reload mechanism through tests before release. Deployment-specific hostname, generated admin prefix/password, device tokens, and quota values are supplied during provisioning, not embedded in this plan.

openspec/changes/add-minimal-remote-scan-workers/proposal.md

@@ -0,0 +1,29 @@
+## Why
+
+Truf needs to use trusted Windows and Linux machines for download/scan work without rewriting its already-debugged parser, queue, error policy, ingestion, or detailed keychecks. A separate source-only workspace and an empty, isolated local test environment let us develop this boundary without copying the large production database or touching the active runtime.
+
+## What Changes
+
+- Add a thin authenticated HTTPS adapter around existing target reservation and canonical `.trb` result handoff, not a second task system.
+- Keep PostgreSQL, discovery, scheduling, ingestion, projection, and detailed keycheck in one server Docker runtime; use a separate Caddy edge.
+- Run existing download/scan logic on trusted clients. The server supplies the task, immutable plan, and required source/scanner settings; the client config contains server URL, device token, and desired execution slots.
+- Claim one task per free client slot, subject to an atomic server-side per-user cap across devices and existing admission/backpressure rules.
+- Use a configurable fixed assignment lifetime, default 24 hours, with periodic expiry recovery and no worker heartbeat. Preserve existing retry/error decisions, allow the same worker to reclaim work, fence stale assignments, and acknowledge result retries idempotently.
+- Expose only the authenticated Worker API and a long-random-path authenticated admin area. Keep the standalone dashboard and backend/control/database ports private; ban admin IPs for 24 hours after two actual failed login attempts within ten minutes without banning Worker API traffic.
+- Reuse existing records/logs for worker counts, durations, assignment outcomes, and last API contact.
+- Validate with empty local storage and synthetic fixtures, including crash/retry/expiry scenarios, without production data or real provider requests.
+
+## Capabilities
+
+### New Capabilities
+
+- `distributed-scan-workers`: Minimal remote scan execution, centralized settings, admission, fixed expiry, durable result handoff, observability, and isolated local verification.
+- `restricted-public-access`: Private backend/dashboard topology, authenticated worker/admin access, admin-only login bans, and safe diagnostics.
+
+### Modified Capabilities
+
+None. `openspec/specs/` is empty in this snapshot. Existing unarchived deltas are design references, not canonical specifications to modify or archive as part of this work.
+
+## Impact
+
+The change touches the execution boundary in `app/console_runner.py` and `app/scanner.py`, reservation/recovery in `app/scanner_db.py`, the existing bundle/ingester pipeline, runtime lifecycle wiring, packaging, Docker/Caddy deployment, and regression tests. New persistent state is limited to necessary worker identity/token/quota bindings and metadata attached to existing reservations; PostgreSQL remains the only server queue authority. Client detailed keychecks, another broker, a parallel result format, worker heartbeats, automatic updates, and production migration are out of scope.

openspec/changes/add-minimal-remote-scan-workers/specs/distributed-scan-workers/spec.md

@@ -0,0 +1,145 @@
+## ADDED Requirements
+
+### Requirement: Existing pipeline semantics remain authoritative
+The system SHALL execute existing download/scan logic on trusted Windows/Linux clients and SHALL retain discovery, queue/reservation authority, ingestion, projection, candidate routing, and detailed keycheck on the server. It MUST NOT create a second queue, reduced result format, client detailed-keycheck flow, or new scanner retry/dead-letter policy.
+
+#### Scenario: Remote scan matches current local behavior
+- **WHEN** local and remote execution process the same synthetic target, immutable plan, and effective scanner settings
+- **THEN** normalized findings, detector/provider identities, origin/context, errors, candidate evidence, coverage decisions, and queue dispositions match apart from transport identities and timing
+- **AND** detailed keychecks run through the existing server candidate pipeline after ingestion, independently of JSONL projection completion
+
+### Requirement: Centralized task settings and bounded client authority
+The server SHALL supply the assigned target, immutable plan, effective source/scanner configuration, compatibility identity, and only task-required credentials. Client-authored operational settings SHALL be server URL, device token, and desired slot count. Clients MUST NOT require PostgreSQL, server supervisor authority, independent provider configuration, or arbitrary remote-command execution.
+
+#### Scenario: A client has no provider configuration
+- **WHEN** an authorized compatible client with only its bootstrap settings claims work
+- **THEN** it receives enough task-specific input to use the existing scanner without database access or worker-maintained provider settings
+- **AND** it receives no database/admin secrets or unrelated discovery credential pool
+
+#### Scenario: Incompatible client cannot silently change scan policy
+- **WHEN** a client's scanner build, detector policy, or source/OS support is incompatible
+- **THEN** admission refuses incompatible work without consuming a target or falling back to different scan settings
+
+### Requirement: Per-slot claims respect atomic shared quotas
+Each free client work slot SHALL claim at most one task. The server SHALL atomically enforce the owner's active-assignment cap across devices together with existing admission/capacity restrictions. Client slots SHALL bound pending unacknowledged work, and quota accounting MUST NOT be confused with node-local scan permits or server spool credits.
+
+#### Scenario: Multiple devices race for the last slots
+- **WHEN** a user capped at three active assignments runs two clients configured for eight slots each and they claim concurrently
+- **THEN** at most three assignments are admitted across both clients and no unused batch/backlog is handed out
+
+#### Scenario: An administrator lowers a running user's cap
+- **WHEN** the new cap is below the user's current active-assignment count
+- **THEN** new claims wait until usage permits admission without inventing cancellation or error outcomes for current work
+
+### Requirement: Ambiguous claim delivery is recoverable
+The client SHALL persist a stable admission/request identity before sending a claim, and retries SHALL reconcile the same existing reservation scoped to the authenticated device rather than allocating another target.
+
+#### Scenario: A claim commits but its reply is lost
+- **WHEN** a client retries the original claim identity after a network failure or restart
+- **THEN** the server returns that assignment or its authoritative terminal state without creating an additional reservation or consuming extra quota
+
+### Requirement: Assignment expiry is fixed and server-owned
+Remote assignments SHALL expire after a configurable fixed interval, default 24 hours from server-side issuance, including upload. API contact SHALL NOT renew this deadline. No worker heartbeat or liveness probe SHALL be required. A periodic server recovery pass SHALL process expired unfinished assignments using existing infrastructure-loss recovery/accounting, preserving existing scan retry policy.
+
+#### Scenario: Worker disappears without reporting a scan outcome
+- **WHEN** its assignment deadline passes and the periodic recovery pass runs
+- **THEN** the unfinished target becomes claimable again with correctly reconciled credits/quota and dependent plan leases
+- **AND** the loss does not become a fabricated scanner/provider error, arbitrary retry limit, or worker blacklist
+
+#### Scenario: The same worker returns after expiry
+- **WHEN** the previous worker requests available work after its expired assignment is recovered
+- **THEN** it is eligible to claim that target again under a new issuance identity
+
+#### Scenario: Client contact does not extend work
+- **WHEN** the client retries an upload or another API request before the deadline
+- **THEN** the original server expiry remains unchanged
+
+### Requirement: Remote ownership fences stale results
+Result acceptance and expiry/reissue SHALL serialize on current reservation ownership and deadline. Local producer PID checks MUST NOT reclaim remote assignments. Dependent plan/blob/target leases SHALL remain consistent with the remote ownership interval. Only the current unexpired assignment can first publish an authoritative result.
+
+#### Scenario: Old worker uploads after another issuance
+- **WHEN** an old worker uploads a previously unaccepted result after expiry or reissue
+- **THEN** the server rejects it as stale without completing the newer assignment, advancing plan coverage, or releasing its credits
+
+#### Scenario: Result races with periodic recovery
+- **WHEN** upload acceptance and expiry recovery race for the same assignment
+- **THEN** exactly one authoritative transition wins and neither duplicate ingestion nor double capacity release occurs
+
+#### Scenario: Server restarts while a remote client is still working
+- **WHEN** runtime recovery cannot find a local producer PID for a valid remote assignment
+- **THEN** it retains remote ownership until the fixed deadline rather than treating the absent local process as worker death
+
+### Requirement: Full canonical results have durable idempotent acceptance
+Clients SHALL send the existing canonical v2 `.trb` as a bounded binary stream, preserving findings, errors, metadata, attribution, exact-plan identity, and candidate evidence. The server SHALL validate identity, format, size, paths, and hash and durably publish the bundle plus ready/recovery state before acknowledging custody. Existing transactional ingestion SHALL remain authoritative. Accepted identity/digest/receipt information SHALL survive ordinary ingestion and spool cleanup in authoritative records independently of the bundle file.
+
+#### Scenario: Acceptance reply is lost
+- **WHEN** the server accepts a bundle but its acknowledgement is lost and the client uploads the identical bundle again
+- **THEN** the server returns the original acceptance without duplicate ingestion, candidates, quota release, or counters
+- **AND** this acknowledgement remains recoverable after the assignment deadline because acceptance already occurred
+
+#### Scenario: Accepted identity receives a different body
+- **WHEN** another bundle with conflicting content is submitted for an accepted identity
+- **THEN** the server rejects the conflict without replacing the accepted result
+
+#### Scenario: Client retries after ingestion and normal cleanup
+- **WHEN** an accepted bundle's reply is lost, ingestion and ordinary cleanup finish, the server restarts, and the client retries after the original deadline
+- **THEN** identical bytes recover the original receipt despite the absence of the spool file
+- **AND** conflicting bytes are rejected without duplicate results, candidates, accounting, or counters
+
+#### Scenario: Upload is truncated or invalid
+- **WHEN** an upload exceeds bounds, fails validation, disconnects, or crosses the deadline before first acceptance
+- **THEN** it produces no successful acknowledgement or authoritative findings and cannot mutate another assignment
+- **AND** bounded partial-file cleanup and existing transport/recovery handling apply
+
+#### Scenario: Server crashes around bundle publication
+- **WHEN** the receiver crashes after durable publication or ready-state recording but before replying
+- **THEN** restart reconciliation and a same-identity client retry recover a single consistent acceptance or authoritative rejection without adopting an unvalidated/stale file
+
+#### Scenario: Ingestion is delayed past the worker deadline
+- **WHEN** an accepted ready bundle awaits server ingestion after its former assignment deadline
+- **THEN** worker expiry recovery does not requeue it as unfinished remote work
+
+### Requirement: Client recovery separates transport from scan outcomes
+The client SHALL persist pending bundle and assignment identity until durable acknowledgement and SHALL retry transport using that identity without consuming scanner/provider retry budgets. Existing scan errors SHALL retain their existing dispositions. Definitive stale rejection SHALL be recorded as a local stale/discard outcome with bounded cleanup, not success or infinite upload retry.
+
+#### Scenario: Client restarts with a pending result
+- **WHEN** a client restarts before confirming server acceptance
+- **THEN** it recovers the pending result and retries/reconciles it before claiming replacement work for that occupied slot
+
+#### Scenario: Scanner reports an existing deferred or terminal error
+- **WHEN** the existing scanner returns an error disposition
+- **THEN** the client/server handoff preserves that disposition and the server applies the existing queue policy rather than a transport-specific retry rule
+
+### Requirement: Pre-bundle terminal reports are replay-safe and fenced
+Pre-bundle failure/release reports SHALL use the current issuance identity and existing outcome/accounting rules. Clients SHALL retain and retry the report identity until its authoritative outcome is acknowledged. Duplicate accepted reports SHALL return the original outcome without duplicate retry charges, counters, or quota/credit release; expired/superseded unaccepted reports SHALL NOT alter newer work.
+
+#### Scenario: Failure or release reply is lost
+- **WHEN** the server commits a pre-bundle terminal disposition but its reply is lost and the client repeats the report
+- **THEN** the original disposition is acknowledged, remote quota/credits are reconciled once, and the client slot resolves once without an extra scanner retry charge
+
+#### Scenario: Same worker reports failure for an old issuance
+- **WHEN** a worker reclaims a target under a new issuance and a delayed unaccepted terminal report for its expired issuance arrives
+- **THEN** the server rejects the stale report without changing the new issuance, its quota, or the target's current outcome
+
+### Requirement: Worker observability derives from authoritative events
+The system SHALL expose safe per-worker unfinished/completed/failed/expired counts, last authenticated API contact, correlated target/source/reservation identities, issue/finish times, duration, and outcome/error category using existing logs and records. It SHALL distinguish result acceptance from later processing and MUST NOT infer online/offline status without a heartbeat.
+
+#### Scenario: Duplicate or stale result arrives
+- **WHEN** a result is duplicated or rejected as stale
+- **THEN** a correlated safe event is recorded without inflating completed counts or exposing credentials/raw findings
+
+#### Scenario: A valid worker is silent during a long scan
+- **WHEN** the worker makes no API request before its assignment deadline
+- **THEN** the admin view shows last contact and outstanding assignment state without declaring the worker dead solely from silence
+
+### Requirement: Local validation is isolated and synthetic
+Implementation SHALL be developed and verified in the independent source-only workspace, with fresh test-owned storage and empty PostgreSQL initialized only for synthetic fixtures. Tests MUST NOT copy/restore/mount production data, reuse active runtime volumes/image tags, inherit production credentials/DSNs, or perform live discovery/provider verification. Windows/Linux real scanner parity and controlled-clock recovery tests SHALL precede release.
+
+#### Scenario: Empty local end-to-end execution
+- **WHEN** a test runtime, Caddy, and clients are started for the worker scenario
+- **THEN** they use isolated project/image/volume/port/config identities, synthetic targets, mocked provider transports, and restricted external egress
+- **AND** existing production checkouts, containers, PGDATA, logs, and results are neither read as runtime inputs nor modified
+
+#### Scenario: Unsafe inherited deployment defaults are detected
+- **WHEN** test setup would reuse the inherited production project, shared image tag, existing data volume, runtime bind mount, or live source configuration
+- **THEN** validation refuses to start runtime services until explicit isolation is established

openspec/changes/add-minimal-remote-scan-workers/specs/restricted-public-access/spec.md

@@ -0,0 +1,75 @@
+## ADDED Requirements
+
+### Requirement: Only authenticated edge routes are public
+The deployment SHALL expose HTTPS through Caddy only for `/api/v1/worker/*` and a configured random administrative prefix. The standalone dashboard, PostgreSQL, supervisor control, Caddy control API, and raw backend ports SHALL remain private. Unknown application paths SHALL return 404; an administrative URL MUST NOT replace authentication.
+
+#### Scenario: Internet visitor requests backend or dashboard access
+- **WHEN** an unauthenticated visitor requests `/dashboard`, a normal `/admin` path, an unknown route, or a backend/control/database port
+- **THEN** no dashboard data or backend/control access is available
+
+#### Scenario: Dashboard information appears inside admin
+- **WHEN** operational dashboard information is rendered in the administrative area
+- **THEN** every page, asset, and data request is protected by the same admin authentication boundary without publishing the standalone backend
+
+### Requirement: Worker access has narrow device-scoped authority
+Every Worker API operation SHALL require a valid revocable opaque device token over certificate-validated HTTPS, scoped to its user/device assignments and quotas. The server SHALL store token hashes, not reusable plaintext tokens. Worker tokens MUST NOT authorize admin actions, another device's result mutation, database access, or generic commands.
+
+#### Scenario: Invalid or revoked token submits work
+- **WHEN** an invalid/revoked device token claims a task or uploads a result
+- **THEN** the request is refused before task/result mutation and does not count toward the admin login jail
+
+#### Scenario: Authorized device tries to finish somebody else's task
+- **WHEN** a device submits another device's reservation identity
+- **THEN** the request is refused without changing that reservation or disclosing task credentials
+
+### Requirement: Admin routes require credentials and safe mutations
+The administrative prefix SHALL contain at least 128 bits of randomness and all administrative routes SHALL require login/password authentication using a supported password hash. Typed mutations SHALL enforce appropriate Origin/CSRF protection. The interface MUST NOT expose a shell or generic supervisor-command passthrough.
+
+#### Scenario: Visitor knows the full administrative URL
+- **WHEN** a visitor accesses the correct administrative prefix without valid credentials
+- **THEN** no administrative page, data, asset, or mutation becomes available
+
+#### Scenario: Cross-site request attempts a queue or token mutation
+- **WHEN** a browser submits an administrative mutation without valid same-origin/CSRF authorization
+- **THEN** the mutation is rejected even if browser-level password authentication is present
+
+### Requirement: Two failed admin logins trigger an admin-only day ban
+Two actual invalid administrative credential submissions from one IP within ten minutes SHALL ban that IP from administrative routes for 24 hours. The ban SHALL persist across service restart, expire automatically, and support SSH/operator removal. Initial unauthenticated authentication challenges, Worker API failures, and unrelated requests MUST NOT count. Enforcement SHALL leave Worker API traffic from the same IP unaffected.
+
+#### Scenario: Repeated invalid admin credentials
+- **WHEN** an IP submits a second invalid admin login within the ten-minute window
+- **THEN** subsequent admin requests from that IP are denied for 24 hours, including otherwise valid credentials, until expiry or operator unban
+
+#### Scenario: Admin and worker share one NAT address
+- **WHEN** admin login failures trigger a ban for an IP also used by an authorized worker
+- **THEN** the worker can still claim/upload over HTTPS and administrative requests remain blocked
+
+#### Scenario: Normal first visit receives an authentication challenge
+- **WHEN** a browser initially requests the protected area without sending credentials
+- **THEN** the normal challenge does not consume either of the two failed-login attempts
+
+#### Scenario: Ban expires or operator recovers access
+- **WHEN** 24 hours elapse or an operator removes the ban through the documented SSH procedure
+- **THEN** credential-authenticated administrative access is restored without restarting or weakening Worker API authorization
+
+### Requirement: IP identification and ban enforcement match the deployed edge
+The initial direct-DNS deployment SHALL derive client IP from the direct connection, ignoring untrusted forwarded headers. Any future proxy SHALL require an explicit trusted-proxy configuration and verification before enabling IP bans. The fail2ban action SHALL enforce the admin route boundary at Caddy rather than indiscriminately blocking shared port 443.
+
+#### Scenario: Attacker supplies another address in a forwarded header
+- **WHEN** a direct client sends arbitrary `X-Forwarded-For` or equivalent headers with failed admin credentials
+- **THEN** another user's address is not selected for banning and the caller cannot evade its own ban
+
+#### Scenario: Deployed ban is verified through Docker ingress
+- **WHEN** the admin jail applies its ban against the actual Caddy/Docker topology
+- **THEN** external administrative requests are denied and worker requests remain available, not merely a host firewall rule being present
+
+### Requirement: Diagnostics and transport protect sensitive data
+All worker/admin transport SHALL use HTTPS without disabling certificate verification. Tokens, passwords, secret admin prefixes in routine access logs, task credentials, and raw findings SHALL be redacted or omitted from diagnostics. Admin responses SHALL disable caching and referrer disclosure and use a compatible restrictive CSP; production HTTPS SHALL use HSTS. Client disk/workspace encryption SHALL NOT be required.
+
+#### Scenario: Auth or upload error is logged
+- **WHEN** authentication, payload validation, or upload handling fails
+- **THEN** logs contain only safe correlation/status/error metadata and enough redacted authentication outcome for the admin jail, not request credentials or raw result content
+
+#### Scenario: Trusted client stores pending work locally
+- **WHEN** a worker downloads or persists a bundle before acknowledgement
+- **THEN** ordinary local storage is supported without an application encryption layer while HTTPS and post-acknowledgement cleanup remain enforced

openspec/changes/add-minimal-remote-scan-workers/tasks.md

@@ -0,0 +1,49 @@
+## 1. Establish Safe Empty Test Isolation
+
+- [x] 1.1 Replace unsafe inherited test launch defaults with explicit test-owned project/image/volume/port identities and refusal checks for production binds/volumes; do not run the copied default Compose or import overrides.
+- [x] 1.2 Reuse existing test harnesses to initialize empty PostgreSQL, scrub inherited secrets/DSNs/proxies before imports, disable live source/keycheck autostart, and provide synthetic targets/provider transports with external egress blocked.
+- [x] 1.3 Establish baseline synthetic scan/bundle/queue/keycheck fixtures and run the reviewed isolated unit selection before changing execution semantics; record relevant pre-existing failures separately.
+
+## 2. Extract The Existing Scan Execution Boundary
+
+- [x] 2.1 Trace `stage_claim`, `scan_target_result`, process authority, bundle candidate extraction, and Git/Docker exact-plan inputs; define the smallest DB-free job input using existing types/identities rather than a parallel task model.
+- [x] 2.2 Adapt one planned download/scan/bundle path to run on Windows/Linux without PostgreSQL or supervisor credentials while preserving `OwnedProcess`, native slot handling, source errors/dispositions, and structured Postman candidates.
+- [x] 2.3 Add centralized effective-config/plan delivery and protocol/scanner/detector-policy compatibility validation, supplying only task-needed credentials and forbidding arbitrary commands or client provider overrides.
+- [x] 2.4 Compare local and DB-free execution on existing source fixtures, including Git/Docker coverage, Postman evidence, and Xai/ZAI custom-detector direction regressions; preserve detailed keycheck exclusively on the server.
+
+## 3. Extend Existing Admission And Recovery
+
+- [x] 3.1 Add only necessary user/device token-hash/quota bindings and remote metadata on existing reservations, with revocation and no second queue or independent remote-job state machine.
+- [x] 3.2 Implement authenticated one-task claims through existing admission with atomic per-user caps across devices, existing capacity limits, stable request identity, ambiguous-claim reconciliation, and bounded empty/capacity polling.
+- [x] 3.3 Apply configurable server-clock fixed expiry (default 24 hours) to remote ownership and dependent target/plan/blob leases, separating it from local PID recovery and scanner timeouts without heartbeat or renewal endpoints.
+- [x] 3.4 Wire periodic expired-assignment recovery into runtime maintenance using existing refund/requeue accounting; permit same-worker reclaim, avoid new error/retry/blacklist policies, and release quota exactly once.
+- [x] 3.5 Test concurrent quota admission, lower-cap behavior, lost claim replies, restart recovery, fixed-clock expiry, dependent plan leases, and stale ownership fencing against isolated PostgreSQL.
+
+## 4. Add Durable Canonical Bundle Transport
+
+- [x] 4.1 Receive `.trb` binary streams into bounded server-owned partial files, enforcing existing capacity/size limits, upload timeouts, ownership, expiry, codec/path/identity validation, and content hash.
+- [x] 4.2 Reuse durable atomic publication and `mark_result_bundle_ready` before acknowledgement; reconcile upload/recovery races and reject stale/conflicting bodies without affecting current plan coverage or credits.
+- [x] 4.3 Preserve accepted identity/digest/receipt in existing records independently of spool cleanup and return the same receipt after ingestion, cleanup, restart, and expiry; reuse existing ingester/projector/candidate/keycheck behavior and exclude ready bundles from worker expiry.
+- [x] 4.4 Preserve current scan-failure dispositions and pre-bundle infrastructure release paths with issuance-fenced idempotent terminal-report replay; test lost/repeated/stale reports, single slot/quota/credit resolution, interrupted/invalid/conflicting uploads, and publication/ready/commit crashes.
+
+## 5. Build The Minimal Client Loop
+
+- [x] 5.1 Implement server-URL/token/N bootstrap and one claim per free slot, with node-local execution containment and no independent source/provider configuration, client keycheck, heartbeat, batching, or updater.
+- [x] 5.2 Persist assignment identity and pending bundles, recover them on restart, retry transport without scan retry charges, free work slots only after authoritative resolution, and handle definitive stale rejection with explicit bounded cleanup.
+- [x] 5.3 Package pinned scanner/config assets for a Windows portable client and Linux client/container; verify certificate-validating HTTPS and avoid secret/raw-finding logs without requiring local encryption.
+- [x] 5.4 Exercise real synthetic scans and complete bundle round trips on both Windows and Linux with N greater than one, restart during pending upload, server outage, and subsequent recovery; do not substitute mocks for cross-platform scanner verification.
+
+## 6. Restrict Edge Access And Add Minimal Administration
+
+- [x] 6.1 Wire API/admin into the single runtime and separate Caddy edge, publishing only authenticated worker routes and a random admin prefix; keep dashboard/backend/database/control ports private and remote admission disabled until configured.
+- [x] 6.2 Add device-scoped token authorization and hash-based admin password authentication with protected assets, typed CSRF/Origin-checked mutations, no-store/same-origin-referrer/CSP/production-HSTS headers, and redacted logs.
+- [x] 6.3 Add a host fail2ban admin jail for two actual bad logins in ten minutes and a 24-hour persisted ban; implement validated admin-only Caddy denylist updates, direct-IP handling, automatic expiry, and documented SSH unban without blocking worker traffic.
+- [x] 6.4 Build only necessary admin user/device-token/quota and existing queue controls plus read-only worker counts, durations, outcomes, and last-contact summaries from existing records; do not add online/offline guesses or a telemetry store.
+- [x] 6.5 Verify unknown/private routes, revoked/wrong-device tokens, cross-site mutations, absent-credentials challenges, spoofed forwarded headers, actual two-failure bans/restart/unban, and an authorized worker sharing the banned admin IP through the isolated edge.
+
+## 7. Complete Regression Gates And Handoff
+
+- [x] 7.1 Run the synthetic empty-database end-to-end flow through claim, real scan, complete bundle, ingestion, projection, and mocked detailed server keycheck; compare normalized output/dispositions with the existing local path.
+- [x] 7.2 Run combined disconnect/restart/expiry/reissue/upload races with controlled clocks; verify single authoritative acceptance, intact plan coverage, no credit leaks, and no duplicate worker statistics.
+- [x] 7.3 Verify test manifests, build context, logs, artifacts, and cleanup exclude production state/credentials and that active production containers/volumes were untouched; retain only test-owned failure evidence.
+- [x] 7.4 Document isolated run commands, client bootstrap, quota/deadline tuning, token revocation, admin unban, accepted-custody semantics, known 24-hour recovery trade-offs, and later rollout/drain rollback; validate OpenSpec and leave unrelated changes unarchived.