truf-server/openspec/changes/add-minimal-remote-scan-workers/specs/distributed-scan-workers/spec.md

ADDED Requirements

Requirement: Existing pipeline semantics remain authoritative

The system SHALL execute existing download/scan logic on trusted Windows/Linux clients and SHALL retain discovery, queue/reservation authority, ingestion, projection, candidate routing, and detailed keycheck on the server. It MUST NOT create a second queue, reduced result format, client detailed-keycheck flow, or new scanner retry/dead-letter policy.

Scenario: Remote scan matches current local behavior

• WHEN local and remote execution process the same synthetic target, immutable plan, and effective scanner settings
• THEN normalized findings, detector/provider identities, origin/context, errors, candidate evidence, coverage decisions, and queue dispositions match apart from transport identities and timing
• AND detailed keychecks run through the existing server candidate pipeline after ingestion, independently of JSONL projection completion

Requirement: Centralized task settings and bounded client authority

The server SHALL supply the assigned target, immutable plan, effective source/scanner configuration, compatibility identity, and only task-required credentials. Client-authored operational settings SHALL be server URL, device token, and desired slot count. Clients MUST NOT require PostgreSQL, server supervisor authority, independent provider configuration, or arbitrary remote-command execution.

Scenario: A client has no provider configuration

• WHEN an authorized compatible client with only its bootstrap settings claims work
• THEN it receives enough task-specific input to use the existing scanner without database access or worker-maintained provider settings
• AND it receives no database/admin secrets or unrelated discovery credential pool

Scenario: Incompatible client cannot silently change scan policy

• WHEN a client's scanner build, detector policy, or source/OS support is incompatible
• THEN admission refuses incompatible work without consuming a target or falling back to different scan settings

Requirement: Per-slot claims respect atomic shared quotas

Each free client work slot SHALL claim at most one task. The server SHALL atomically enforce the owner's active-assignment cap across devices together with existing admission/capacity restrictions. Client slots SHALL bound pending unacknowledged work, and quota accounting MUST NOT be confused with node-local scan permits or server spool credits.

Scenario: Multiple devices race for the last slots

• WHEN a user capped at three active assignments runs two clients configured for eight slots each and they claim concurrently
• THEN at most three assignments are admitted across both clients and no unused batch/backlog is handed out

Scenario: An administrator lowers a running user's cap

• WHEN the new cap is below the user's current active-assignment count
• THEN new claims wait until usage permits admission without inventing cancellation or error outcomes for current work

Requirement: Ambiguous claim delivery is recoverable

The client SHALL persist a stable admission/request identity before sending a claim, and retries SHALL reconcile the same existing reservation scoped to the authenticated device rather than allocating another target.

Scenario: A claim commits but its reply is lost

• WHEN a client retries the original claim identity after a network failure or restart
• THEN the server returns that assignment or its authoritative terminal state without creating an additional reservation or consuming extra quota

Requirement: Assignment expiry is fixed and server-owned

Remote assignments SHALL expire after a configurable fixed interval, default 24 hours from server-side issuance, including upload. API contact SHALL NOT renew this deadline. No worker heartbeat or liveness probe SHALL be required. A periodic server recovery pass SHALL process expired unfinished assignments using existing infrastructure-loss recovery/accounting, preserving existing scan retry policy.

Scenario: Worker disappears without reporting a scan outcome

• WHEN its assignment deadline passes and the periodic recovery pass runs
• THEN the unfinished target becomes claimable again with correctly reconciled credits/quota and dependent plan leases
• AND the loss does not become a fabricated scanner/provider error, arbitrary retry limit, or worker blacklist

Scenario: The same worker returns after expiry

• WHEN the previous worker requests available work after its expired assignment is recovered
• THEN it is eligible to claim that target again under a new issuance identity

Scenario: Client contact does not extend work

• WHEN the client retries an upload or another API request before the deadline
• THEN the original server expiry remains unchanged

Requirement: Remote ownership fences stale results

Result acceptance and expiry/reissue SHALL serialize on current reservation ownership and deadline. Local producer PID checks MUST NOT reclaim remote assignments. Dependent plan/blob/target leases SHALL remain consistent with the remote ownership interval. Only the current unexpired assignment can first publish an authoritative result.

Scenario: Old worker uploads after another issuance

• WHEN an old worker uploads a previously unaccepted result after expiry or reissue
• THEN the server rejects it as stale without completing the newer assignment, advancing plan coverage, or releasing its credits

Scenario: Result races with periodic recovery

• WHEN upload acceptance and expiry recovery race for the same assignment
• THEN exactly one authoritative transition wins and neither duplicate ingestion nor double capacity release occurs

Scenario: Server restarts while a remote client is still working

• WHEN runtime recovery cannot find a local producer PID for a valid remote assignment
• THEN it retains remote ownership until the fixed deadline rather than treating the absent local process as worker death

Requirement: Full canonical results have durable idempotent acceptance

Clients SHALL send the existing canonical v2 .trb as a bounded binary stream, preserving findings, errors, metadata, attribution, exact-plan identity, and candidate evidence. The server SHALL validate identity, format, size, paths, and hash and durably publish the bundle plus ready/recovery state before acknowledging custody. Existing transactional ingestion SHALL remain authoritative. Accepted identity/digest/receipt information SHALL survive ordinary ingestion and spool cleanup in authoritative records independently of the bundle file.

Scenario: Acceptance reply is lost

• WHEN the server accepts a bundle but its acknowledgement is lost and the client uploads the identical bundle again
• THEN the server returns the original acceptance without duplicate ingestion, candidates, quota release, or counters
• AND this acknowledgement remains recoverable after the assignment deadline because acceptance already occurred

Scenario: Accepted identity receives a different body

• WHEN another bundle with conflicting content is submitted for an accepted identity
• THEN the server rejects the conflict without replacing the accepted result

Scenario: Client retries after ingestion and normal cleanup

• WHEN an accepted bundle's reply is lost, ingestion and ordinary cleanup finish, the server restarts, and the client retries after the original deadline
• THEN identical bytes recover the original receipt despite the absence of the spool file
• AND conflicting bytes are rejected without duplicate results, candidates, accounting, or counters

Scenario: Upload is truncated or invalid

• WHEN an upload exceeds bounds, fails validation, disconnects, or crosses the deadline before first acceptance
• THEN it produces no successful acknowledgement or authoritative findings and cannot mutate another assignment
• AND bounded partial-file cleanup and existing transport/recovery handling apply

Scenario: Server crashes around bundle publication

• WHEN the receiver crashes after durable publication or ready-state recording but before replying
• THEN restart reconciliation and a same-identity client retry recover a single consistent acceptance or authoritative rejection without adopting an unvalidated/stale file

Scenario: Ingestion is delayed past the worker deadline

• WHEN an accepted ready bundle awaits server ingestion after its former assignment deadline
• THEN worker expiry recovery does not requeue it as unfinished remote work

Requirement: Client recovery separates transport from scan outcomes

The client SHALL persist pending bundle and assignment identity until durable acknowledgement and SHALL retry transport using that identity without consuming scanner/provider retry budgets. Existing scan errors SHALL retain their existing dispositions. Definitive stale rejection SHALL be recorded as a local stale/discard outcome with bounded cleanup, not success or infinite upload retry.

Scenario: Client restarts with a pending result

• WHEN a client restarts before confirming server acceptance
• THEN it recovers the pending result and retries/reconciles it before claiming replacement work for that occupied slot

Scenario: Scanner reports an existing deferred or terminal error

• WHEN the existing scanner returns an error disposition
• THEN the client/server handoff preserves that disposition and the server applies the existing queue policy rather than a transport-specific retry rule

Requirement: Pre-bundle terminal reports are replay-safe and fenced

Pre-bundle failure/release reports SHALL use the current issuance identity and existing outcome/accounting rules. Clients SHALL retain and retry the report identity until its authoritative outcome is acknowledged. Duplicate accepted reports SHALL return the original outcome without duplicate retry charges, counters, or quota/credit release; expired/superseded unaccepted reports SHALL NOT alter newer work.

Scenario: Failure or release reply is lost

• WHEN the server commits a pre-bundle terminal disposition but its reply is lost and the client repeats the report
• THEN the original disposition is acknowledged, remote quota/credits are reconciled once, and the client slot resolves once without an extra scanner retry charge

Scenario: Same worker reports failure for an old issuance

• WHEN a worker reclaims a target under a new issuance and a delayed unaccepted terminal report for its expired issuance arrives
• THEN the server rejects the stale report without changing the new issuance, its quota, or the target's current outcome

Requirement: Worker observability derives from authoritative events

The system SHALL expose safe per-worker unfinished/completed/failed/expired counts, last authenticated API contact, correlated target/source/reservation identities, issue/finish times, duration, and outcome/error category using existing logs and records. It SHALL distinguish result acceptance from later processing and MUST NOT infer online/offline status without a heartbeat.

Scenario: Duplicate or stale result arrives

• WHEN a result is duplicated or rejected as stale
• THEN a correlated safe event is recorded without inflating completed counts or exposing credentials/raw findings

Scenario: A valid worker is silent during a long scan

• WHEN the worker makes no API request before its assignment deadline
• THEN the admin view shows last contact and outstanding assignment state without declaring the worker dead solely from silence

Requirement: Local validation is isolated and synthetic

Implementation SHALL be developed and verified in the independent source-only workspace, with fresh test-owned storage and empty PostgreSQL initialized only for synthetic fixtures. Tests MUST NOT copy/restore/mount production data, reuse active runtime volumes/image tags, inherit production credentials/DSNs, or perform live discovery/provider verification. Windows/Linux real scanner parity and controlled-clock recovery tests SHALL precede release.

Scenario: Empty local end-to-end execution

• WHEN a test runtime, Caddy, and clients are started for the worker scenario
• THEN they use isolated project/image/volume/port/config identities, synthetic targets, mocked provider transports, and restricted external egress
• AND existing production checkouts, containers, PGDATA, logs, and results are neither read as runtime inputs nor modified

Scenario: Unsafe inherited deployment defaults are detected

• WHEN test setup would reuse the inherited production project, shared image tag, existing data volume, runtime bind mount, or live source configuration
• THEN validation refuses to start runtime services until explicit isolation is established