truf-server/docs/end-to-end-scanner-validation.md

# End-to-End Scanner Validation

This runbook validates the real discovery, remote-worker, result-ingestion, and
compatibility-projection path with a bounded cohort of 30-40 targets. It is an
evidence procedure, not a claim that every repository feature and environment
has been proven correct.

The completed 2026-09-22 production evidence is recorded in
[`end-to-end-scanner-validation-2026-09-22.md`](end-to-end-scanner-validation-2026-09-22.md).

## Validation Questions

The run must answer all of the following:

1. Does bounded discovery create the expected immutable queue identities?
2. Do remote workers receive each authoritative assignment with the correct
   source, target identity, execution snapshot, and lease fencing?
3. Does each worker run the intended TruffleHog scan and upload a canonical
   protocol-2 result bundle?
4. Does the server accept a result exactly once and make receipt replay
   idempotent?
5. Does the ingester transactionally connect the reservation, queue row,
   target scan, findings, errors, and bundle record?
6. Does the JSONL projector reproduce the Windows-compatible
   `scan_results.jsonl` and `found_secrets.jsonl` structures without becoming
   a second source of truth?
7. Are naturally found or controlled-canary findings stored with safe identity,
   location, detector, verification, redaction, and provenance fields?
8. After the test, are queues settled, projections caught up, no pipeline item
   quarantined, and the normal production configuration restored?

## Authorities and Expected Data Flow

The authoritative sequence is:

```text
discovery cycle
  -> target_queue
  -> result_reservation / immutable remote assignment
  -> worker TruffleHog execution
  -> canonical .trb upload
  -> durable accepted receipt
  -> result ingester transaction
  -> target_scans + findings + errors + queue completion
  -> projection_jobs
  -> scan_results.jsonl + found_secrets.jsonl
```

PostgreSQL is authoritative. Result bundles are durable pipeline artifacts.
JSONL files are rebuildable compatibility projections and may lag briefly.
`accepted` proves durable server receipt; `ingested` proves the database
transaction completed. These states must not be treated as synonyms.

## Safety Rules

- Use only the approved test server and approved worker devices.
- Never print device tokens, source credentials, raw secrets, active runtime
  YAML, worker argv, or complete unredacted findings into a terminal/log.
- Query finding structure using IDs, hashes, redacted values, lengths, booleans,
  detector names, and location metadata. Review any raw secret only through the
  already protected admin workflow if explicitly required.
- Apply and restore configuration through Preview -> Save candidate -> Apply.
  Do not edit the active runtime document in place.
- Pause discovery and dispatch and drain before each runtime-document apply.
- Record the original config SHA-256 and require byte-identical restoration at
  the end.
- Use a unique test-run label and database high-water marks. Never infer the
  cohort from wall-clock time alone.
- Do not delete queue, bundle, finding, projection, or receipt evidence to make
  a failed test look clean.

## Bounded Test Configuration

Use a temporary candidate derived from the active document. Preserve all
secrets and unrelated settings. The exact candidate must be reviewed before it
is applied.

### DockerHub

- `mode: search`
- `pages: 1`
- `per_page: 1`
- `docker_images_per_repository: 1`
- Use a reviewed finite query list for the test window.

One query is consumed per source cycle. An already-known or unsuitable search
result can produce no new queue row, so the number of cycles is not the cohort
size.

### GitLab

- `mode: search`
- `pages: 1`
- `per_page: 1`
- Use a reviewed finite query list for the test window.
- Keep current age, commit-boundary, exact-ref, visibility, and history-depth
  safety controls unless the test explicitly records a different expectation.

### HuggingFace

HuggingFace recent discovery does not support a real `per_page: 1` keyword
test. Its API runner fetches newest-modified Spaces and the current API page
size is fixed at 100; the configured query is only a rotation placeholder.

For a bounded cohort, use `mode: custom` with a reviewed private `target_file`
containing a small list of Space IDs. Do not claim that changing `per_page` to
1 bounded this source when it did not.

### Recommended Cohort

Target 36 authoritative terminal scans:

- 16 DockerHub immutable digest targets;
- 16 GitLab exact-ref/commit-planned targets;
- 4 HuggingFace custom Space targets.

The exact split may vary between 30 and 40 when discovery deduplicates known
targets or a target becomes permanently inaccessible. Continue only until the
recorded cohort reaches the agreed bound. Do not inflate discovery simply to
hit an exact aesthetic number.

## Positive-Finding Requirement

A random public cohort may correctly produce zero findings. Zero findings
cannot validate the finding-storage and `found_secrets.jsonl` path.

Include at least one separately identified, non-live controlled fixture that is
expected to trigger an already approved detector. The fixture must contain no
usable credential. Record its expected detector and identity before scanning.
Do not weaken verification, introduce a new detector, or publish a real secret
merely to force a positive result.

If no approved positive fixture is available, report the finding path as
unverified by this run even if all zero-finding scans succeed.

## Phase 1: Baseline

With runtime healthy, record a secret-safe baseline:

- active config SHA-256 and semantic config SHA-256;
- runtime-control revision and open/paused/drain state;
- enabled source set and active worker package manifests;
- remote worker/device count, recent contact, and package capability match;
- high-water IDs for `target_queue`, `result_reservations`, `target_scans`,
  `findings`, `errors`, `result_bundles`, and `projection_jobs`;
- queue counts by source and status;
- active reservation count and oldest age;
- pipeline worker readiness and capacity counters;
- pending/leased/quarantined bundle, projection, and keycheck counts;
- current projection stream/cursor identity;
- byte size and final complete-line identity of active JSONL files.

The baseline collector must print aggregates and hashes only. It must not emit
targets, assignment payloads, tokens, raw findings, or runtime documents.

## Phase 2: Apply the Test Candidate

1. Pause discovery.
2. Pause dispatch.
3. Start drain and wait for blocker count zero and `drained`.
4. Preview the bounded candidate and review the semantic diff.
5. Save and apply the candidate through the host-agent lifecycle.
6. Require reconciled `succeeded`, no failed hold, strict runtime health, edge
   health, admin health, and Worker API health.
7. Cancel drain, then resume discovery and dispatch in that order.

Do not continue if the lifecycle operation rolls back or enters failed hold.

## Phase 3: Build and Freeze the Cohort

Record the baseline `target_queue.id` high-water mark. Let the bounded sources
cycle until 30-40 new eligible queue rows have been created after that mark.

Then:

1. Pause discovery so the cohort cannot grow.
2. Leave dispatch open until the selected queue rows settle.
3. Record cohort queue IDs and only their safe identities: source, normalized
   target hash, query hash, immutable planning kind, and creation order.
4. Separate deduplicated, permanently inaccessible, deferred, retried, and
   actually assigned items. Do not count an API result as a scan.

The authoritative cohort is a fixed set of queue IDs, not "whatever completed
during the same hour."

## Phase 4: Observe Remote Execution

For every cohort queue ID, verify:

- no more than one current authoritative reservation;
- assignment package/platform capability matches the registered worker;
- lease token and execution snapshot are bound but never printed;
- Docker targets are immutable `repo@sha256` identities;
- GitLab targets have the intended exact planning/ref identity;
- HuggingFace targets use the direct Space execution kind;
- terminal report classification is success, permanent target failure, or
  retryable provider failure as designed;
- retries preserve queue identity and increment attempts without creating a
  second authoritative acceptance;
- accepted receipt replay returns the same durable result.

Physical work can repeat after a lease expiry or network partition. Correctness
means fencing permits one authoritative acceptance and one queue completion,
not that duplicate physical execution is impossible.

## Phase 5: Validate Bundles and PostgreSQL Structure

For each accepted result, validate without dumping body content:

- bundle exists at the registered private relative path;
- bundle byte count and SHA-256 match database metadata;
- bundle schema/version, event ID/hash, reservation ID, queue ID, source,
  normalized target identity, execution snapshot identity, and scan policy are
  internally consistent;
- result is ingested exactly once;
- `target_queue.target_scan_id` references the corresponding `target_scans.id`;
- queue completion is applied once with a terminal disposition;
- `target_scans.queue_id` and claim lease identity refer back to the cohort row;
- `target_scans.findings_count` and `error_count` equal actual child-row counts;
- every finding/error references the same target scan, source, cycle, and run;
- no legacy `raw_result_json`, publication outbox row, or orphan relation is
  introduced;
- ingested bundle credit and pipeline-capacity counters are released exactly
  according to the durable state machine;
- no cohort item enters `pipeline_quarantine`.

Aggregate checks must cover the entire cohort. Additionally inspect a small
redacted structural sample from every source and every terminal disposition.

## Phase 6: Validate Findings

For every finding in the cohort, inspect structure only:

- stable `finding_uid` and finding fingerprint;
- detector name/type and verification flag;
- source, target hash, file path, line/commit/source timestamp where applicable;
- redacted secret and secret/detector hashes;
- provider and credential-kind enrichment;
- required-context and raw-payload-omitted flags;
- bounded source metadata and enrichment JSON decode successfully;
- no unexpected raw-secret exposure in logs, queue rows, assignment metadata,
  admin list views, or compatibility scan summaries.

For the controlled positive fixture, require the expected finding to exist in
PostgreSQL and to project once to `found_secrets.jsonl`.

## Phase 7: Validate Windows-Compatible Files

Use the active configured `global.results_dir`. The relevant compatibility
outputs are:

- `scan_results.jsonl` for one sanitized scan event per projected scan;
- `found_secrets.jsonl` for projected finding events;
- their publication ledgers, active stream metadata, and rotated segments;
- per-service keycheck result files only if keychecks run for the finding.

For the cohort, verify:

- every required `projection_job` reaches `completed`;
- projector cursor and append ledger advance monotonically;
- each cohort scan event appears exactly once by `scan_event_id`;
- each cohort finding appears exactly once by `finding_uid`;
- JSON lines parse and match the current compatibility schema;
- scan summaries match PostgreSQL counts and terminal status;
- finding projections are redacted as designed and preserve safe provenance;
- active and rotated segments together contain the events; checking only the
  active file is insufficient when rotation occurs;
- no torn-tail quarantine, duplicate append, skipped cursor, or unpublished
  completed job exists.

These files should have the same logical structure as the Windows deployment,
but path separators and host/container root paths are platform-specific.

## Phase 8: Queue and Pipeline Closure

After all cohort rows settle, require:

- 30-40 cohort queue rows accounted for by terminal, deferred, or explicitly
  classified retry state;
- no expired active reservation remains unreaped;
- no queue row has multiple authoritative accepted results;
- no accepted result remains un-ingested beyond the bounded pipeline window;
- no completed scan remains unprojected beyond the bounded projector window;
- no stale pipeline lease or capacity leak;
- no unexpected quarantine;
- runtime, Worker API, edge, host-agent, host Caddy, and unrelated host service
  health remain good.

The final report must show counts for discovered, deduplicated, assigned,
retried, accepted, ingested, projected, succeeded, skipped/permanent,
retryable/deferred, findings, errors, and quarantines.

## Phase 9: Restore Production Configuration

1. Pause discovery and dispatch.
2. Drain to zero blockers.
3. Apply the exact original runtime document through the normal lifecycle.
4. Require byte-identical original config SHA-256, reconciled lifecycle success,
   strict health, and no failed hold.
5. Cancel drain, resume discovery, then resume dispatch according to the
   original control state.
6. Confirm worker contact and normal post-test assignment flow.

Do not restore by manually editing YAML or replacing files behind the
host-agent.

## Pass Criteria

The run passes only when:

- at least 30 and at most 40 fixed-cohort rows are fully accounted for;
- all accepted cohort results ingest exactly once;
- all required cohort projections complete exactly once;
- queue/reservation/scan/finding/error/bundle relationships are consistent;
- the controlled positive finding reaches PostgreSQL and
  `found_secrets.jsonl`, or the report explicitly marks positive-finding
  validation incomplete because no approved fixture existed;
- no unexplained retry, orphan, duplicate acceptance, capacity leak,
  quarantine, failed hold, or projection gap remains;
- the original production config is restored exactly and services are healthy.

Any failure must retain its operation IDs, queue IDs, reservation IDs, hashes,
safe categories, and aggregate evidence for diagnosis. A partial pass must not
be reported as "100% scanner correctness."

## Evidence Report

Append or link a dated report containing:

- environment and worker package identities;
- original/test/restored config hashes;
- cohort definition and aggregate source split;
- lifecycle operation IDs for test apply and restore;
- queue and pipeline baseline/final aggregates;
- per-stage reconciliation counts;
- redacted structural examples for a scan, an error/skip, and a finding;
- JSONL/ledger reconciliation counts;
- deviations, retries, quarantines, and unresolved questions;
- final verdict with explicit tested and untested boundaries.