Initial server source import

openspec/changes/add-adaptive-docker-payload-scanning/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-02

openspec/changes/add-adaptive-docker-payload-scanning/design.md

@@ -0,0 +1,352 @@
+## Context
+
+Docker scanning currently has two materially different paths. The full-image TruffleHog source is
+authoritative for new and normally completing images, but it treats an immutable image as one
+indivisible operation. The bounded layer path can resume and globally reuse content digests, but its
+fixed highest-eight-layer selector was admitted only for prior full-image timeouts. In the completed
+control cohort it retained 21.1% of routed identities and 5.8% of detector identities, so enabling
+that selector broadly would trade too much useful coverage for speed.
+
+The layer path already provides the expensive safety primitives this change needs: exact manifest
+resolution, authenticated bounded Registry transfer, digest verification, private artifacts,
+contained TruffleHog filesystem execution, canonical reservation-bound plans, policy-scoped global
+blob leases, fenced result ingestion, and explicit per-image coverage. The missing pieces are a
+content-aware selector, a reusable execution-policy identity that is independent of selector
+budgets, and trustworthy evidence for deciding whether the adaptive path is safe to broaden.
+
+Docker/OCI configuration contains an ordered `history` list that can help identify `COPY`, `ADD`,
+application setup, package installation, generic `RUN`, and likely bulk-data layers. That history is
+untrusted and may itself contain secret material. It is therefore only a bounded selection hint; it
+must never become an authority for content identity or successful coverage and its raw commands
+must never be persisted or logged.
+
+The production database already contains durable version-one layer plans and covered blob rows.
+Changing their interpretation in place would invalidate audit and lease fences. The migration must
+be additive, retain version-one validation, and allow only explicitly proven compatible successful
+coverage to enter the new execution namespace.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Scan all supported unique content for images that fit conservative bounds.
+- Prioritize likely application, configuration, and source-bearing layers when a large image cannot
+  fit those bounds.
+- Reuse successful immutable blob coverage across images and selector revisions when execution
+  semantics are unchanged.
+- Freeze the first deterministic selection for an image and selector policy across every checkpoint,
+  retry, and execution-policy transition.
+- Reduce per-image checkpoint overhead with bounded multi-blob leases without weakening per-blob
+  execution and ingestion fences.
+- Preserve exact selected, reused, skipped, failed, and partial coverage semantics.
+- Produce private, aggregate, non-authoritative shadow evidence against 50-100 completed full-image
+  controls before any broad adaptive rollout.
+- Retain full-image scanning and the existing timeout-only layer canary as immediate rollback paths.
+
+**Non-Goals:**
+
+- Infer arbitrary file contents without downloading a compressed layer.
+- Claim complete coverage for an image with unsupported, oversized, failed, or budget-excluded
+  descriptors.
+- Persist raw Docker config history, shadow findings, provider keys, target names, or Registry bearer
+  tokens in rollout evidence.
+- Change detector classification, keycheck routing, Docker account ownership, repository resolver
+  scheduling, guaranteed scan-slot capacity, worker count, or non-Docker scanners.
+- Reconstruct a merged container filesystem or remove historical whiteout content from individual
+  layer evidence.
+- Destructively rewrite or delete version-one plans and coverage rows.
+
+## Decisions
+
+### 1. Introduce a version-two immutable content plan
+
+Adaptive execution uses a canonical version-two plan. It retains the version-one image, repository,
+manifest, platform, media, limits, scan-policy, descriptor, reservation, and plan-hash fences and
+adds:
+
+- a versioned selector algorithm and `selection_policy_sha256`;
+- an `execution_policy_sha256` for reusable successful blob evidence;
+- one bounded classifier value per descriptor;
+- exact selection and omission reasons; and
+- bounded checkpoint lease limits that do not alter the frozen selection set.
+
+Version-one plan and execution validators remain exact because their JSON is already durable.
+Version-two validation dispatches by the exact integer version and rejects unknown fields, unknown
+classes, malformed hashes, descriptor reordering, and oversized canonical JSON. A reservation can
+bind one canonical plan only; idempotent replay must be byte-identical.
+
+After exact manifest resolution and before plan binding, the scanner fetches the configuration blob
+through the existing bounded, authenticated, digest-verifying Registry path. It parses the JSON in
+bounded private memory/storage and maps non-`empty_layer` history entries from base to top onto the
+ordered manifest layers. The optional `rootfs.diff_ids` count and the non-empty history count must
+agree with the layer count. A missing field, malformed value, excess entry count, or alignment
+mismatch classifies every layer as `unknown`; it does not change descriptor identity or fail a
+valid immutable manifest.
+
+The classifier uses a small versioned allow-list and emits only these bounded classes:
+`config`, `copy_add`, `app_config_run`, `package_run`, `other_run`, `bulk_data`, and `unknown`.
+Raw `created_by` values are discarded before plan construction and are excluded from logs, result
+metadata, errors, and database rows.
+
+Alternatives rejected:
+
+- Persisting normalized command text would retain unnecessary secret-bearing input.
+- Treating history as authoritative would let malformed or adversarial metadata hide content.
+- Mutating version-one plans would break exact replay and auditability.
+
+### 2. Separate scan, execution, and selection identities
+
+Three hashes have distinct responsibilities:
+
+- `scan_policy_sha256` remains the installed scanner/detector/config fingerprint.
+- `execution_policy_sha256` hashes the scan policy plus versioned content validation and all archive
+  semantics that can change which bytes a successful command examines. It excludes image/layer
+  selection budgets, classifier weights, retry counts, lease duration, checkpoint size, and delay.
+- `selection_policy_sha256` hashes the selector version, class ordering, deterministic tie-breaks,
+  supported descriptor classes, and all limits that determine the initial selected set. It excludes
+  mutable global coverage and execution scheduling.
+
+For version-two rows, the existing `docker_content_blobs.coverage_policy_sha256` key stores the
+execution-policy hash. Selector changes therefore do not force an identical successfully scanned
+digest through TruffleHog again, while archive or detector semantic changes still create a separate
+coverage namespace.
+
+`docker_image_blob_coverage` receives a non-null `selection_policy_sha256`. Existing rows are
+backfilled from the exact bound plan on their linked reservation and indexed by
+`(queue_id, manifest_digest, selection_policy_sha256, position, reservation_id)`. The earliest full
+position map under that key is the immutable selection baseline. Coverage-policy changes may require
+new execution but cannot expand or contract that baseline silently.
+
+Successfully covered version-one evidence may be copied lazily into the version-two execution
+namespace only in the same transaction that validates all of the following:
+
+- the old row is durably `covered`, not pending, leased, submitted, failed, or ambiguous;
+- a linked covered image row and reservation contain an exact valid version-one plan;
+- the descriptor digest, kind, declared bytes, and semantic media class match;
+- the old coverage key is exactly the legacy hash derived from that plan; and
+- the scan fingerprint and every scan-affecting archive semantic equal the requested version-two
+  execution policy.
+
+The alias keeps the original successful reservation, plan, byte count, and completion provenance.
+Legacy rows are never rekeyed or deleted. If any compatibility proof is absent, the new namespace
+starts uncovered and normal fenced execution is required.
+
+Alternatives rejected:
+
+- Keeping selector limits in the coverage hash defeats global reuse whenever budgets are tuned.
+- Reusing every covered digest across scanner versions can suppress required rescans.
+- Bulk-rekeying legacy rows destroys provenance and races active leases.
+
+### 3. Select all-fit images and rank large-image payload deterministically
+
+Configuration remains independently eligible under its hard configuration-byte bound. Supported
+unique layer digests are evaluated under the hard per-layer bound. Covered digests in the matching
+execution namespace and duplicate positions in the same image are selected at zero new transfer
+bytes and zero new execution count.
+
+If every supported unique descriptor fits the configured aggregate bytes and unique-layer count,
+the selector selects all of them regardless of history class. This is the complete bounded path for
+small images and avoids reducing their coverage merely because history hints are absent.
+
+When the complete set does not fit, new unique layer candidates are sorted by:
+
+1. class priority: `copy_add`, `app_config_run`, `package_run`, `unknown`, `other_run`, `bulk_data`;
+2. highest manifest position first;
+3. smallest compressed descriptor first; and
+4. lexical digest as the final stable tie-break.
+
+The selector greedily admits candidates while both aggregate compressed-byte and unique-layer-count
+limits permit them. Hard per-descriptor limits are never exceeded. Each descriptor records one exact
+reason, including selected class, `already_covered`, `duplicate_digest`, `unsupported_media_type`,
+`config_too_large`, `layer_too_large`, `image_budget_exhausted`, or `layer_limit_exhausted`.
+Changing any class order, classifier rule, supported-media rule, or selection bound changes the
+selector hash.
+
+The selection algorithm receives a transactionally consistent coverage snapshot, but mutable
+coverage is not part of its identity. The first complete descriptor-position map is written before
+any new lease and reused exactly on later checkpoints. Consequently a layer skipped by the original
+budget never becomes newly selected merely because an earlier selected layer became globally
+covered.
+
+Alternatives rejected:
+
+- Fixed highest-first selection has already failed the completed-control recall gate.
+- A whole-image byte cutoff loses small application layers above giant data layers.
+- Selecting only recognized commands lets missing or unusual history hide useful payload.
+- Selecting globally covered content only when it still fits the current budget wastes verified
+  immutable evidence.
+
+### 4. Lease bounded multi-blob checkpoints
+
+The current executor and ingestion format already support more than one leased descriptor, but the
+binder leases one new digest and then defers the parent for 60 seconds. Adaptive execution leases a
+deterministic bounded batch from the frozen selected set under configurable maximum blob count and
+compressed bytes. A first eligible blob larger than the checkpoint-byte target but within its hard
+per-layer bound may be leased alone so it cannot starve indefinitely.
+
+Every digest still has its own advisory lock, lease token, attempt count, execution record, digest
+verification, and final state. The executor processes the batch sequentially inside the same owned
+slot and bundle. Ingestion may cover successful earlier blobs while returning a later retryable blob
+to pending. A crash before durable handoff covers none of the un-ingested batch and normal exact
+lease expiry/recovery applies.
+
+Checkpoint count, byte target, retry count, lease duration, and continuation delay are scheduling
+controls. They do not enter execution or selector hashes because they cannot turn an incomplete blob
+into successful coverage or change the frozen selected set.
+
+### 5. Keep image coverage explicit and policy-specific
+
+An image is complete only when its configuration and every manifest layer position are successfully
+covered under the requested execution policy. Reused and duplicate digests count as covered only
+after exact policy-compatible evidence exists. Any unsupported, oversized, budget-excluded, failed,
+or otherwise unselected descriptor makes the image bounded partial coverage.
+
+Selected retryable work keeps the parent deferred. Shared active work does not consume another blob
+attempt. Exhausted selected work produces terminal incomplete disposition. Findings from completed
+selected blobs retain image, digest, kind, class, and position provenance and use the existing
+authoritative ingestion, projection, and keycheck paths.
+
+Config history classification affects only selection order. It never changes detector output,
+finding authority, digest identity, or completion criteria.
+
+### 6. Add adaptive modes without changing legacy rollout semantics
+
+Existing `full`, timeout-only `canary`, and legacy `layer` meanings remain available for durable
+version-one work and rollback. Two explicit version-two modes are added:
+
+- `adaptive-canary` assigns a configured basis-point cohort across all immutable Docker manifests by
+  a stable versioned hash; cohort members use adaptive plans and non-members use full-image scanning.
+- `adaptive` uses adaptive plans for every eligible immutable Docker claim.
+
+Neither mode depends on a previous full-image timeout. Retry and checkpoint attempts for the same
+manifest and selector retain the same assignment. Invalid mode, policy, migration, or gate state
+fails closed to full-image execution before any adaptive plan is bound. Returning configuration to
+`full` changes only new claims and leaves adaptive plans and audit rows intact.
+
+The existing timeout-only canary remains independent and may continue while adaptive shadow evidence
+is gathered. The broad legacy `layer` mode remains operationally disabled because its selector did
+not pass recall gates.
+
+### 7. Gate rollout with non-authoritative aggregate shadow evidence
+
+An operator-invoked bounded shadow evaluator selects 50-100 exact immutable images whose authoritative
+full-image scans completed successfully under one scan fingerprint. It executes the candidate
+adaptive policy using the same downloader, validators, process containment, deadlines, and scanner
+fingerprint, but under a shadow authority that cannot call normal result ingestion or mutate target
+status, result reservations, global blob coverage, findings, keycheck candidates, projections, or
+source counters.
+
+For each paired control, full routed identities are read from the protected database as
+`(service, provider_key_hash)` and adaptive routed identities are derived in private memory through
+the same candidate normalization. Detector identities use `detector_secret_hash`. Identity sets,
+raw findings, commands, provider material, image names, and bearer tokens are discarded after
+intersection counts are computed.
+
+The durable report contains only policy hashes, cohort and completion counts, aggregate full,
+adaptive, and intersection counts, aggregate slot milliseconds, bounded failure counts, threshold
+results, and timestamps. It records no per-image row or identity. Slot timing uses the same outer
+monotonic boundary from admitted work through durable shadow sink completion for both paths; scan
+subprocess duration remains a diagnostic, not the gate denominator.
+
+A report passes only when:
+
+- 50-100 controls completed both paths without integrity, containment, or fence failure;
+- aggregate routed-identity recall, `intersection / full`, is at least 85%;
+- aggregate adaptive/full slot-time ratio is at most 40%;
+- every adaptive omission is represented in coverage counts; and
+- no credential persistence, quarantine, projection, source-failure, or resource-bound regression
+  is observed.
+
+Reports are bound to exact selector, execution, and scan policy hashes. Stale or incomplete reports
+cannot authorize another policy. Adaptive canary remains fail-closed until a matching report passes.
+Broad `adaptive` enablement additionally requires a stable low-percentage production canary over at
+least one repository-refresh interval. Operators change rollout configuration explicitly; shadow
+evidence never changes execution mode by itself.
+
+Alternatives rejected:
+
+- Routing shadow candidates through keycheck would make the experiment authoritative and consume
+  external capacity.
+- Persisting per-image shadow identities creates unnecessary sensitive correlation data.
+- Comparing only detector counts does not measure the routed identities the scanner is intended to
+  produce.
+- Automatically enabling adaptive mode from a report removes the operational rollback checkpoint.
+
+### 8. Preserve incomplete warning semantics without redundant retries
+
+The first completed 50-control production shadow report failed closed. Routed recall was 12 of 18
+identities (66.7%), adaptive/full slot time was 46.6%, and the report recorded 43 aggregate
+failures. Its selection evidence showed 230 descriptors omitted by the eight-layer limit and 14
+oversized descriptors, so selector recall remains the primary rollout blocker.
+
+The same evidence exposed a separate execution defect. TruffleHog diagnostics such as
+`chunk_processing` and `detector_timeout` are explicitly deterministic, non-retryable warnings.
+Their findings must be retained, but the affected blob cannot establish complete coverage. The
+diagnostic adapter previously discarded the non-retryable bit, causing the layer executor to
+download and scan the same incomplete blob up to three times before reaching the same terminal
+state. The adapter now preserves aggregate warning retryability and the layer executor terminates
+that blob after the first deterministic warning. It does not mark the blob covered or remove the
+report failure.
+
+The historical report schema retained only a total failure count, so its 43 failures cannot be
+decomposed exactly after the fact. Future shadow runs keep a fixed allow-list of aggregate-only
+failure categories in protected memory and print their totals in the final operator summary without
+changing report authority or persisting target-level evidence.
+
+Shadow execution also suppresses target labels in finding-filter logs. Production scans retain their
+existing target logging, while both private full and private layer paths emit only aggregate filter
+counts. This closes a privacy gap found in the first report log without weakening normal operational
+diagnostics.
+
+## Risks / Trade-offs
+
+- [History is malformed, misleading, or secret-bearing] -> Bound and validate it, persist only an
+  enum, fall back to `unknown`, and keep identity/coverage independent of classification.
+- [Application secrets exist in a low-priority or giant layer] -> Scan all-fit images, keep unknown
+  ahead of generic/bulk classes, record partial scope, retain full controls, and enforce the 85%
+  routed-recall gate.
+- [Unsafe policy reuse suppresses a required rescan] -> Separate hashes and permit legacy aliasing
+  only from exact successful compatible evidence in one fenced transaction.
+- [Selector changes expand coverage during retry] -> Freeze the earliest full position map under the
+  selector hash before leases are issued.
+- [Multi-blob checkpoints increase work lost on crash] -> Bound count/bytes and retain independent
+  per-blob leases and ingestion records; no pre-handoff result becomes covered.
+- [Config prefetch adds Registry traffic] -> Reuse the already bounded authenticated downloader and
+  avoid a second fetch when the config is leased in the same plan.
+- [Individual layer scans expose whiteouted historical files] -> Preserve position provenance and
+  describe evidence as image-content coverage, not merged-root state.
+- [Shadow evaluation consumes slots] -> Keep it operator-invoked, bounded, deterministic, and
+  subject to the existing slot/resource controls.
+- [Aggregate reports hide individual anomalies] -> Fail the whole report on incomplete paired work
+  and retain bounded failure counts without persisting target identity.
+- [Deterministic scanner warnings consume repeated transfer and slot time] -> Preserve their
+  non-retryable policy, keep findings and incomplete coverage, and terminate the blob on its first
+  bounded attempt.
+
+## Migration Plan
+
+1. Ship version-one and version-two validators, new modes, and migration code while production stays
+   on its existing timeout-only canary.
+2. Stop authoritative runtime and verify no active result, queue, or blob leases remain.
+3. Add the selector-policy coverage column, aggregate shadow-report state, required timing/fence
+   columns, indexes, and a new migration marker.
+4. Backfill every historical image-coverage row from its exact valid reservation plan. Abort and
+   roll back the migration transaction on an orphan, malformed plan, or invalid hash; then make the
+   column non-null and run exact schema validation.
+5. Restart with unchanged mode and verify legacy claims, projection, keycheck, resolver, quarantine,
+   and source health before creating version-two work.
+6. Run the private shadow evaluator for 50-100 completed controls. Keep adaptive modes fail-closed if
+   the matching report misses recall, timing, safety, or completion gates.
+7. Enable a low deterministic `adaptive-canary`, monitor at least one repository-refresh interval,
+   and compare source failures, coverage reasons, routed yield, slot time, and quarantine.
+8. Increase canary basis points and finally enable `adaptive` only after every gate remains satisfied.
+9. Roll back immediately by setting mode to `full` or the existing timeout-only `canary`. Keep all
+   version-two plans, reports, and coverage rows for audit and exact future resume.
+
+## Open Questions
+
+- Which initial bounded checkpoint count and byte target provide the best reduction in continuation
+  delay without increasing crash rework materially?
+- Which classifier allow-list revisions improve routed recall in the first 50-100 controls? Every
+  revision will receive a new selector-policy hash rather than changing an existing policy.
+- What adaptive-canary basis-point sequence should operators use after the shadow gate passes?

openspec/changes/add-adaptive-docker-payload-scanning/proposal.md

@@ -0,0 +1,29 @@
+## Why
+
+The timeout-only Docker layer fallback cuts heavy-image work to roughly 5-6% of full-image time, but its fixed highest-eight-layer policy retained only 21.1% of routed identities on completed controls. Broad Docker throughput therefore remains limited by indivisible full-image scans, while enabling the existing bounded layer selector globally would lose too much useful coverage.
+
+## What Changes
+
+- Add a deterministic adaptive Docker payload policy that scans all eligible content for small images and prioritizes application-bearing content for large images using bounded, untrusted config-history hints.
+- Reuse successfully covered immutable blobs across images under a scan-execution policy independent of selector budgets, without weakening digest, reservation, lease, plan, or ingestion fences.
+- Give every adaptive plan a versioned selector identity and freeze its first selection across checkpoints and retries.
+- Record explicit reasons and bytes for every selected, reused, unsupported, oversized, and budget-excluded descriptor; adaptive completion remains partial whenever any content is omitted.
+- Add non-authoritative shadow evaluation and aggregate privacy-preserving evidence for completed full-image controls.
+- Keep full-image scanning as the default and rollback path; permit broad adaptive rollout only after a 50-100 image control cohort retains at least 85% routed-identity recall while using at most 40% of full-image slot time.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `docker-layer-content-scanning`: Replace fixed highest-first broad selection with versioned adaptive payload selection, separate selector identity from reusable execution evidence, and require non-authoritative shadow gates before broad rollout.
+
+## Impact
+
+- Affects Docker Registry config access, post-claim mode assignment, layer-plan binding, policy hashing, image-to-blob coverage state, aggregate rollout evidence, Docker configuration, and related PostgreSQL migration/runtime validation.
+- Reuses the existing immutable digest identities, account pool, bounded Registry downloader, global blob leases, scan slots, Windows Job containment, result bundles, findings projection, and keycheck pipeline.
+- Does not change detector classification, credential persistence, Git or Hugging Face scanning, guaranteed scan-slot capacity, Docker worker count, or repository resolver scheduling.
+- Requires an additive stopped-runtime migration before adaptive execution can be enabled; rollback leaves durable adaptive audit state intact.

openspec/changes/add-adaptive-docker-payload-scanning/specs/docker-layer-content-scanning/spec.md

@@ -0,0 +1,218 @@
+## MODIFIED Requirements
+
+### Requirement: Immutable image content plans are bound after claim
+The system SHALL resolve a claimed Docker image's exact platform manifest into a canonical bounded plan containing its configuration and ordered layer descriptors, and SHALL bind that plan to the active result reservation before content execution. Adaptive plans SHALL use an exactly validated version-two schema that binds versioned selector and execution-policy identities plus one bounded payload class per descriptor while retaining exact validation of durable version-one plans.
+
+#### Scenario: Valid immutable manifest
+- **WHEN** the claimed `repository@sha256:<digest>` resolves to a valid requested-platform manifest
+- **THEN** the bound plan identifies the same manifest digest and contains only bounded valid SHA-256 content descriptors, sizes, media types, order, policy hashes, classes, and selection reasons
+
+#### Scenario: Changed replay
+- **WHEN** the same reservation attempts to bind a different content plan
+- **THEN** the system rejects the replay as a fenced conflict and executes neither plan
+
+#### Scenario: Invalid or oversized manifest
+- **WHEN** the Registry manifest is malformed, exceeds descriptor bounds, or disagrees with the immutable target
+- **THEN** the system fails closed without claiming complete content coverage
+
+#### Scenario: Valid aligned configuration history
+- **WHEN** a bounded digest-verified configuration has non-empty history entries aligned base-to-top with every manifest layer
+- **THEN** the selector classifies each layer through the versioned bounded classifier and persists only its class enum
+
+#### Scenario: Untrusted or misaligned configuration history
+- **WHEN** configuration history is absent, malformed, oversized, or inconsistent with layer or rootfs counts
+- **THEN** every affected layer is deterministically classified `unknown` and no raw history command is persisted or logged
+
+#### Scenario: Durable version-one replay
+- **WHEN** recovery loads a previously bound valid version-one plan
+- **THEN** the system validates and executes it under its original exact schema without rewriting it as version two
+
+### Requirement: Content selection is bounded and application-first
+The system SHALL select bounded image configuration and SHALL deterministically select supported unique layers under configured per-layer, aggregate compressed-byte, and unique-layer-count limits. It SHALL select every eligible unique descriptor when the complete set fits, and for larger images SHALL prioritize versioned payload classes before manifest position and stable tie-breaks.
+
+#### Scenario: Complete bounded image fits
+- **WHEN** configuration and every supported unique layer fit all configured descriptor, aggregate-byte, and unique-layer-count bounds
+- **THEN** the system selects every descriptor regardless of history class
+
+#### Scenario: Large image requires prioritization
+- **WHEN** all new unique layers cannot fit the configured aggregate bounds
+- **THEN** the system greedily considers `copy_add`, `app_config_run`, `package_run`, `unknown`, `other_run`, and `bulk_data` in that order, then highest position, smallest compressed size, and lexical digest
+
+#### Scenario: Giant base or model layer
+- **WHEN** a layer exceeds the configured per-layer limit
+- **THEN** the layer is not downloaded by the normal layer scanner and coverage records `layer_too_large`
+
+#### Scenario: Image byte budget is exhausted
+- **WHEN** another unscanned layer would exceed the remaining per-image budget
+- **THEN** the layer remains unselected and coverage records `image_budget_exhausted`
+
+#### Scenario: Unique layer count is exhausted
+- **WHEN** another unscanned unique layer would exceed the configured selected-layer count
+- **THEN** the layer remains unselected and coverage records `layer_limit_exhausted`
+
+#### Scenario: Shared layer is already covered
+- **WHEN** a layer digest has successful coverage under the matching execution policy
+- **THEN** the image reuses that coverage without consuming its transfer-byte or new-execution-count budget
+
+#### Scenario: Duplicate positions share one digest
+- **WHEN** multiple positions in one manifest reference the same eligible immutable digest
+- **THEN** the system selects all matching positions but budgets and executes that digest at most once
+
+#### Scenario: Selector input changes
+- **WHEN** a classifier rule, class order, supported-media rule, or selection bound changes
+- **THEN** the system derives a different versioned `selection_policy_sha256`
+
+### Requirement: Layer coverage is globally deduplicated and fenced
+The system SHALL maintain one authoritative content-scan state per immutable digest and scan-execution policy and SHALL change successful coverage only through matching reservation, lease, plan, and ingestion fences. Selection budgets, classifier weights, retries, lease timing, and checkpoint scheduling MUST NOT partition otherwise identical successful execution evidence.
+
+#### Scenario: Concurrent images share a layer
+- **WHEN** two image plans reference the same unscanned digest under the same execution policy concurrently
+- **THEN** at most one reservation owns its active scan and the other image records shared pending work without duplicate execution
+
+#### Scenario: Selector policy changes only
+- **WHEN** an immutable digest was covered successfully and a later plan changes only selector or scheduling policy
+- **THEN** the later plan reuses the existing execution-policy coverage without launching another scan
+
+#### Scenario: Execution semantics change
+- **WHEN** the scanner fingerprint, content validator, or scan-affecting archive semantics differ
+- **THEN** the system uses a distinct execution-policy namespace and does not reuse incompatible successful coverage
+
+#### Scenario: Compatible legacy coverage exists
+- **WHEN** an exact valid version-one plan proves a linked digest is durably covered with execution semantics identical to the requested version-two policy
+- **THEN** the binding transaction may create a covered version-two alias that retains the original successful reservation, plan, byte count, and completion provenance
+
+#### Scenario: Legacy evidence is incomplete or ambiguous
+- **WHEN** legacy evidence is pending, leased, submitted, failed, orphaned, malformed, or not exactly execution-compatible
+- **THEN** the system does not alias it and requires normal fenced execution under the new policy
+
+#### Scenario: Successful matching ingestion
+- **WHEN** a result bundle contains a successful execution for a blob leased by its exact bound plan
+- **THEN** ingestion marks that digest globally covered for the matching execution policy in the same durable transaction
+
+#### Scenario: Stale completion
+- **WHEN** a bundle or worker presents an expired, refunded, or mismatched blob lease
+- **THEN** it cannot mark the digest covered or advance image coverage
+
+#### Scenario: Reservation is refunded
+- **WHEN** an image reservation is durably refunded before handoff
+- **THEN** only blob leases owned by that reservation are released for bounded reclamation
+
+### Requirement: Image coverage is explicit and honest
+The system SHALL persist and expose selected, covered, shared-pending, failed, and intentionally skipped content for each immutable image plan, including bounded payload class, exact reason, and execution and selector policy identities. Configuration history classification SHALL influence priority only and SHALL NOT establish content identity or successful coverage.
+
+#### Scenario: Every descriptor is covered
+- **WHEN** configuration and all image layer positions have successful compatible execution-policy coverage
+- **THEN** the image records complete content coverage
+
+#### Scenario: Bounds skip content
+- **WHEN** one or more descriptors are excluded by configured size, budget, count, or format bounds
+- **THEN** the image may finish as bounded partial coverage but SHALL NOT report complete content coverage
+
+#### Scenario: Adaptive priority skips content
+- **WHEN** a large-image descriptor loses deterministic selection to a higher-priority candidate
+- **THEN** its class, declared bytes, omission reason, and partial image scope remain durable without persisting raw history
+
+#### Scenario: Selected content remains retryable
+- **WHEN** at least one selected blob failed retryably or is actively covered by another reservation
+- **THEN** the image remains deferred without claiming complete coverage
+
+#### Scenario: Selected content exhausts retries
+- **WHEN** required selected content reaches its terminal attempt limit
+- **THEN** the image receives terminal incomplete disposition with durable coverage detail
+
+#### Scenario: Covered duplicate appears at multiple positions
+- **WHEN** one successfully covered digest backs multiple positions in an image
+- **THEN** every matching position records compatible covered scope without duplicate execution
+
+### Requirement: Layer work resumes without repeating completed content
+The system SHALL resume an incomplete image from its earliest complete descriptor-position selection map for the same manifest and selector policy, SHALL NOT expand or contract that selection because mutable coverage changed, and SHALL NOT relaunch content with compatible successful execution-policy coverage. It MAY lease a bounded deterministic batch while retaining independent per-blob fences.
+
+#### Scenario: Parent image retries
+- **WHEN** an image retry follows partial layer completion
+- **THEN** the new plan reuses the exact frozen selection, reuses compatible covered digests, and leases only remaining eligible content
+
+#### Scenario: Covered content changes the available budget
+- **WHEN** a selected digest becomes globally covered after the first plan was bound
+- **THEN** a descriptor skipped by the original byte or count budget remains skipped on every later checkpoint under that selector policy
+
+#### Scenario: Execution policy changes
+- **WHEN** the same frozen selector baseline runs under a new incompatible execution policy
+- **THEN** its selected positions remain unchanged while only content lacking compatible coverage becomes executable
+
+#### Scenario: Bounded multi-blob checkpoint
+- **WHEN** multiple remaining selected digests fit the configured checkpoint count and byte targets
+- **THEN** one reservation may lease that deterministic batch while each digest retains an independent lease token, attempt, execution record, and ingestion transition
+
+#### Scenario: Process crashes after one layer
+- **WHEN** one layer was durably ingested before a later checkpoint or parent process failed
+- **THEN** recovery preserves the completed layer and reclaims only unfinished leased content
+
+#### Scenario: Batch fails before durable handoff
+- **WHEN** a process scans one or more leased blobs but crashes before their result bundle is durably accepted
+- **THEN** none of those un-ingested blobs becomes covered and exact lease recovery remains required
+
+### Requirement: Full-image compatibility and deterministic canary are retained
+The system SHALL retain the existing full-image scanner, timeout-only canary, and legacy layer path behind configuration. It SHALL additionally provide deterministic `adaptive-canary` and `adaptive` version-two modes, fail closed to full execution without a matching passed rollout gate, and preserve full-image rollback without deleting durable layer state.
+
+#### Scenario: Full mode
+- **WHEN** Docker layer mode is disabled or set to `full`
+- **THEN** the existing immutable full-image execution path remains authoritative
+
+#### Scenario: Legacy canary retry
+- **WHEN** an image with a durable previous full-image timeout is retried under the existing canary
+- **THEN** its immutable manifest digest selects the same legacy scanner mode as its previous attempt
+
+#### Scenario: Non-timeout image during legacy canary rollout
+- **WHEN** an image has no durable previous full-image command timeout
+- **THEN** existing canary configuration keeps that image on the full-image execution path
+
+#### Scenario: Adaptive canary assignment
+- **WHEN** a matching passed shadow gate permits `adaptive-canary`
+- **THEN** a versioned stable hash of every immutable manifest identity selects the same configured adaptive cohort across retries while non-members remain full-image controls
+
+#### Scenario: Adaptive gate is absent or stale
+- **WHEN** adaptive configuration lacks a completed passing report for its exact scan, execution, and selector policy hashes
+- **THEN** no adaptive plan is bound and the claim uses full-image execution
+
+#### Scenario: Broad adaptive mode
+- **WHEN** matching shadow evidence passes and the low-percentage production canary remains within safety gates for at least one repository-refresh interval
+- **THEN** operators may explicitly configure `adaptive` for all eligible immutable Docker claims
+
+#### Scenario: Layer mode rollback
+- **WHEN** operators return configuration from `layer`, `canary`, `adaptive-canary`, or `adaptive` to `full`
+- **THEN** new claims use full-image execution without deleting durable layer plans, coverage, or rollout evidence
+
+### Requirement: Controlled evidence gates production rollout
+The system SHALL compare adaptive scanning with completed full-image controls through a bounded non-authoritative evaluator and SHALL keep adaptive production modes fail-closed until a policy-exact aggregate report passes security, coverage, completion, and throughput gates. Shadow execution MUST NOT mutate authoritative queue, reservation, coverage, finding, candidate, keycheck, projection, or source-counter state.
+
+#### Scenario: Controlled adaptive shadow cohort
+- **WHEN** the evaluator runs against 50-100 exact immutable images with completed full-image controls under one scan fingerprint
+- **THEN** it executes the candidate adaptive policy under the same bounded scanner semantics and records only aggregate policy hashes, counts, slot milliseconds, failures, thresholds, and timestamps
+
+#### Scenario: Shadow identity comparison
+- **WHEN** routed and detector recall are calculated
+- **THEN** `(service, provider_key_hash)` and `detector_secret_hash` sets exist only in protected memory long enough to calculate aggregate full, adaptive, and intersection counts
+
+#### Scenario: Shadow privacy
+- **WHEN** a shadow run completes, fails, or logs diagnostics
+- **THEN** no target name, raw config command, finding, secret, provider material, identity set, or Registry bearer is persisted in rollout evidence or ordinary logs
+
+#### Scenario: Shadow non-authority
+- **WHEN** shadow execution emits candidate material or completes a blob
+- **THEN** it cannot create authoritative findings or candidates, route keychecks, mark global blob coverage, alter image disposition, or change production mode
+
+#### Scenario: Routed recall or slot-time gate fails
+- **WHEN** paired routed-identity recall is below 85% or aggregate adaptive slot time exceeds 40% of aggregate full slot time
+- **THEN** the report does not authorize adaptive production execution
+
+#### Scenario: Safety or completion gate fails
+- **WHEN** fewer than 50 paired controls complete, any digest/fence/containment requirement fails, omissions are unaccounted, or resource, quarantine, projection, credential, or source health regresses
+- **THEN** the report does not authorize adaptive production execution
+
+#### Scenario: Policy changes after a passing report
+- **WHEN** scan, execution, or selector policy identity changes
+- **THEN** the earlier report is stale and adaptive execution remains fail-closed until the new exact policy passes another shadow cohort
+
+#### Scenario: Acceptance criteria pass
+- **WHEN** 50-100 paired controls retain at least 85% routed identities at no more than 40% slot time with every safety gate satisfied
+- **THEN** operators may start a low deterministic adaptive canary but SHALL NOT enable broad adaptive mode until that canary remains stable for at least one repository-refresh interval

openspec/changes/add-adaptive-docker-payload-scanning/tasks.md

@@ -0,0 +1,46 @@
+## 1. Add Versioned Adaptive Policy Inputs
+
+- [x] 1.1 Add fail-closed `adaptive-canary` and `adaptive` configuration/CLI parsing, bounded selector and checkpoint settings, and stable immutable-manifest assignment
+- [x] 1.2 Separate scanner, execution, and selector policy hashes so selection and scheduling changes do not partition compatible successful blob coverage
+- [x] 1.3 Fetch and verify bounded image configuration, map aligned non-empty history to ordered layers, and emit only versioned bounded payload classes with deterministic `unknown` fallback
+- [x] 1.4 Add exact version-two plan validation and canonical hashing while retaining unchanged version-one plan and execution validation
+
+## 2. Migrate Durable Policy And Evidence State
+
+- [x] 2.1 Add and exactly validate the image-coverage selector-policy column, policy-specific baseline index, aggregate shadow-report state, timing/fence fields, and migration marker
+- [x] 2.2 Backfill selector identities transactionally from exact linked version-one plans and fail the migration on orphaned, malformed, or ambiguous coverage rows
+- [x] 2.3 Implement transactionally fenced legacy-to-execution-policy coverage aliasing only for exact successful semantically compatible evidence
+- [x] 2.4 Add stopped-runtime migration preconditions and prove rollback leaves all legacy plans and coverage rows intact
+
+## 3. Implement Adaptive Selection And Resume
+
+- [x] 3.1 Select all supported unique descriptors for complete bounded images and implement deterministic class, position, size, and digest ordering for larger images
+- [x] 3.2 Persist exact classes and selected, reused, duplicate, unsupported, oversized, byte-budget, and count-budget reasons with honest partial coverage
+- [x] 3.3 Freeze the earliest complete descriptor-position map by queue, manifest, and selector policy across checkpoints, retries, and execution-policy changes
+- [x] 3.4 Lease deterministic bounded multi-blob checkpoints while retaining independent digest locks, lease tokens, attempts, execution records, and reclaim behavior
+- [x] 3.5 Execute and ingest version-two plans with mixed per-blob outcomes, immutable class/provenance metadata, and no repeat work for compatible covered digests
+
+## 4. Add Non-Authoritative Shadow Gates
+
+- [x] 4.1 Implement a bounded operator-invoked paired evaluator for 50-100 completed full-image controls using the exact candidate scan, execution, and selector policies
+- [x] 4.2 Derive routed and detector identity intersections only in protected memory and persist only aggregate counts, slot timing, failures, thresholds, policy hashes, and timestamps
+- [x] 4.3 Fence shadow execution from queue disposition, reservations, global coverage, findings, candidates, keychecks, projections, source counters, and automatic mode changes
+- [x] 4.4 Require a matching completed report with routed recall at least 85% and adaptive/full slot ratio at most 40% before adaptive canary assignment
+- [x] 4.5 Expose aggregate adaptive selection, reuse, omission, checkpoint, completion, slot-time, and gate metrics without target or secret material
+
+## 5. Verify Safety And Behavior
+
+- [x] 5.1 Add unit tests for bounded history parsing, secret-free class persistence, all-fit selection, large-image ranking, stable hashes, exact reasons, and malformed-plan rejection
+- [x] 5.2 Add PostgreSQL tests for atomic migration/backfill, selector-frozen resume, compatible coverage aliasing, incompatible policy partitioning, concurrent deduplication, and stale fences
+- [x] 5.3 Add checkpoint tests proving bounded multi-blob mixed outcomes, crash recovery, independent attempts, and no post-coverage selection expansion
+- [x] 5.4 Add rollout tests for stable adaptive canary assignment, stale/missing gate fallback, shadow non-authority/privacy, aggregate recall, and claim-through-handoff slot timing
+- [x] 5.5 Run targeted unit, runtime-safety, migration, query-shape, and real PostgreSQL integration suites plus strict OpenSpec validation
+- [x] 5.6 Preserve deterministic warning retryability so incomplete findings remain visible without repeated blob downloads or false successful coverage
+- [x] 5.7 Suppress target labels in private shadow filter logs and expose only fixed aggregate failure categories for future evidence
+
+## 6. Migrate And Roll Out Conservatively
+
+- [x] 6.1 Stop runtime, verify lease quiescence, apply the additive migration, restart in unchanged timeout-only canary mode, and verify source/keycheck/projection/quarantine health
+- [ ] 6.2 Run the aggregate-only shadow evaluator on 50-100 completed controls and keep adaptive execution disabled unless every recall, timing, completion, privacy, and safety gate passes
+- [ ] 6.3 Enable a low deterministic adaptive canary only after a matching passing report and monitor it for at least one repository-refresh interval
+- [ ] 6.4 Expand canary or enable broad adaptive mode only if runtime gates remain satisfied; otherwise return new claims to full or timeout-only canary without deleting audit state