truf-server/openspec/changes/add-adaptive-docker-payload-scanning/design.md

Context

Docker scanning currently has two materially different paths. The full-image TruffleHog source is authoritative for new and normally completing images, but it treats an immutable image as one indivisible operation. The bounded layer path can resume and globally reuse content digests, but its fixed highest-eight-layer selector was admitted only for prior full-image timeouts. In the completed control cohort it retained 21.1% of routed identities and 5.8% of detector identities, so enabling that selector broadly would trade too much useful coverage for speed.

The layer path already provides the expensive safety primitives this change needs: exact manifest resolution, authenticated bounded Registry transfer, digest verification, private artifacts, contained TruffleHog filesystem execution, canonical reservation-bound plans, policy-scoped global blob leases, fenced result ingestion, and explicit per-image coverage. The missing pieces are a content-aware selector, a reusable execution-policy identity that is independent of selector budgets, and trustworthy evidence for deciding whether the adaptive path is safe to broaden.

Docker/OCI configuration contains an ordered history list that can help identify COPY, ADD, application setup, package installation, generic RUN, and likely bulk-data layers. That history is untrusted and may itself contain secret material. It is therefore only a bounded selection hint; it must never become an authority for content identity or successful coverage and its raw commands must never be persisted or logged.

The production database already contains durable version-one layer plans and covered blob rows. Changing their interpretation in place would invalidate audit and lease fences. The migration must be additive, retain version-one validation, and allow only explicitly proven compatible successful coverage to enter the new execution namespace.

Goals / Non-Goals

Goals:

• Scan all supported unique content for images that fit conservative bounds.
• Prioritize likely application, configuration, and source-bearing layers when a large image cannot fit those bounds.
• Reuse successful immutable blob coverage across images and selector revisions when execution semantics are unchanged.
• Freeze the first deterministic selection for an image and selector policy across every checkpoint, retry, and execution-policy transition.
• Reduce per-image checkpoint overhead with bounded multi-blob leases without weakening per-blob execution and ingestion fences.
• Preserve exact selected, reused, skipped, failed, and partial coverage semantics.
• Produce private, aggregate, non-authoritative shadow evidence against 50-100 completed full-image controls before any broad adaptive rollout.
• Retain full-image scanning and the existing timeout-only layer canary as immediate rollback paths.

Non-Goals:

• Infer arbitrary file contents without downloading a compressed layer.
• Claim complete coverage for an image with unsupported, oversized, failed, or budget-excluded descriptors.
• Persist raw Docker config history, shadow findings, provider keys, target names, or Registry bearer tokens in rollout evidence.
• Change detector classification, keycheck routing, Docker account ownership, repository resolver scheduling, guaranteed scan-slot capacity, worker count, or non-Docker scanners.
• Reconstruct a merged container filesystem or remove historical whiteout content from individual layer evidence.
• Destructively rewrite or delete version-one plans and coverage rows.

Decisions

1. Introduce a version-two immutable content plan

Adaptive execution uses a canonical version-two plan. It retains the version-one image, repository, manifest, platform, media, limits, scan-policy, descriptor, reservation, and plan-hash fences and adds:

• a versioned selector algorithm and selection_policy_sha256;
• an execution_policy_sha256 for reusable successful blob evidence;
• one bounded classifier value per descriptor;
• exact selection and omission reasons; and
• bounded checkpoint lease limits that do not alter the frozen selection set.

Version-one plan and execution validators remain exact because their JSON is already durable. Version-two validation dispatches by the exact integer version and rejects unknown fields, unknown classes, malformed hashes, descriptor reordering, and oversized canonical JSON. A reservation can bind one canonical plan only; idempotent replay must be byte-identical.

After exact manifest resolution and before plan binding, the scanner fetches the configuration blob through the existing bounded, authenticated, digest-verifying Registry path. It parses the JSON in bounded private memory/storage and maps non-empty_layer history entries from base to top onto the ordered manifest layers. The optional rootfs.diff_ids count and the non-empty history count must agree with the layer count. A missing field, malformed value, excess entry count, or alignment mismatch classifies every layer as unknown; it does not change descriptor identity or fail a valid immutable manifest.

The classifier uses a small versioned allow-list and emits only these bounded classes: config, copy_add, app_config_run, package_run, other_run, bulk_data, and unknown. Raw created_by values are discarded before plan construction and are excluded from logs, result metadata, errors, and database rows.

Alternatives rejected:

• Persisting normalized command text would retain unnecessary secret-bearing input.
• Treating history as authoritative would let malformed or adversarial metadata hide content.
• Mutating version-one plans would break exact replay and auditability.

2. Separate scan, execution, and selection identities

Three hashes have distinct responsibilities:

• scan_policy_sha256 remains the installed scanner/detector/config fingerprint.
• execution_policy_sha256 hashes the scan policy plus versioned content validation and all archive semantics that can change which bytes a successful command examines. It excludes image/layer selection budgets, classifier weights, retry counts, lease duration, checkpoint size, and delay.
• selection_policy_sha256 hashes the selector version, class ordering, deterministic tie-breaks, supported descriptor classes, and all limits that determine the initial selected set. It excludes mutable global coverage and execution scheduling.

For version-two rows, the existing docker_content_blobs.coverage_policy_sha256 key stores the execution-policy hash. Selector changes therefore do not force an identical successfully scanned digest through TruffleHog again, while archive or detector semantic changes still create a separate coverage namespace.

docker_image_blob_coverage receives a non-null selection_policy_sha256. Existing rows are backfilled from the exact bound plan on their linked reservation and indexed by (queue_id, manifest_digest, selection_policy_sha256, position, reservation_id). The earliest full position map under that key is the immutable selection baseline. Coverage-policy changes may require new execution but cannot expand or contract that baseline silently.

Successfully covered version-one evidence may be copied lazily into the version-two execution namespace only in the same transaction that validates all of the following:

• the old row is durably covered, not pending, leased, submitted, failed, or ambiguous;
• a linked covered image row and reservation contain an exact valid version-one plan;
• the descriptor digest, kind, declared bytes, and semantic media class match;
• the old coverage key is exactly the legacy hash derived from that plan; and
• the scan fingerprint and every scan-affecting archive semantic equal the requested version-two execution policy.

The alias keeps the original successful reservation, plan, byte count, and completion provenance. Legacy rows are never rekeyed or deleted. If any compatibility proof is absent, the new namespace starts uncovered and normal fenced execution is required.

Alternatives rejected:

• Keeping selector limits in the coverage hash defeats global reuse whenever budgets are tuned.
• Reusing every covered digest across scanner versions can suppress required rescans.
• Bulk-rekeying legacy rows destroys provenance and races active leases.

3. Select all-fit images and rank large-image payload deterministically

Configuration remains independently eligible under its hard configuration-byte bound. Supported unique layer digests are evaluated under the hard per-layer bound. Covered digests in the matching execution namespace and duplicate positions in the same image are selected at zero new transfer bytes and zero new execution count.

If every supported unique descriptor fits the configured aggregate bytes and unique-layer count, the selector selects all of them regardless of history class. This is the complete bounded path for small images and avoids reducing their coverage merely because history hints are absent.

When the complete set does not fit, new unique layer candidates are sorted by:

1. class priority: copy_add, app_config_run, package_run, unknown, other_run, bulk_data;
2. highest manifest position first;
3. smallest compressed descriptor first; and
4. lexical digest as the final stable tie-break.

The selector greedily admits candidates while both aggregate compressed-byte and unique-layer-count limits permit them. Hard per-descriptor limits are never exceeded. Each descriptor records one exact reason, including selected class, already_covered, duplicate_digest, unsupported_media_type, config_too_large, layer_too_large, image_budget_exhausted, or layer_limit_exhausted. Changing any class order, classifier rule, supported-media rule, or selection bound changes the selector hash.

The selection algorithm receives a transactionally consistent coverage snapshot, but mutable coverage is not part of its identity. The first complete descriptor-position map is written before any new lease and reused exactly on later checkpoints. Consequently a layer skipped by the original budget never becomes newly selected merely because an earlier selected layer became globally covered.

Alternatives rejected:

• Fixed highest-first selection has already failed the completed-control recall gate.
• A whole-image byte cutoff loses small application layers above giant data layers.
• Selecting only recognized commands lets missing or unusual history hide useful payload.
• Selecting globally covered content only when it still fits the current budget wastes verified immutable evidence.

4. Lease bounded multi-blob checkpoints

The current executor and ingestion format already support more than one leased descriptor, but the binder leases one new digest and then defers the parent for 60 seconds. Adaptive execution leases a deterministic bounded batch from the frozen selected set under configurable maximum blob count and compressed bytes. A first eligible blob larger than the checkpoint-byte target but within its hard per-layer bound may be leased alone so it cannot starve indefinitely.

Every digest still has its own advisory lock, lease token, attempt count, execution record, digest verification, and final state. The executor processes the batch sequentially inside the same owned slot and bundle. Ingestion may cover successful earlier blobs while returning a later retryable blob to pending. A crash before durable handoff covers none of the un-ingested batch and normal exact lease expiry/recovery applies.

Checkpoint count, byte target, retry count, lease duration, and continuation delay are scheduling controls. They do not enter execution or selector hashes because they cannot turn an incomplete blob into successful coverage or change the frozen selected set.

5. Keep image coverage explicit and policy-specific

An image is complete only when its configuration and every manifest layer position are successfully covered under the requested execution policy. Reused and duplicate digests count as covered only after exact policy-compatible evidence exists. Any unsupported, oversized, budget-excluded, failed, or otherwise unselected descriptor makes the image bounded partial coverage.

Selected retryable work keeps the parent deferred. Shared active work does not consume another blob attempt. Exhausted selected work produces terminal incomplete disposition. Findings from completed selected blobs retain image, digest, kind, class, and position provenance and use the existing authoritative ingestion, projection, and keycheck paths.

Config history classification affects only selection order. It never changes detector output, finding authority, digest identity, or completion criteria.

6. Add adaptive modes without changing legacy rollout semantics

Existing full, timeout-only canary, and legacy layer meanings remain available for durable version-one work and rollback. Two explicit version-two modes are added:

• adaptive-canary assigns a configured basis-point cohort across all immutable Docker manifests by a stable versioned hash; cohort members use adaptive plans and non-members use full-image scanning.
• adaptive uses adaptive plans for every eligible immutable Docker claim.

Neither mode depends on a previous full-image timeout. Retry and checkpoint attempts for the same manifest and selector retain the same assignment. Invalid mode, policy, migration, or gate state fails closed to full-image execution before any adaptive plan is bound. Returning configuration to full changes only new claims and leaves adaptive plans and audit rows intact.

The existing timeout-only canary remains independent and may continue while adaptive shadow evidence is gathered. The broad legacy layer mode remains operationally disabled because its selector did not pass recall gates.

7. Gate rollout with non-authoritative aggregate shadow evidence

An operator-invoked bounded shadow evaluator selects 50-100 exact immutable images whose authoritative full-image scans completed successfully under one scan fingerprint. It executes the candidate adaptive policy using the same downloader, validators, process containment, deadlines, and scanner fingerprint, but under a shadow authority that cannot call normal result ingestion or mutate target status, result reservations, global blob coverage, findings, keycheck candidates, projections, or source counters.

For each paired control, full routed identities are read from the protected database as (service, provider_key_hash) and adaptive routed identities are derived in private memory through the same candidate normalization. Detector identities use detector_secret_hash. Identity sets, raw findings, commands, provider material, image names, and bearer tokens are discarded after intersection counts are computed.

The durable report contains only policy hashes, cohort and completion counts, aggregate full, adaptive, and intersection counts, aggregate slot milliseconds, bounded failure counts, threshold results, and timestamps. It records no per-image row or identity. Slot timing uses the same outer monotonic boundary from admitted work through durable shadow sink completion for both paths; scan subprocess duration remains a diagnostic, not the gate denominator.

A report passes only when:

• 50-100 controls completed both paths without integrity, containment, or fence failure;
• aggregate routed-identity recall, intersection / full, is at least 85%;
• aggregate adaptive/full slot-time ratio is at most 40%;
• every adaptive omission is represented in coverage counts; and
• no credential persistence, quarantine, projection, source-failure, or resource-bound regression is observed.

Reports are bound to exact selector, execution, and scan policy hashes. Stale or incomplete reports cannot authorize another policy. Adaptive canary remains fail-closed until a matching report passes. Broad adaptive enablement additionally requires a stable low-percentage production canary over at least one repository-refresh interval. Operators change rollout configuration explicitly; shadow evidence never changes execution mode by itself.

Alternatives rejected:

• Routing shadow candidates through keycheck would make the experiment authoritative and consume external capacity.
• Persisting per-image shadow identities creates unnecessary sensitive correlation data.
• Comparing only detector counts does not measure the routed identities the scanner is intended to produce.
• Automatically enabling adaptive mode from a report removes the operational rollback checkpoint.

8. Preserve incomplete warning semantics without redundant retries

The first completed 50-control production shadow report failed closed. Routed recall was 12 of 18 identities (66.7%), adaptive/full slot time was 46.6%, and the report recorded 43 aggregate failures. Its selection evidence showed 230 descriptors omitted by the eight-layer limit and 14 oversized descriptors, so selector recall remains the primary rollout blocker.

The same evidence exposed a separate execution defect. TruffleHog diagnostics such as chunk_processing and detector_timeout are explicitly deterministic, non-retryable warnings. Their findings must be retained, but the affected blob cannot establish complete coverage. The diagnostic adapter previously discarded the non-retryable bit, causing the layer executor to download and scan the same incomplete blob up to three times before reaching the same terminal state. The adapter now preserves aggregate warning retryability and the layer executor terminates that blob after the first deterministic warning. It does not mark the blob covered or remove the report failure.

The historical report schema retained only a total failure count, so its 43 failures cannot be decomposed exactly after the fact. Future shadow runs keep a fixed allow-list of aggregate-only failure categories in protected memory and print their totals in the final operator summary without changing report authority or persisting target-level evidence.

Shadow execution also suppresses target labels in finding-filter logs. Production scans retain their existing target logging, while both private full and private layer paths emit only aggregate filter counts. This closes a privacy gap found in the first report log without weakening normal operational diagnostics.

Risks / Trade-offs

• [History is malformed, misleading, or secret-bearing] -> Bound and validate it, persist only an enum, fall back to unknown, and keep identity/coverage independent of classification.
• [Application secrets exist in a low-priority or giant layer] -> Scan all-fit images, keep unknown ahead of generic/bulk classes, record partial scope, retain full controls, and enforce the 85% routed-recall gate.
• [Unsafe policy reuse suppresses a required rescan] -> Separate hashes and permit legacy aliasing only from exact successful compatible evidence in one fenced transaction.
• [Selector changes expand coverage during retry] -> Freeze the earliest full position map under the selector hash before leases are issued.
• [Multi-blob checkpoints increase work lost on crash] -> Bound count/bytes and retain independent per-blob leases and ingestion records; no pre-handoff result becomes covered.
• [Config prefetch adds Registry traffic] -> Reuse the already bounded authenticated downloader and avoid a second fetch when the config is leased in the same plan.
• [Individual layer scans expose whiteouted historical files] -> Preserve position provenance and describe evidence as image-content coverage, not merged-root state.
• [Shadow evaluation consumes slots] -> Keep it operator-invoked, bounded, deterministic, and subject to the existing slot/resource controls.
• [Aggregate reports hide individual anomalies] -> Fail the whole report on incomplete paired work and retain bounded failure counts without persisting target identity.
• [Deterministic scanner warnings consume repeated transfer and slot time] -> Preserve their non-retryable policy, keep findings and incomplete coverage, and terminate the blob on its first bounded attempt.

Migration Plan

1. Ship version-one and version-two validators, new modes, and migration code while production stays on its existing timeout-only canary.
2. Stop authoritative runtime and verify no active result, queue, or blob leases remain.
3. Add the selector-policy coverage column, aggregate shadow-report state, required timing/fence columns, indexes, and a new migration marker.
4. Backfill every historical image-coverage row from its exact valid reservation plan. Abort and roll back the migration transaction on an orphan, malformed plan, or invalid hash; then make the column non-null and run exact schema validation.
5. Restart with unchanged mode and verify legacy claims, projection, keycheck, resolver, quarantine, and source health before creating version-two work.
6. Run the private shadow evaluator for 50-100 completed controls. Keep adaptive modes fail-closed if the matching report misses recall, timing, safety, or completion gates.
7. Enable a low deterministic adaptive-canary, monitor at least one repository-refresh interval, and compare source failures, coverage reasons, routed yield, slot time, and quarantine.
8. Increase canary basis points and finally enable adaptive only after every gate remains satisfied.
9. Roll back immediately by setting mode to full or the existing timeout-only canary. Keep all version-two plans, reports, and coverage rows for audit and exact future resume.

Open Questions

• Which initial bounded checkpoint count and byte target provide the best reduction in continuation delay without increasing crash rework materially?
• Which classifier allow-list revisions improve routed recall in the first 50-100 controls? Every revision will receive a new selector-policy hash rather than changing an existing policy.
• What adaptive-canary basis-point sequence should operators use after the shadow gate passes?