Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-08-28
|
||||
@@ -0,0 +1,120 @@
|
||||
## Context
|
||||
|
||||
DockerHub discovery currently inspects a bounded tag page but stops after the first eligible digest. Different tags often alias the same manifest or share the same ordered layers, so increasing the tag count without content-aware selection would mostly multiply duplicate work. The Hub tag response identifies platform digests but does not expose their layers; those come from the Docker Registry v2 manifest API.
|
||||
|
||||
GitHub and GitLab updated-target admission currently uses provider timestamps. A claimed scan then points TruffleHog at a mutable repository URL with a rolling age boundary and a depth cap. The timestamp is useful for deciding that something may have changed, but it neither identifies the exact ref nor proves which commit was scanned. Existing PostgreSQL reservations provide the fence under which an immutable plan can be attached.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- Spend each repository's Docker budget on up to three distinct ordered layer graphs.
|
||||
- Keep Docker tag, manifest, request, cache, and emitted-target counts explicitly bounded.
|
||||
- Bind GitHub and GitLab scans to a provider-resolved ref and commit SHA after a fenced claim.
|
||||
- Use the last successfully covered SHA as the incremental boundary for the same ref.
|
||||
- Preserve exact plan and coverage evidence through retries, worker failure, and mid-scan updates.
|
||||
- Keep first-scan history bounded while making every later delta independent of rolling age and depth limits.
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- Persist a global historical inventory of Docker layers.
|
||||
- Guarantee discovery of a branch that appears and disappears between metadata polling cycles.
|
||||
- Replace repository metadata search with a global push-event feed.
|
||||
- Remove source API rate limits, target timeouts, result bounds, or queue admission bounds.
|
||||
- Claim that a bounded first scan covered history older than its configured baseline depth.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Resolve platform manifests before selecting Docker targets
|
||||
|
||||
For each bounded Hub tag candidate, the resolver will choose the requested platform child digest, obtain a short-lived pull token for the public repository, and fetch that child manifest from the Docker Registry v2 API. A candidate identity contains its tag, update time, immutable platform manifest digest, and ordered layer digest tuple.
|
||||
|
||||
The top-level multi-platform index digest will not be emitted when a matching child digest exists. Registry responses must be JSON manifests of bounded size with valid `sha256` layer digests. Token requests use a fixed Docker authentication origin and repository pull scope; credentials are never placed in cache records or logs.
|
||||
|
||||
Alternatives rejected:
|
||||
|
||||
- Comparing tag names or manifest digests alone, because different manifests can still contain the same layer chain.
|
||||
- Pulling every candidate image before selection, because manifest metadata is sufficient and much cheaper.
|
||||
- Persisting every layer immediately, because within-resolution novelty provides the requested bounded diversity without a new authoritative subsystem.
|
||||
|
||||
### Select up to three graphs deterministically
|
||||
|
||||
The source setting `docker_images_per_repository` is clamped to one through three. After exact ordered-tuple deduplication, selection uses stable tie breaking:
|
||||
|
||||
1. The newest resolved graph.
|
||||
2. The remaining graph that contributes the most layers not present in the selected union, with recency as the tie breaker.
|
||||
3. The oldest remaining distinct graph.
|
||||
|
||||
If fewer distinct graphs exist, fewer targets are emitted. Reordered layer tuples remain distinct because layer order changes the image filesystem. Selected targets retain the existing immutable `repository@sha256:...` identity, so queue deduplication and Docker scan execution do not change.
|
||||
|
||||
Only complete graph-selection results are stored in the disposable tag cache. A partial manifest-resolution failure can return successfully resolved targets for the current cycle, but it is not cached as complete and is reported as partial coverage.
|
||||
|
||||
### Resolve and bind a Git plan after claim
|
||||
|
||||
Repository timestamps remain coarse admission signals. Once PostgreSQL has fenced a queue row and result reservation, the worker resolves the source-provided ref hint or the repository's current default branch through the GitHub or GitLab API. The resolver returns a normalized ref and exact head SHA.
|
||||
|
||||
The reservation is then bound transactionally to an immutable plan containing ref, head SHA, optional covered base SHA, plan mode, and baseline bounds. Binding requires the active reservation and claim lease token. A missing or malformed revision is a retryable source failure; the worker does not silently scan a mutable URL and does not advance exact coverage.
|
||||
|
||||
Metadata search can identify only repository-level activity, so its exact scope is the provider-resolved default branch. Event-backed discovery may supply a more specific ref. Polling cannot guarantee capture of transient or unadvertised refs; that limitation remains explicit.
|
||||
|
||||
Alternatives rejected:
|
||||
|
||||
- Resolving one SHA for every search result before admission, because that would spend scarce API quota on records that are never claimed.
|
||||
- Encoding SHA in queue target identity, because it would create unbounded rows and bypass existing changed-target coalescing.
|
||||
- Copying a timestamp into a covered field at claim, because failed work would look complete.
|
||||
|
||||
### Execute pinned baselines and deltas
|
||||
|
||||
An initial or ref-changed plan scans the exact head with the configured first-scan depth bound. A same-ref plan with a different successfully covered head scans the pinned head with `--since-commit <covered SHA>` and omits rolling age and maximum-depth limits. A plan whose resolved head already equals the covered head produces an exact no-op result.
|
||||
|
||||
The installed TruffleHog binary's ability to accept a commit SHA as `--branch` is a deployment contract and will be covered by a local repository contract test. If the covered base is unavailable after a force push or ref recreation, execution falls back to a pinned bounded baseline and labels the result as such; it never reports an incremental range as covered when the base was not usable.
|
||||
|
||||
This delta means all commits reachable from the pinned head after the covered boundary, not merely the final filesystem diff. An add-then-delete sequence in separate new commits therefore remains visible.
|
||||
|
||||
### Advance covered SHA only during fenced successful ingestion
|
||||
|
||||
`target_queue` stores the last successfully covered ref and head SHA. `result_reservations` stores the immutable claimed plan, and normalized scan metadata stores the executed plan and whether execution remained pinned. On fenced ingestion with queue disposition `done`, the plan in metadata must match the reservation. Only a successful pinned baseline, successful incremental scan, or exact no-op advances or confirms the covered head. Failed, deferred, unbound, or mutable fallback work leaves coverage unchanged.
|
||||
|
||||
Because the claimed head is immutable, a newer provider update during execution remains discoverable after completion and can create a later plan from the just-covered head.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Registry manifest calls increase Docker API traffic] -> Keep tag candidates and emitted graphs bounded, reuse one scoped token per repository resolution, and do not cache partial results as complete.
|
||||
- [Many tags alias one graph] -> Deduplicate exact ordered layer tuples before queue insertion.
|
||||
- [A source API is unavailable after claim] -> Produce a retryable source failure and refund through the existing bounded lifecycle without changing coverage.
|
||||
- [TruffleHog SHA branch behavior differs by version] -> Add a local contract test against the configured binary and fail closed when immutable pinning is unsupported.
|
||||
- [Force-pushed base is no longer reachable] -> Retry as a pinned bounded baseline and label the loss of incremental continuity.
|
||||
- [Default-branch resolution misses non-default branch activity] -> Record the resolved scope honestly and allow event-backed ref hints; a complete ref-event feed remains future work.
|
||||
- [First-scan depth remains bounded] -> Treat the first head as the future delta baseline without claiming unbounded historical coverage.
|
||||
- [Three Docker graphs can triple downstream work] -> Clamp the per-repository setting to three and retain all existing source, queue, timeout, and output bounds.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add nullable Git coverage and reservation-plan columns through idempotent PostgreSQL schema initialization.
|
||||
2. Deploy graph resolution, plan binding, and tests with explicit configuration gates.
|
||||
3. Enable `docker_images_per_repository: 3` for DockerHub while retaining the existing twenty-tag candidate bound.
|
||||
4. Enable exact Git planning for GitHub and GitLab; legacy rows start with no covered SHA and receive a pinned bounded baseline on their next admitted scan.
|
||||
5. Observe partial manifest resolution, distinct graph counts, exact no-ops, baseline resets, delta scans, failures, and strict usable yield.
|
||||
|
||||
Rollback is configuration-first. Set Docker images per repository back to one and disable exact Git planning; nullable schema additions remain inert and require no destructive migration.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether production evidence supports increasing the Docker candidate tag page beyond twenty without exhausting Hub rate limits.
|
||||
- Whether a later change should snapshot all advertised refs or consume a dedicated push-event feed for complete non-default-branch coverage.
|
||||
|
||||
## Implementation Evidence
|
||||
|
||||
Implementation completed and locally validated on 2026-08-28:
|
||||
|
||||
- Docker resolution uses the bounded Registry v2 bearer flow, immutable platform-child digests, ordered-layer graph deduplication, and deterministic newest/novel/oldest selection. Partial graph resolution emits only proven targets, retains the repository for retry, and is not cached as complete.
|
||||
- PostgreSQL reservations bind one canonical exact Git plan under the active queue/reservation lease fence. Covered ref/head state advances in the same fenced transaction as successful result ingestion after exact plan and execution-evidence comparison.
|
||||
- GitHub and GitLab resolve either an explicit branch ref or the provider default branch to an exact commit. Baseline, delta, no-op, and continuity-reset execution remain pinned to the bound head.
|
||||
- The checked-in core configuration enables three Docker graphs and exact Git planning with a baseline depth of 100, two ref-resolution attempts, a 10-second shared timeout, and a 1 MiB response limit.
|
||||
- `python -B -m pytest -p no:cacheprovider -q tests/test_exact_git_scan_planning.py tests/test_validated_high_scanner_fixes.py::DockerTagIdentityTests tests/test_runtime_safety_layer.py tests/test_pipeline_cutover_invariants.py tests/test_migration_runtime_safety.py tests/test_pipeline_postgres_integration.py::PipelinePostgresIntegrationTests::test_exact_git_plan_binding_and_coverage_are_fenced` completed with `141 passed`.
|
||||
- The PostgreSQL integration scenario covers idempotent/conflicting binding, baseline to delta to no-op progression, durable exact metadata, a provider update observed during an active scan, successful fenced coverage advancement, and a post-bind worker refund that cannot advance coverage.
|
||||
- The configured local TruffleHog binary passed the exact-SHA `--branch` contract test included in the focused suite.
|
||||
- `python -B -m py_compile app/scanner.py app/scanner_db.py app/console_runner.py tests/test_exact_git_scan_planning.py tests/test_pipeline_postgres_integration.py` completed successfully.
|
||||
- `openspec validate improve-core-scan-coverage --strict` reported the change as valid.
|
||||
|
||||
Retained rollout limits are twenty Docker tag candidates, at most three emitted distinct graphs, 8 MiB per Registry manifest, 1,000 index descriptors, 2,048 layers, bounded source/queue/result limits, and PostgreSQL-only exact Git plan binding. No production migration, process restart, or rollout was performed as part of implementation. Deployment still requires the offline idempotent runtime-safety migration before restarting sources. Configuration-first rollback remains `docker_images_per_repository: 1` plus disabling `exact_git_planning_enabled`; nullable schema additions may remain in place.
|
||||
@@ -0,0 +1,27 @@
|
||||
## Why
|
||||
|
||||
The core DockerHub, GitHub, and GitLab sources spend most of their scan budget on repeated image contents or mutable repository snapshots, while recent strict-usable yield remains near zero. The scanner needs to cover materially different Docker layers and bind Git work to exact revisions so that additional work buys new evidence rather than another pass over the same surface.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Select up to three Docker images per repository whose ordered layer graphs are distinct, instead of stopping at the first eligible tag.
|
||||
- Prefer the newest graph, a graph adding the most not-yet-selected layers, and an older divergent graph while continuing to deduplicate immutable digest targets.
|
||||
- Resolve Git discovery observations into immutable ref and commit identities before scanning.
|
||||
- Scan all commits introduced since the last successfully covered commit for that ref, rather than relying on a moving repository URL, age cutoff, and depth cap.
|
||||
- Persist immutable Git scan plans and successfully covered heads; never advance exact coverage on a failed or unpinned fallback scan.
|
||||
- Bound API enumeration and Docker/Git expansion through explicit configuration and report partial or inexact coverage honestly.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `docker-layer-graph-selection`: Bounded selection of materially distinct platform-specific Docker image layer graphs.
|
||||
- `git-ref-delta-scanning`: Immutable, per-ref Git scan planning and successful incremental coverage tracking.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
None.
|
||||
|
||||
## Impact
|
||||
|
||||
The change affects Docker Hub tag and registry-manifest resolution, GitHub and GitLab metadata resolution, source-cycle configuration, PostgreSQL queue/reservation/scan state, TruffleHog command construction, and focused scanner/runtime integration tests. It adds bounded registry and source API requests but does not change external service APIs or credential output formats.
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Platform-specific layer graphs are resolved
|
||||
The system SHALL resolve each bounded Docker tag candidate to the requested platform child manifest and SHALL represent its image contents as the ordered sequence of valid layer digests.
|
||||
|
||||
#### Scenario: Multi-platform tag contains the requested platform
|
||||
- **WHEN** a tag exposes a matching `linux/amd64` child manifest
|
||||
- **THEN** the system uses that child manifest digest and its ordered layers rather than the top-level index digest
|
||||
|
||||
#### Scenario: Candidate manifest is malformed
|
||||
- **WHEN** a registry response is oversized, malformed, or contains invalid layer identities
|
||||
- **THEN** the candidate is not represented as a resolved layer graph
|
||||
|
||||
### Requirement: Docker selection covers distinct graphs
|
||||
The system SHALL emit no more than the configured one-to-three image targets per repository and SHALL NOT emit two candidates with identical ordered layer digest sequences.
|
||||
|
||||
#### Scenario: Tags alias one graph
|
||||
- **WHEN** multiple tags resolve to the same ordered layer sequence
|
||||
- **THEN** only the newest alias remains eligible for selection
|
||||
|
||||
#### Scenario: Three or more graphs are available
|
||||
- **WHEN** at least three distinct graphs resolve successfully
|
||||
- **THEN** the system selects the newest graph, the remaining graph adding the most not-yet-selected layers, and the oldest remaining distinct graph
|
||||
|
||||
#### Scenario: Fewer graphs are available
|
||||
- **WHEN** fewer distinct graphs resolve successfully than the configured maximum
|
||||
- **THEN** the system emits only the distinct graphs that exist
|
||||
|
||||
#### Scenario: Layer order differs
|
||||
- **WHEN** two manifests contain the same layer identities in a different order
|
||||
- **THEN** the system treats them as distinct graphs
|
||||
|
||||
### Requirement: Selected Docker targets remain immutable
|
||||
The system SHALL emit each selected target as the canonical requested-platform manifest digest identity `repository@sha256:<digest>`.
|
||||
|
||||
#### Scenario: Selected tag moves later
|
||||
- **WHEN** a tag is republished after discovery
|
||||
- **THEN** the queued target continues to identify the originally selected platform manifest digest
|
||||
|
||||
### Requirement: Docker graph resolution is bounded and honest
|
||||
The system SHALL retain hard tag-candidate, selected-graph, response-size, retry, and timeout bounds and SHALL NOT cache partial graph resolution as complete.
|
||||
|
||||
#### Scenario: Some manifest requests fail
|
||||
- **WHEN** at least one candidate graph resolves and another candidate fails transiently
|
||||
- **THEN** the system may emit the resolved targets for the cycle but records partial resolution and does not write a complete positive cache entry
|
||||
|
||||
#### Scenario: Every manifest request fails transiently
|
||||
- **WHEN** no candidate graph can be resolved because registry metadata is unavailable
|
||||
- **THEN** the repository remains retryable through the existing deferred-resolution lifecycle
|
||||
@@ -0,0 +1,68 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Git scans are bound to an exact revision
|
||||
The system SHALL resolve a normalized ref and exact commit SHA for each claimed GitHub or GitLab repository before invoking TruffleHog and SHALL bind that immutable plan to the active reservation.
|
||||
|
||||
#### Scenario: Repository search supplies no ref hint
|
||||
- **WHEN** a claimed repository came from metadata search without an exact ref
|
||||
- **THEN** the system resolves the provider's current default branch and its exact head SHA
|
||||
|
||||
#### Scenario: Discovery supplies an exact ref hint
|
||||
- **WHEN** an event-backed target includes a valid branch ref
|
||||
- **THEN** the system resolves and binds that specific ref instead of substituting the default branch
|
||||
|
||||
#### Scenario: Revision lookup fails
|
||||
- **WHEN** the provider API cannot return a valid ref and commit SHA
|
||||
- **THEN** the claim receives a bounded retryable source failure and no exact coverage state advances
|
||||
|
||||
### Requirement: Git updates scan every newly introduced commit
|
||||
The system SHALL scan the exact claimed head after the last successfully covered head for the same ref and SHALL NOT apply rolling age or maximum-depth limits to that incremental range.
|
||||
|
||||
#### Scenario: Same ref advances
|
||||
- **WHEN** ref `R` was successfully covered at commit `A` and now resolves to descendant commit `D`
|
||||
- **THEN** the scan is pinned to `D` with `A` as its boundary and includes commits introduced between them
|
||||
|
||||
#### Scenario: Secret is added and then deleted in the delta
|
||||
- **WHEN** one newly introduced commit adds a secret and a later newly introduced commit removes it
|
||||
- **THEN** both commits remain in scan scope even though the final filesystem snapshot is clean
|
||||
|
||||
#### Scenario: Head is unchanged
|
||||
- **WHEN** the resolved head equals the successfully covered head for the same ref
|
||||
- **THEN** the system records an exact no-op without launching a redundant repository scan
|
||||
|
||||
### Requirement: Git baseline and discontinuity handling remain pinned
|
||||
The system SHALL use a pinned bounded baseline for a first-seen ref or an unusable incremental base and SHALL identify that mode without claiming unbounded historical coverage.
|
||||
|
||||
#### Scenario: Ref has no covered head
|
||||
- **WHEN** an exact ref is claimed without prior successful coverage
|
||||
- **THEN** the system scans its pinned head using the configured baseline depth bound and establishes that head as the future delta boundary on success
|
||||
|
||||
#### Scenario: Covered base is unavailable
|
||||
- **WHEN** force push, ref recreation, or remote history removal makes the covered SHA unusable
|
||||
- **THEN** the system falls back to a pinned bounded baseline and records the continuity reset
|
||||
|
||||
### Requirement: Git coverage advances only after successful fenced work
|
||||
The system SHALL update a queue row's covered ref and head only when successful ingestion applies a matching immutable reservation plan.
|
||||
|
||||
#### Scenario: Exact scan succeeds
|
||||
- **WHEN** a pinned baseline or delta result is ingested with queue disposition `done` and its plan matches the active reservation
|
||||
- **THEN** the queue's covered ref and head advance to the claimed head
|
||||
|
||||
#### Scenario: Exact scan fails or is deferred
|
||||
- **WHEN** execution fails, times out, loses its fence, or receives a deferred disposition
|
||||
- **THEN** the previously covered ref and head remain unchanged
|
||||
|
||||
#### Scenario: Remote advances during a scan
|
||||
- **WHEN** a newer commit appears after the worker binds its immutable head
|
||||
- **THEN** successful completion advances coverage only to the bound head and leaves the newer update eligible for later discovery
|
||||
|
||||
### Requirement: Exact Git scope is observable
|
||||
The system SHALL durably record the executed ref, head, base, scan mode, baseline bound, and whether immutable execution was preserved.
|
||||
|
||||
#### Scenario: Operator inspects an incremental scan
|
||||
- **WHEN** an exact delta result is committed
|
||||
- **THEN** its normalized scan metadata identifies the covered range without exposing source credentials
|
||||
|
||||
#### Scenario: Metadata discovery observes repository-level activity
|
||||
- **WHEN** no branch-specific event exists
|
||||
- **THEN** observability identifies the provider-resolved default-branch scope rather than implying coverage of every repository ref
|
||||
@@ -0,0 +1,20 @@
|
||||
## 1. Docker Layer Graph Selection
|
||||
|
||||
- [x] 1.1 Add and clamp `docker_images_per_repository` configuration through source argument construction and all Docker tag-resolution call sites.
|
||||
- [x] 1.2 Implement bounded Docker Registry token, platform-manifest, and ordered-layer resolution with strict response validation.
|
||||
- [x] 1.3 Implement deterministic distinct-graph selection, immutable child-digest targets, partial-result handling, and cache-version invalidation.
|
||||
- [x] 1.4 Add focused tests for aliases, platform children, novelty and age selection, malformed manifests, partial failures, and resolution plumbing.
|
||||
|
||||
## 2. Exact Git Revision Planning
|
||||
|
||||
- [x] 2.1 Add idempotent PostgreSQL fields and fenced methods for immutable reservation plans and last successfully covered Git ref/head.
|
||||
- [x] 2.2 Implement bounded GitHub and GitLab ref/head resolvers with default-branch and explicit-ref handling.
|
||||
- [x] 2.3 Extend Git scan execution for exact no-op, pinned baseline, and unbounded-by-age/depth delta modes, including continuity-reset fallback.
|
||||
- [x] 2.4 Bind plans after claims, include them in result metadata, and advance covered heads only during matching successful ingestion.
|
||||
- [x] 2.5 Add focused tests for plan resolution, command construction, unchanged heads, failure/refund fencing, successful coverage, force-push fallback, and mid-scan updates.
|
||||
|
||||
## 3. Configuration And Verification
|
||||
|
||||
- [x] 3.1 Enable three distinct Docker graphs and exact Git planning for the core sources with bounded documented defaults.
|
||||
- [x] 3.2 Run focused Docker, Git, queue, schema, lifecycle, and OpenSpec validation suites.
|
||||
- [x] 3.3 Record implementation evidence and any retained rollout limits in the change artifacts.
|
||||
Reference in New Issue
Block a user