truf-server/openspec/changes/optimize-dockerhub-discovery-rotation/proposal.md

Why

DockerHub discovery currently downloads its full configured page range before database deduplication, and one exhausted page discards every successful page while blocking keyword rotation. The authenticated 30-page window makes deeper discovery possible, but it needs incremental persistence, seen-page stopping, and durable retry delegation to remain efficient and avoid silent gaps or head-of-line blocking.

What Changes

• BREAKING: replace complete-or-fail DockerHub pagination with page-level durable persistence; successful pages remain admitted when another page exhausts its two request attempts.
• Stop an ordinary query pass after two consecutive nonempty pages whose repository identities were already known before the pass.
• Run every DockerHub query with an effective ceiling of 30 pages and 100 results per page.
• Bypass seen-page stopping for a full deep pass of each exact query at least once per 72-hour scheduling interval.
• Delegate failed page/query acquisition to a fenced PostgreSQL discovery retry backlog before allowing the main keyword rotation to advance.
• Add 12 DockerHub product/framework queries: open-webui, ragflow, dify, flowise, crewai, n8n, langflow, autogen, browser-use, openhands, anythingllm, and agent-zero.
• Temporarily disable periodic re-resolution of completed DockerHub repository anchors while preserving initial resolution, partial/error resolver retries, immutable-digest scan retries, and normal keyword discovery.
• Preserve authenticated fail-closed search, the two-GET page budget, target uniqueness, cold/failed target policy, and secret-safe diagnostics.

Capabilities

New Capabilities

• dockerhub-incremental-discovery: Incremental DockerHub page admission, seen-page stopping, 72-hour deep passes, and durable failed-page retry work.

Modified Capabilities

None.

Impact

• Affects DockerHub pagination and authentication integration in app/scanner.py, source-cycle orchestration/state in app/console_runner.py, PostgreSQL schema and retry claims in app/scanner_db.py, and DockerHub settings in app/config.yaml.
• Adds a PostgreSQL discovery retry queue with bounded leases, fencing, backoff, and configured-query/policy validation.
• Changes DockerHub query rotation from failure-blocking to durable retry delegation and adds an initial bounded backlog from 12 new deep searches.
• Requires focused unit, SQL-shape, migration, PostgreSQL integration, source-state, configuration, and runtime health verification.
• Does not add dependencies or credential formats and does not change Docker tag selection, Registry authentication, layer scanning, scan workers, keychecks, or immutable-digest retry policy.