sashatrask
2.7 KiB
2.7 KiB
Why
DockerHub discovery currently downloads its full configured page range before database deduplication, and one exhausted page discards every successful page while blocking keyword rotation. The authenticated 30-page window makes deeper discovery possible, but it needs incremental persistence, seen-page stopping, and durable retry delegation to remain efficient and avoid silent gaps or head-of-line blocking.
What Changes
- BREAKING: replace complete-or-fail DockerHub pagination with page-level durable persistence; successful pages remain admitted when another page exhausts its two request attempts.
- Stop an ordinary query pass after two consecutive nonempty pages whose repository identities were already known before the pass.
- Run every DockerHub query with an effective ceiling of 30 pages and 100 results per page.
- Bypass seen-page stopping for a full deep pass of each exact query at least once per 72-hour scheduling interval.
- Delegate failed page/query acquisition to a fenced PostgreSQL discovery retry backlog before allowing the main keyword rotation to advance.
- Add 12 DockerHub product/framework queries:
open-webui,ragflow,dify,flowise,crewai,n8n,langflow,autogen,browser-use,openhands,anythingllm, andagent-zero. - Temporarily disable periodic re-resolution of completed DockerHub repository anchors while preserving initial resolution, partial/error resolver retries, immutable-digest scan retries, and normal keyword discovery.
- Preserve authenticated fail-closed search, the two-GET page budget, target uniqueness, cold/failed target policy, and secret-safe diagnostics.
Capabilities
New Capabilities
dockerhub-incremental-discovery: Incremental DockerHub page admission, seen-page stopping, 72-hour deep passes, and durable failed-page retry work.
Modified Capabilities
None.
Impact
- Affects DockerHub pagination and authentication integration in
app/scanner.py, source-cycle orchestration/state inapp/console_runner.py, PostgreSQL schema and retry claims inapp/scanner_db.py, and DockerHub settings inapp/config.yaml. - Adds a PostgreSQL discovery retry queue with bounded leases, fencing, backoff, and configured-query/policy validation.
- Changes DockerHub query rotation from failure-blocking to durable retry delegation and adds an initial bounded backlog from 12 new deep searches.
- Requires focused unit, SQL-shape, migration, PostgreSQL integration, source-state, configuration, and runtime health verification.
- Does not add dependencies or credential formats and does not change Docker tag selection, Registry authentication, layer scanning, scan workers, keychecks, or immutable-digest retry policy.