Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-26
@@ -0,0 +1,68 @@
+## Context
+
+The OpenAI checker, candidate router, scheduler, and projection path are healthy, but the fresh input funnel collapsed after old target backlogs drained. In the last six days, core sources produced only 27 OpenAI findings and 11 new OpenAI credentials. A read-only exact-query probe found one unseen GitHub repository, 27 safely rescan-eligible GitLab projects, and 20 unseen repositories in the first 20 DockerHub results.
+
+The current source configuration accepts one plain string query per cycle and applies one source-wide page/target policy. Adding `openai` without a query-specific Docker bound could resolve and enqueue up to 200 repositories in one cycle. Runtime source state is authority-managed and must not be edited casually.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Restore explicit OpenAI-oriented discovery in the three query-driven core sources.
+- Bound only the exact `openai` query while preserving every other query's current limits.
+- Make the first post-deployment cycle deterministic without directly modifying runner state.
+- Preserve queue, revision, digest, pipeline, and keycheck authority guarantees.
+- Measure whether the added coverage yields new OpenAI credentials and usable checks.
+
+**Non-Goals:**
+- Reclassifying provider API outcomes or weakening the successful-generation requirement.
+- Re-enabling `package_git` or other broad inactive sources.
+- Rechecking known credentials, changing HuggingFace discovery, or increasing global scan concurrency.
+- Guaranteeing that newly discovered credentials are valid or funded.
+
+## Decisions
+
+### Add an exact provider query to existing rotations
+
+GitHub, GitLab, and DockerHub each receive one literal `openai` query. The term remains a normal persisted rotation entry, so completed cycles advance naturally and failures retain the query under existing semantics.
+
+For rollout, each entry is inserted at that source's current persisted query index while the runtime is coordinately stopped. The first restarted cycle therefore exercises `openai` without mutating state files; the next successful cycle advances to the query that previously occupied that index.
+
+Alternative: append the term and wait for a full rotation. Rejected because Docker cycles can be long and the canary would be delayed and difficult to attribute.
+
+### Apply a small allowlisted query override
+
+`build_args_from_source_config()` merges only `pages`, `per_page`, and `max_targets` from `query_overrides.<exact-query>`. Overrides are exact string matches, non-mutating, and validated before use. Unknown keys or non-mapping override shapes fail closed.
+
+Initial bounds:
+- GitHub `openai`: one page, five scan claims.
+- GitLab `openai`: one page, five scan claims; changed-target promotion remains capped at one.
+- DockerHub `openai`: two ten-result pages, twenty scan claims.
+
+The Docker page limit bounds discovery admission to at most 20 repositories before tag resolution. `max_targets` separately bounds claims in that source cycle. Other queries continue using their source-wide page and target values.
+
+Alternative: temporarily reduce source-wide pages. Rejected because it would silently reduce all-provider coverage after rollback mistakes. Alternative: add a dedicated Docker source alias. Rejected because it would duplicate lifecycle and queue ownership code.
+
+### Preserve downstream authority and deduplication
+
+The change ends at target discovery arguments. Existing normalized identity deduplication, revision-aware completed-target promotion, Docker digest resolution, result-bundle fencing, credential deduplication, cached-known occurrence handling, and API keycheck rules remain authoritative.
+
+## Risks / Trade-offs
+
+- [Exact search still produces many known targets] -> Persist separate new/updated counts and evaluate the full candidate funnel, not fetched volume.
+- [Docker results could create expensive scans] -> Bound search to 20 repositories and claims to 20 while retaining the global three-slot limit.
+- [Query override could accidentally alter unrelated settings] -> Allow only three numeric discovery/claim keys and test that ordinary queries retain source defaults.
+- [Config edit triggers immutable-authority shutdown] -> Use coordinated stop/start scripts and verify PostgreSQL, pipeline workers, and every core source after deployment.
+- [No valid OpenAI keys appear] -> Treat zero usable outcomes as yield evidence, not as proof of pipeline failure, provided candidates complete with explicit API outcomes.
+
+## Migration Plan
+
+1. Add query-override parsing and focused tests.
+2. Add exact queries and bounded overrides at each source's current persisted index.
+3. Run focused and full regression suites plus strict OpenSpec validation.
+4. Coordinately restart the authority-managed runtime.
+5. Observe exactly the first `openai` source cycle for GitHub, GitLab, and DockerHub; verify limits, queue admission, scan completion, candidate checks, and pipeline drain.
+6. Keep the query in normal rotation if safety bounds hold. Roll back by removing the query and overrides, then coordinately restart; no data migration is required.
+
+## Open Questions
+
+None. Further query weighting or expansion depends on measured canary yield.
@@ -0,0 +1,24 @@
+## Why
+
+OpenAI credential discovery fell from hundreds of new identities to almost none after historical Docker and package backlogs drained. The core GitHub, GitLab, and DockerHub discovery rotations do not contain the literal `openai` query, even though a read-only production probe showed that exact query exposes previously unseen supply.
+
+## What Changes
+
+- Add exact `openai` discovery to the GitHub, GitLab, and DockerHub core query rotations.
+- Support narrowly allowlisted per-query bounds so the exact query can use smaller page and target limits without reducing coverage for every other query.
+- Bound the first and recurring exact-query windows to one GitHub page, one GitLab page, and two DockerHub pages with source-appropriate scan limits.
+- Preserve existing target deduplication, revision-aware rescan limits, queue authority, and Docker digest requirements.
+- Measure the exact-query canary from discovery through scans, OpenAI candidates, API checks, and usable outcomes.
+
+## Capabilities
+
+### New Capabilities
+- `openai-discovery-coverage`: Exact provider-term discovery with query-scoped bounds and observable production rollout.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+The change affects `app/config.yaml`, query argument construction in `app/console_runner.py`, focused runner/config tests, source-cycle behavior for GitHub/GitLab/DockerHub, and production canary operations. It adds no dependency or schema migration and does not alter HuggingFace, detector routing, keycheck classification, or Docker target identity.
@@ -0,0 +1,57 @@
+## ADDED Requirements
+
+### Requirement: Exact OpenAI core discovery
+The system SHALL include the literal `openai` query in the normal GitHub, GitLab, and DockerHub core discovery rotations.
+
+#### Scenario: Exact provider term is rotated
+- **WHEN** each supported source reaches its configured exact-query position
+- **THEN** the source SHALL execute discovery using the literal `openai` term and persist normal cycle attribution
+
+#### Scenario: Successful exact-query cycle advances
+- **WHEN** an exact-query source cycle completes successfully
+- **THEN** the source SHALL advance to its next configured query through the existing persisted rotation
+
+#### Scenario: Failed exact-query cycle is retained
+- **WHEN** exact-query discovery fails before a successful cycle completion
+- **THEN** the source SHALL retain the same query according to existing failure semantics
+
+### Requirement: Query-scoped safety bounds
+The system SHALL support exact-query overrides for only `pages`, `per_page`, and `max_targets`, without changing source-wide defaults for other queries.
+
+#### Scenario: Exact query receives bounded arguments
+- **WHEN** a source builds arguments for `openai`
+- **THEN** it SHALL apply that source's configured page, page-size, and target overrides
+
+#### Scenario: Ordinary query retains source defaults
+- **WHEN** the same source builds arguments for any query without an override
+- **THEN** it SHALL retain the source-wide page, page-size, and target values
+
+#### Scenario: Invalid override fails closed
+- **WHEN** a query override is not a mapping or contains a key outside the allowlist
+- **THEN** argument construction SHALL fail before discovery or queue mutation
+
+### Requirement: Source-specific rollout limits
+The initial production policy SHALL constrain GitHub and GitLab exact discovery to one page each and DockerHub exact discovery to two pages of ten results, while preserving the existing global scan limit and changed-target promotion cap.
+
+#### Scenario: Docker exact discovery is bounded
+- **WHEN** DockerHub executes the exact `openai` query
+- **THEN** it SHALL request at most two pages of ten repositories and claim at most twenty targets in that cycle
+
+#### Scenario: Revision-aware source remains bounded
+- **WHEN** GitLab exact discovery observes multiple changed completed projects
+- **THEN** updated-target promotion SHALL remain capped by the existing one-per-cycle policy
+
+#### Scenario: Docker target authority is unchanged
+- **WHEN** DockerHub exact discovery returns repository names
+- **THEN** only targets satisfying the existing immutable digest requirement SHALL reach scanning
+
+### Requirement: End-to-end canary evidence
+The rollout SHALL be evaluated from source discovery through durable scan completion, candidate completion, provider result, and projection drain without exposing credential or target values.
+
+#### Scenario: Safe canary completes
+- **WHEN** the first exact-query cycles run after deployment
+- **THEN** operators SHALL verify source limits, new and updated admissions, queue dispositions, pipeline completion, and runtime health using aggregate evidence
+
+#### Scenario: Useful yield is reported accurately
+- **WHEN** exact-query scans create OpenAI candidates
+- **THEN** operators SHALL report genuinely new credentials and their explicit API outcomes separately from cached-known occurrences and stale legacy file state
@@ -0,0 +1,16 @@
+## 1. Query-scoped controls
+
+- [x] 1.1 Add validated allowlisted per-query overrides for pages, page size, and maximum targets.
+- [x] 1.2 Add focused tests proving exact-query overrides apply and ordinary queries remain unchanged.
+
+## 2. OpenAI discovery coverage
+
+- [x] 2.1 Add literal `openai` queries and source-specific safety bounds for GitHub, GitLab, and DockerHub.
+- [x] 2.2 Test canonical configuration coverage, Docker isolation, and rollout limits.
+
+## 3. Verification and rollout
+
+- [x] 3.1 Run focused and full regression suites with bytecode writes disabled.
+- [x] 3.2 Run strict OpenSpec validation and verify implementation against artifacts.
+- [x] 3.3 Coordinately restart the authority-managed runtime and verify PostgreSQL, pipeline, and core sources.
+- [x] 3.4 Observe the first bounded exact-query cycles and measure admissions, scan outcomes, OpenAI candidates, API outcomes, and pipeline drain.