Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-07
@@ -0,0 +1,81 @@
+## Context
+
+The earlier global pruning pass removed 38 terms with zero strict-usable yield but deliberately preserved their existing backlog. The later cold-policy change added an audited reversible hold and applied it only to the retired Docker cohort. The current canonical configuration still has 324 exact source/query pairs across 95 query texts.
+
+A fresh PostgreSQL dry-run attributed scans and candidates through immutable `target_scans.query` and `keycheck_candidates.query`, then linked candidates by `credential_id` to every historical `keycheck_results.status_group='alive'` result. After excluding dedicated source sentinels, 38 exact source/query pairs each have at least 1,000 successful scans, zero ever-alive credentials, and zero pending candidate checks. They account for 114,226 successful scans and 75,551 currently unfenced pending/deferred queue rows.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Retire source/query pairs only after substantial completed exposure and fully matured zero-alive evidence.
+- Preserve a visible canonical rejection record with the evidence and decision reason.
+- Stop future discovery and existing claimable work for the rejected pairs.
+- Preserve every historical and queue authority record and make queue holds reversible.
+- Apply the decision without racing workers or partially transitioning a reviewed cohort.
+
+**Non-Goals:**
+- Delete queue rows, scans, findings, candidates, credentials, keycheck results, reservations, or coverage.
+- Treat source sentinels such as `gharchive`, `gharchive-files`, `gists`, `logs`, or `spaces` as discovery keywords.
+- Retire a query globally because it failed in one source.
+- Automatically reevaluate or reactivate rejected pairs during normal runtime.
+- Claim that a rejected pair can never become productive in the future.
+
+## Decisions
+
+### Evaluate exact source/query pairs
+
+The decision unit is the case-sensitive exact `(source, query)` pair. A keyword that produced an alive credential in DockerHub does not justify retaining a large npm backlog when the npm pair itself has substantial zero-alive evidence.
+
+Alternative: retire only globally zero-alive query text. Rejected because the dry-run would hold only 3,626 rows and preserve most demonstrated source-specific waste.
+
+### Require 1,000 successful scans and mature keychecks
+
+A pair qualifies only when it has at least 1,000 ended scans with status `clean`, `found`, or `degraded`, no credential linked through any of its candidate occurrences has ever had a historical `status_group='alive'` result, and no candidate for the pair remains pending or leased. Historical alive status is intentionally used instead of only current status so a once-working credential permanently proves yield.
+
+The 1,000-scan floor is deliberately stricter than a 100-scan cut. The latter selected 131 pairs and 187,917 rows but is too weak for rare useful credentials. Pair attribution uses immutable scan/candidate query snapshots for evidence; queue transition uses the row's current exact query because that is the durable admission attribution.
+
+Alternative: use raw finding count, unique candidates, current status only, or the dashboard strict-usable tier. Rejected because the operator requested actual `alive`, findings do not prove provider utility, current-only status forgets historical success, and pending checks make zero yield unresolved.
+
+### Preserve a canonical rejected registry
+
+Each removed pair remains under canonical `query_policy.rejected` with status `rejected_zero_alive`, evidence cutoff, successful scan count, finding count, unique credential count, pending count, ever-alive count, and reviewed queue count. Active query lists and rejected entries must be disjoint. The registry is evidence and operator visibility; existing append-only queue policy events remain the state-transition audit.
+
+Alternative: leave comments beside removed YAML entries. Rejected because comments are not machine-checkable and cannot fence future accidental reintroduction.
+
+### Use the existing cold lifecycle
+
+After configuration removal, the existing hash-fenced stopped-source manifest flow selects only unfenced `pending` or `deferred` rows whose exact query is absent from active policy. Each selected row becomes `cold`; no target value appears in review output, and no other row field or linked record changes. Active/fenced rows make apply fail closed. Reactivation remains possible only through the existing reviewed reverse action.
+
+### Approve the exact cohort
+
+- DockerHub: `OR`, `agent`.
+- GitHub: `coding`, `memory`.
+- npm: `OR`, `agent`, `agents`, `ai`, `assistant`, `benchmark`, `bot`, `chat`, `chats`, `completion`, `completions`, `conversation`, `gemini`, `groq`, `langchain`, `llm`, `mcp`, `open`, `openrouter`, `prompt`, `rag`, `semantic`, `studio`, `xai`.
+- package-git: `agent`, `bot`, `completion`, `llm`, `open`, `semantic`, `studio`.
+- Postman: `XAI_API_KEY`.
+- PyPI: `langchain`, `open`.
+
+GitLab has no pair meeting the 1,000-scan rule. Dedicated source sentinel queries remain untouched.
+
+## Risks / Trade-offs
+
+- [Rare future yield is lost] -> Preserve evidence, history, and reversible cold events; reactivation requires explicit review.
+- [Current queue query may differ from an older scan query after rediscovery] -> Use immutable scan/candidate attribution for yield evidence, exact current queue attribution for holding, and preserve append-only reversal evidence.
+- [A candidate becomes alive between dry-run and apply] -> Regenerate and verify evidence immediately before apply; fail if the approved zero-alive cohort drifts.
+- [A worker owns a selected row] -> Stop sources and fail the entire manifest on any queue, resolver, reservation, or blob fence.
+- [Configuration accidentally reintroduces a rejected pair] -> Validate active/rejected disjointness in configuration tests and policy loading.
+
+## Migration Plan
+
+1. Add canonical rejected-query evidence and exact configuration tests for the approved 38-pair cohort.
+2. Verify runtime sources are stopped and cleanly stop the current maintenance database before changing authority-covered configuration.
+3. Remove each rejected pair from its active source list and retain it in `query_policy.rejected` with reviewed evidence.
+4. Restart maintenance PostgreSQL, rerun the zero-alive evidence query, and require the exact cohort and queue counts to match the reviewed decision.
+5. Generate bounded private cold manifests per affected source/platform and apply them through the existing cluster lock and stopped-source action.
+6. Verify 75,551 rows became cold, no selected row remains claimable, history counts are unchanged, and policy events reconcile exactly.
+7. Cleanly stop maintenance PostgreSQL, start the canonical runtime, and verify retained queries, pipeline workers, sources, and queue movement.
+8. Roll back through the existing reviewed reactivation manifests plus restoration of the active query entries.
+
+## Open Questions
+
+None.
@@ -0,0 +1,26 @@
+## Why
+
+The remaining discovery rotations still contain source/query pairs that have each completed at least 1,000 successful scans without ever producing an `alive` credential. Their retained pending and deferred work consumes most of the avoidable backlog even though the existing audited `cold` lifecycle can preserve it reversibly.
+
+## What Changes
+
+- Retire only exact source/query pairs with at least 1,000 successful scans, zero historically `alive` linked credentials, and zero pending candidate checks at the evidence cutoff.
+- Keep a durable rejected-query registry with the source, exact query, evidence counters, cutoff, and reason `rejected_zero_alive` while removing each pair from active rotation.
+- Preserve dedicated source sentinel queries and evaluate the same query independently in different sources.
+- Move every eligible unfenced pending or deferred target attributed to the rejected pairs into the existing audited, reversible `cold` state instead of deleting queue or history rows.
+- Fail closed if attribution, evidence, policy identity, queue selection, or runtime quiescence changes between review and apply.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `discovery-keyword-pruning`: Add source/query-specific retirement based on substantial zero-`alive` evidence and retain explicit rejected-query evidence.
+- `target-queue-policy-holds`: Apply the existing audited cold lifecycle to every reviewed source scope in the retired cohort.
+
+## Impact
+
+The change affects `app/config.yaml`, query-policy validation and tests, private reviewed cold manifests, and `target_queue` policy events. The approved cohort contains 38 source/query pairs and 75,551 currently eligible queue rows. No target, scan, finding, credential, result, reservation, deduplication, or coverage record is deleted.
@@ -0,0 +1,56 @@
+## ADDED Requirements
+
+### Requirement: Source-specific zero-alive retirement rule
+The system SHALL retire an exact source/query pair only when it has at least 1,000 successful completed scans, zero historically alive linked credentials, and zero pending candidate checks at the reviewed evidence cutoff.
+
+#### Scenario: Historical alive result preserves the pair
+- **WHEN** any credential linked through a candidate occurrence for the exact source/query pair has ever produced `status_group='alive'`
+- **THEN** that source/query pair SHALL remain active regardless of the credential's current status
+
+#### Scenario: Pending candidate preserves the pair
+- **WHEN** an otherwise zero-alive source/query pair has a pending or leased candidate check
+- **THEN** the pair SHALL remain active until the candidate evidence matures
+
+#### Scenario: Insufficient scan exposure preserves the pair
+- **WHEN** a zero-alive mature pair has fewer than 1,000 successful completed scans
+- **THEN** the pair SHALL remain active
+
+#### Scenario: Exact pair qualifies for retirement
+- **WHEN** the exact pair has at least 1,000 successful completed scans, zero historical alive credentials, and zero pending or leased candidates
+- **THEN** the pair SHALL qualify for reviewed retirement without affecting the same query in another source
+
+### Requirement: Rejected query evidence remains visible
+Canonical configuration SHALL retain a machine-checkable rejection record for every retired source/query pair while keeping rejected pairs absent from active query rotation.
+
+#### Scenario: Rejected pair is loaded
+- **WHEN** canonical query policy is validated
+- **THEN** every rejected entry SHALL identify its exact source, query, status `rejected_zero_alive`, evidence cutoff, successful scans, findings, unique credentials, pending candidates, historical alive credentials, and reviewed queue count
+
+#### Scenario: Active and rejected policy overlaps
+- **WHEN** the same exact source/query pair appears in both active rotation and rejected evidence
+- **THEN** policy validation SHALL fail closed
+
+#### Scenario: Rejection evidence is incomplete
+- **WHEN** a rejected entry omits or weakens the approved zero-alive evidence fields
+- **THEN** policy validation SHALL fail closed
+
+## MODIFIED Requirements
+
+### Requirement: Operational and historical authority is preserved
+Keyword retirement SHALL stop future discovery and SHALL permit existing unfenced pending/deferred targets attributed to retired exact source/query pairs to enter an audited, reversible cold state without deleting or rewriting historical authority.
+
+#### Scenario: Dedicated source sentinels remain
+- **WHEN** archive, gist, CI-log, or HuggingFace source rotations are loaded
+- **THEN** their operational sentinel queries SHALL remain configured and SHALL NOT be evaluated as interchangeable discovery keywords
+
+#### Scenario: Persisted rotation index remains valid
+- **WHEN** an existing query index exceeds a shortened query list
+- **THEN** normal modulo-based rotation SHALL select a valid configured query without a state-file edit
+
+#### Scenario: Existing backlog is preserved but held
+- **WHEN** a previously admitted unfenced target is attributed to a retired exact source/query pair and selected by reviewed policy
+- **THEN** its queue row SHALL remain present with all attribution, retry, deduplication, scan, reservation, and coverage history preserved while its status becomes unclaimable `cold`
+
+#### Scenario: Historical records remain unchanged
+- **WHEN** a zero-alive policy cold transition is applied
+- **THEN** existing target scans, findings, candidates, credentials, keycheck results, completed queue rows, and coverage records SHALL NOT be deleted or rewritten
@@ -0,0 +1,37 @@
+## ADDED Requirements
+
+### Requirement: Reviewed zero-alive backlog is held across source scopes
+The system SHALL apply the existing exact audited cold lifecycle to every approved source/platform/query scope in a reviewed zero-alive cohort.
+
+#### Scenario: Eligible rejected-query row is selected
+- **WHEN** an unfenced `pending` or `deferred` row has an exact source/platform/query absent from active policy and present in approved rejected evidence
+- **THEN** the reviewed manifest SHALL be permitted to transition the row to `cold`
+
+#### Scenario: Rejected cohort contains a fenced row
+- **WHEN** any selected row has an active queue, resolver, reservation, or Docker content lease
+- **THEN** apply SHALL fail closed without partially transitioning that manifest
+
+#### Scenario: Retired pair is rediscovered
+- **WHEN** ordinary enqueue or rediscovery encounters a cold target previously attributed to a rejected pair
+- **THEN** the target SHALL remain cold until an explicit reviewed reactivation
+
+## MODIFIED Requirements
+
+### Requirement: Stale-query selection follows canonical policy
+The system SHALL evaluate query staleness using case-sensitive exact source, platform, and query policy derived from canonical active and rejected configuration.
+
+#### Scenario: Configured query remains active
+- **WHEN** a queue row's exact source/platform/query triple remains configured in active policy
+- **THEN** automatic stale-policy planning SHALL NOT select the row
+
+#### Scenario: Attribution cannot be classified safely
+- **WHEN** query attribution is null, blank, operational, non-rotation, or belongs to an unknown source/platform pair
+- **THEN** automatic planning SHALL skip and report the row rather than inferring retirement
+
+#### Scenario: Rejected source cohort is planned
+- **WHEN** policy planning is scoped to an affected source/platform from the reviewed zero-alive cohort
+- **THEN** it SHALL include only eligible pending/deferred rows attributed to exact rejected queries and SHALL expose no target values
+
+#### Scenario: Rejected evidence and active policy disagree
+- **WHEN** rejected evidence does not match the canonical active-query omission or approved evidence identity
+- **THEN** planning and apply SHALL fail closed
@@ -0,0 +1,22 @@
+## 1. Canonical Rejection Policy
+
+- [x] 1.1 Add strict machine-checkable rejected-query evidence validation with active/rejected disjointness and sentinel protection.
+- [x] 1.2 Record the approved 38 source/query pairs and evidence in canonical configuration while removing them from active rotations.
+- [x] 1.3 Make reviewed stale-row planning select only exact registered rejected pairs when a rejection registry is present.
+
+## 2. Verification
+
+- [x] 2.1 Add focused unit tests for evidence validation, exact source-specific retention, malformed evidence, overlap, and sentinel rejection.
+- [x] 2.2 Extend PostgreSQL policy-hold tests for multi-source exact rejected selection, fences, idempotency, history preservation, and reactivation.
+- [x] 2.3 Run focused configuration, migration, queue, integration, and strict OpenSpec validation.
+
+## 3. Reviewed Queue Transition
+
+- [x] 3.1 Stop maintenance PostgreSQL before editing authority-covered configuration, restart it afterward, and rerun the aggregate evidence cutoff.
+- [x] 3.2 Generate and review bounded privacy-safe cold manifests for every affected source/platform scope.
+- [x] 3.3 Apply the exact manifests with sources stopped and verify counts, append-only policy events, unclaimability, and unchanged history.
+
+## 4. Deployment
+
+- [x] 4.1 Cleanly stop maintenance PostgreSQL and start the canonical runtime with the reduced rotations.
+- [x] 4.2 Verify authenticated database/pipeline/source health, rejected-pair absence, retained query loading, and normal queue movement.