Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-20
@@ -0,0 +1,63 @@
+## Context
+
+GitLab repository scans use the shared TruffleHog Git command under the project's external supervisor, scan-slot leases, timeout enforcement, and Windows Job containment. Unlike Docker, Git commands still launch TruffleHog's embedded overseer. Twelve recent GitLab scans emitted no fatal diagnostic and no `finished scanning` marker, exited code 1, and were committed as non-retryable terminal failures after one attempt.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Give the existing external runtime sole lifecycle ownership for GitLab TruffleHog processes.
+- Require positive completion evidence for GitLab process success.
+- Retry incomplete GitLab scans through the existing three-attempt target policy.
+- Preserve partial findings and isolate behavior from other Git sources.
+- Validate with a GitLab-only canary and bounded exact-signature replay.
+
+**Non-Goals:**
+
+- Change GitLab discovery API behavior.
+- Change GitHub, package Git, or generic ad-hoc Git commands.
+- Upgrade TruffleHog or change detector selection.
+- Make clone-unavailable repositories retry forever.
+
+## Decisions
+
+### Enable local process mode only for configured GitLab scans
+
+`scan_git_repo` will accept an explicit lifecycle flag. GitLab source configuration enables it, causing the command to add `--local-dev`; callers that do not opt in retain the current command.
+
+Alternative considered: enable `--local-dev` for every Git source. Rejected because current production evidence is GitLab-specific and a narrow canary is easier to attribute and roll back.
+
+### Generalize explicit completion without changing existing defaults
+
+The diagnostic parser will accept an explicit completion-required option while preserving Docker as completion-required by default. GitLab local-process scans will enable that option. Missing completion or unexplained nonzero exit becomes retryable `command_incomplete`; a nonzero unexplained exit after completion becomes retryable `wrapper_exit`.
+
+Alternative considered: classify every Git RC=1 as retryable. Rejected because completion evidence distinguishes lifecycle interruption from explicit permanent diagnostics.
+
+### Reuse target retries and finding identity
+
+The existing maximum of three attempts, exponential delay, queue dispositions, finding UID assignment, and durable deduplication remain authoritative. Findings emitted before interruption are retained without marking the target complete.
+
+### Stage rollout and replay
+
+A controlled A/B run must first show that Git `--local-dev` completes a previously affected public target. Production then observes at least 100 GitLab attempts or two hours before a small exact-signature historical replay.
+
+## Risks / Trade-offs
+
+- [The flag behaves differently for Git than Docker] -> Require controlled A/B evidence and a GitLab-only canary.
+- [Retries increase clone traffic] -> Keep the existing attempt cap and replay only exact-signature targets in small batches.
+- [Completion logging changes upstream] -> Treat missing evidence as bounded incomplete coverage rather than success.
+- [Shared Git scanner plumbing leaks behavior] -> Use an explicit source-configured flag and assert non-GitLab command isolation.
+- [Partial findings repeat] -> Retain scan events while relying on existing stable finding identities downstream.
+
+## Migration Plan
+
+1. Run controlled baseline and local-process scans without exposing findings.
+2. Add command, completion, retry, and isolation regression tests.
+3. Enable the flag only for GitLab and restart the managed runtime.
+4. Observe at least 100 GitLab attempts or two hours with no new unexplained terminal RC=1.
+5. Replay one bounded exact-signature batch and verify queue disposition and deduplication.
+6. Roll back by disabling the GitLab lifecycle flag and restarting the source runtime.
+
+## Open Questions
+
+- Whether the policy should later become common to all externally supervised Git scans remains a separate change.
@@ -0,0 +1,28 @@
+## Why
+
+Recent GitLab repository scans produced 12 exit-code-1 outcomes without a fatal diagnostic or `finished scanning` marker. All were treated as non-retryable terminal failures after one attempt, leaving the same process-lifecycle coverage gap previously observed and corrected for Docker scans.
+
+## What Changes
+
+- Run GitLab TruffleHog Git scans without TruffleHog's redundant embedded overseer while retaining external supervision, scan-slot control, timeout enforcement, and Windows Job containment.
+- Require the normal completion marker before treating a GitLab scan process as complete.
+- Classify incomplete or unexplained GitLab process exits as retryable through the existing three-attempt target policy.
+- Preserve findings emitted before an incomplete exit.
+- Run a GitLab-only canary before replaying a bounded exact-signature historical batch.
+- Keep GitHub, package Git, and other Git scan behavior unchanged.
+
+## Capabilities
+
+### New Capabilities
+
+- `gitlab-scan-lifecycle`: Defines external lifecycle ownership, explicit completion, bounded retry, and controlled replay for GitLab repository scans.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+- Affects GitLab command construction and TruffleHog diagnostic disposition in `app/scanner.py` and source plumbing in `app/console_runner.py`.
+- Adds focused scanner and queue-policy regression coverage.
+- Does not change GitLab discovery requests, non-GitLab commands, detector selection, keychecks, or the installed TruffleHog binary.
@@ -0,0 +1,56 @@
+## ADDED Requirements
+
+### Requirement: External GitLab scan lifecycle ownership
+The system SHALL bypass TruffleHog's embedded overseer for configured GitLab repository scans while retaining external supervision, scan-slot leases, timeout enforcement, output bounds, and Windows Job containment.
+
+#### Scenario: GitLab command construction
+- **WHEN** the GitLab source constructs a TruffleHog Git command with external lifecycle enabled
+- **THEN** the command includes both `--local-dev` and `--no-update`
+
+#### Scenario: Non-GitLab command construction
+- **WHEN** another source constructs a TruffleHog Git command without external lifecycle enabled
+- **THEN** the command does not gain `--local-dev` or GitLab completion policy
+
+### Requirement: Explicit GitLab scan completion
+The system SHALL require both exit code 0 and the exact `finished scanning` marker before treating an externally managed GitLab TruffleHog process as complete.
+
+#### Scenario: Normal GitLab completion
+- **WHEN** the process exits code 0 after emitting `finished scanning`
+- **THEN** completion metadata is recorded and no lifecycle error is added
+
+#### Scenario: Missing GitLab completion marker
+- **WHEN** the process exits without emitting `finished scanning`
+- **THEN** the result is classified as retryable `command_incomplete` and is not treated as complete
+
+#### Scenario: Nonzero exit after completion marker
+- **WHEN** the process emits `finished scanning` and exits nonzero without a more specific fatal diagnostic
+- **THEN** the result is classified as retryable `wrapper_exit`
+
+### Requirement: Bounded retry for incomplete GitLab scans
+The system SHALL route incomplete GitLab lifecycle outcomes through the existing bounded target retry policy.
+
+#### Scenario: Retry remains available
+- **WHEN** an incomplete GitLab scan occurs before the configured maximum target attempt
+- **THEN** the queue defers the target using the configured retry delay
+
+#### Scenario: Attempt limit is reached
+- **WHEN** an incomplete GitLab scan occurs at the maximum target attempt
+- **THEN** the queue records a terminal failed target without an unbounded loop
+
+### Requirement: Partial GitLab finding preservation
+The system SHALL retain findings emitted before an incomplete GitLab process exit without representing the repository as fully scanned.
+
+#### Scenario: Findings precede incomplete exit
+- **WHEN** TruffleHog emits findings and then exits before completion is confirmed
+- **THEN** those findings remain durable while the target receives retryable incomplete disposition
+
+### Requirement: Controlled GitLab lifecycle rollout
+The system SHALL enable and replay GitLab lifecycle behavior only through bounded, observable stages.
+
+#### Scenario: Canary has not passed
+- **WHEN** the GitLab lifecycle canary has not reached its observation threshold
+- **THEN** historical terminal failures are not mass-requeued
+
+#### Scenario: Canary has passed
+- **WHEN** the canary is healthy and a bounded exact-signature batch is selected
+- **THEN** only targets in that batch are returned to the pending queue
@@ -0,0 +1,19 @@
+## 1. GitLab Lifecycle Policy
+
+- [x] 1.1 Run controlled baseline and `--local-dev` scans for previously affected public GitLab targets.
+- [x] 1.2 Add source-configured external lifecycle command construction only for GitLab scans.
+- [x] 1.3 Generalize explicit completion parsing and classify missing completion or wrapper exit as retryable.
+- [x] 1.4 Confirm partial findings use the existing three-attempt queue policy and stable identities.
+
+## 2. Regression Coverage
+
+- [x] 2.1 Add command tests proving GitLab receives `--local-dev` while other Git sources do not.
+- [x] 2.2 Add completion, incomplete exit, wrapper exit, partial finding, and queue exhaustion tests.
+- [x] 2.3 Run focused and broader regression suites with bytecode writes disabled.
+
+## 3. Validation And Rollout
+
+- [x] 3.1 Run strict OpenSpec validation and verify implementation against artifacts.
+- [x] 3.2 Restart the managed runtime and confirm GitLab command, source, scan-slot, PostgreSQL, and pipeline health.
+- [x] 3.3 Observe at least 100 GitLab attempts or two hours before historical replay.
+- [x] 3.4 Requeue one bounded exact-signature historical batch and verify completion, deduplication, and queue drain.