Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-02
@@ -0,0 +1,52 @@
+## Context
+
+The current Streamlit dashboard has six visible pages. PostgreSQL is authoritative for scans, findings, queues, and keychecks, but several pages still read compatibility files or filter a pre-limited client-side frame. The dashboard is read-only, loopback-only, supervisor-managed, and disabled by default; those safety boundaries must remain.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Present the operational summary and finding lookup on one understandable page.
+- Apply one selected UTC time window consistently to all historical statistics.
+- Use PostgreSQL for every result, queue, and validation value.
+- Accept convenient pasted lookup values without querying or rendering raw secret columns.
+- Keep queries bounded and preserve degraded behavior when PostgreSQL is unavailable.
+
+**Non-Goals:**
+- Add scanner lifecycle controls or expose the dashboard beyond loopback.
+- Add a REST service, database migration, or write path.
+- Preserve retired queue-file and compatibility-file diagnostics in the visible UI.
+- Turn the dashboard into a raw log or raw credential browser.
+
+## Decisions
+
+1. **Keep Streamlit and replace only the visible information architecture.** The existing supervisor launch, dependency isolation, read-only connection, redaction, and health checks are valuable. A new frontend/backend split would add operational surface without improving this local dashboard.
+
+2. **Render one page with four sections.** The order is lookup, time-window KPIs, source/alive breakdowns, then compact runtime health. The lookup is placed first because it is a direct operator task; all secondary diagnostics are omitted or collapsed.
+
+3. **Use bound UTC timestamps in SQL before aggregation and limits.** Presets cover 1 hour, 24 hours, 7 days, and 30 days; custom start/end values are converted to UTC. This replaces the current newest-5000-rows client filter.
+
+4. **Use only authoritative stores.** Historical metrics come from `source_cycles`, `target_scans`, `findings`, `keycheck_credentials`, `keycheck_current_state`, and `keycheck_results`. Current backlog and pipeline state come from PostgreSQL. Supervisor status and `scan_limiter.db` remain valid operational sidecars. Retired queue files, global runner state, TSV summaries, and JSONL metrics are not rendered.
+
+5. **Normalize lookup input before database access.** A 64-hex value is treated as a digest; numeric and UID-like values are searched by indexed identity; credential-like text is SHA-256 hashed in memory and only the digest reaches SQL. Short or structured non-secret metadata uses escaped `LIKE` predicates. No query selects raw payload columns, and the submitted raw value is never echoed.
+
+6. **Lookup overrides statistical filters.** Search results always show current validation state and all bounded matching origins regardless of the selected reporting period or access-tier preset. This avoids hiding a valid credential behind an unrelated dashboard filter.
+
+7. **Retain bounded rendering and query failure isolation.** Lookup origins and breakdowns have explicit limits. Query failure rolls back the PostgreSQL transaction and degrades only the affected section.
+
+## Risks / Trade-offs
+
+- **Raw pasted credentials traverse the local Streamlit session before hashing** -> Keep loopback-only authority, hash immediately, never log/echo/persist the input, and offer SHA-256 lookup as the safest path.
+- **Broad metadata lookup can be expensive** -> Escape wildcard characters, require a useful minimum length, search indexed exact identities first, and cap returned origins.
+- **Removing advanced pages hides forensic diagnostics** -> PostgreSQL and log files remain available to engineering tools; the operator dashboard intentionally prioritizes clarity.
+- **Existing helper functions may remain temporarily unused** -> Remove the visible routes first and prune only when tests establish that no safety helper depends on them.
+
+## Migration Plan
+
+1. Add and test the new query/lookup helpers.
+2. Switch `main()` to the single page and disable Streamlit telemetry.
+3. Run focused dashboard tests and a live supervisor-launched desktop/mobile smoke test.
+4. Keep dashboard startup disabled by default. Rollback is a source revert; no persisted data changes are involved.
+
+## Open Questions
+
+None required for the initial implementation.
@@ -0,0 +1,29 @@
+## Why
+
+The existing six-page dashboard mixes PostgreSQL authority with retired compatibility files, producing contradictory queue counts and incomplete time-window statistics. Operators need one fast, obvious view that answers the same questions as the current manual status checks and can locate a finding without exposing raw credentials.
+
+## What Changes
+
+- Replace the visible multi-page dashboard with one PostgreSQL-authoritative observability page.
+- Add accurate preset and custom time windows applied in SQL before aggregation or row limits.
+- Show scanner activity, findings, errors, new and current alive credentials, queue state, and compact runtime health in one view.
+- Add unified lookup by pasted credential, SHA-256 identity, finding ID/UID, target, path, commit, or other redacted metadata.
+- Hash credential-like lookup input immediately and never query or render raw secret columns.
+- Remove legacy queue-file, global runner-state, TSV-gated, log-browsing, and advanced/debug panels from the visible interface.
+- Keep the dashboard read-only, loopback-only, and supervisor-managed.
+
+## Capabilities
+
+### New Capabilities
+- `single-page-observability`: Accurate time-window scanner statistics, current runtime state, alive credential summaries, and safe finding lookup on one page.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+- Primary implementation: `app/dashboard.py`.
+- Focused behavior and security coverage: `tests/test_dashboard_behavior.py` and `tests/test_dashboard_secret_guard.py`.
+- PostgreSQL remains authoritative; no database migration, mutation endpoint, new service, or external API is introduced.
+- Existing supervisor lifecycle and disabled-by-default startup policy remain unchanged.
@@ -0,0 +1,72 @@
+## ADDED Requirements
+
+### Requirement: Single-page operator view
+The dashboard SHALL present finding lookup, reporting metrics, source results, alive credential results, and current runtime health on one page without legacy page navigation.
+
+#### Scenario: Operator opens the dashboard
+- **WHEN** the supervisor-launched dashboard session connects
+- **THEN** one page presents the primary lookup and observability sections in operational priority order
+
+#### Scenario: Retired diagnostics remain hidden
+- **WHEN** the page renders successfully
+- **THEN** it does not present compatibility queue files, global runner state, TSV-gated summaries, raw logs, or advanced/debug navigation
+
+### Requirement: Accurate reporting window
+The dashboard SHALL apply the selected start and end timestamps in PostgreSQL before aggregating or limiting historical rows.
+
+#### Scenario: Preset period is selected
+- **WHEN** the operator selects 1 hour, 24 hours, 7 days, or 30 days
+- **THEN** all historical KPI and breakdown queries use that exact UTC interval
+
+#### Scenario: Custom period is selected
+- **WHEN** the operator supplies a valid custom start and end
+- **THEN** the page reports data only from the normalized custom interval
+
+#### Scenario: Invalid custom period is supplied
+- **WHEN** the end is not later than the start
+- **THEN** the dashboard explains the error and does not run historical aggregation queries
+
+### Requirement: PostgreSQL-authoritative summary
+The dashboard SHALL derive scanner, finding, queue, candidate, and validation statistics from PostgreSQL and SHALL label current runtime values separately from period values.
+
+#### Scenario: Period contains activity
+- **WHEN** scans and keychecks exist in the selected interval
+- **THEN** the page shows scanned targets, findings, errors, newly alive credentials, current alive credentials, and grouped source/provider results
+
+#### Scenario: No period activity exists
+- **WHEN** no matching historical rows exist
+- **THEN** the page renders zero-valued KPIs and clear empty states without falling back to compatibility files
+
+### Requirement: Safe unified finding lookup
+The dashboard SHALL locate current validation state and finding origins using a pasted credential, SHA-256 identity, finding ID/UID, masked value, target, path, commit, or other redacted metadata without selecting raw database payload columns.
+
+#### Scenario: Raw credential-like value is pasted
+- **WHEN** the operator submits a credential-like value
+- **THEN** the dashboard hashes it in memory and sends only its SHA-256 identity to PostgreSQL
+
+#### Scenario: Exact identity is pasted
+- **WHEN** the operator submits a finding ID, finding UID, fingerprint, or SHA-256 identity
+- **THEN** indexed exact predicates locate matching current status and bounded origins independently of reporting filters
+
+#### Scenario: Non-secret metadata is pasted
+- **WHEN** the operator submits a sufficiently specific target, path, commit, detector, source, or query fragment
+- **THEN** escaped metadata predicates return bounded redacted matches
+
+#### Scenario: Match has validation state
+- **WHEN** a finding or credential is linked to current keycheck state
+- **THEN** the result includes service, provider status, status group, checked time, source, query, target, and available location metadata
+
+### Requirement: Read-only safety boundaries
+The simplified dashboard SHALL remain supervisor-authorized, loopback-only, read-only, redacted, and bounded.
+
+#### Scenario: Dashboard issues database queries
+- **WHEN** any page section loads or a lookup is submitted
+- **THEN** no mutation statement or forbidden raw payload column is requested
+
+#### Scenario: PostgreSQL query fails
+- **WHEN** a query times out or the connection enters an error transaction
+- **THEN** the dashboard rolls back and renders a bounded degraded message instead of failing the process
+
+#### Scenario: Dashboard is launched without authority
+- **WHEN** the process lacks canonical supervisor and loopback launch markers
+- **THEN** startup is refused before argument parsing or database access
@@ -0,0 +1,17 @@
+## 1. Authoritative Data Model
+
+- [x] 1.1 Add validated preset/custom UTC reporting-window helpers.
+- [x] 1.2 Add PostgreSQL-authoritative KPI, source activity, alive breakdown, queue, and runtime queries.
+- [x] 1.3 Add safe lookup normalization and bounded current-status/origin queries.
+
+## 2. Single-Page Interface
+
+- [x] 2.1 Build the one-page lookup, KPI, breakdown, and runtime layout with clear empty/error states.
+- [x] 2.2 Replace visible legacy navigation and compatibility panels with the single-page route.
+- [x] 2.3 Disable Streamlit usage telemetry while retaining loopback supervisor authority and disabled-by-default startup.
+
+## 3. Verification
+
+- [x] 3.1 Add focused tests for time-window validation, SQL-before-limit behavior, lookup hashing, identity search, and raw-column exclusion.
+- [x] 3.2 Run dashboard and security tests, then the broader relevant suite.
+- [x] 3.3 Launch through the supervisor and smoke-test desktop/mobile rendering, search, accurate 24-hour totals, and clean shutdown.