sashatrask
53 lines
4.2 KiB
Markdown
53 lines
4.2 KiB
Markdown
## Context
|
|
|
|
The current Streamlit dashboard has six visible pages. PostgreSQL is authoritative for scans, findings, queues, and keychecks, but several pages still read compatibility files or filter a pre-limited client-side frame. The dashboard is read-only, loopback-only, supervisor-managed, and disabled by default; those safety boundaries must remain.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Present the operational summary and finding lookup on one understandable page.
|
|
- Apply one selected UTC time window consistently to all historical statistics.
|
|
- Use PostgreSQL for every result, queue, and validation value.
|
|
- Accept convenient pasted lookup values without querying or rendering raw secret columns.
|
|
- Keep queries bounded and preserve degraded behavior when PostgreSQL is unavailable.
|
|
|
|
**Non-Goals:**
|
|
- Add scanner lifecycle controls or expose the dashboard beyond loopback.
|
|
- Add a REST service, database migration, or write path.
|
|
- Preserve retired queue-file and compatibility-file diagnostics in the visible UI.
|
|
- Turn the dashboard into a raw log or raw credential browser.
|
|
|
|
## Decisions
|
|
|
|
1. **Keep Streamlit and replace only the visible information architecture.** The existing supervisor launch, dependency isolation, read-only connection, redaction, and health checks are valuable. A new frontend/backend split would add operational surface without improving this local dashboard.
|
|
|
|
2. **Render one page with four sections.** The order is lookup, time-window KPIs, source/alive breakdowns, then compact runtime health. The lookup is placed first because it is a direct operator task; all secondary diagnostics are omitted or collapsed.
|
|
|
|
3. **Use bound UTC timestamps in SQL before aggregation and limits.** Presets cover 1 hour, 24 hours, 7 days, and 30 days; custom start/end values are converted to UTC. This replaces the current newest-5000-rows client filter.
|
|
|
|
4. **Use only authoritative stores.** Historical metrics come from `source_cycles`, `target_scans`, `findings`, `keycheck_credentials`, `keycheck_current_state`, and `keycheck_results`. Current backlog and pipeline state come from PostgreSQL. Supervisor status and `scan_limiter.db` remain valid operational sidecars. Retired queue files, global runner state, TSV summaries, and JSONL metrics are not rendered.
|
|
|
|
5. **Normalize lookup input before database access.** A 64-hex value is treated as a digest; numeric and UID-like values are searched by indexed identity; credential-like text is SHA-256 hashed in memory and only the digest reaches SQL. Short or structured non-secret metadata uses escaped `LIKE` predicates. No query selects raw payload columns, and the submitted raw value is never echoed.
|
|
|
|
6. **Lookup overrides statistical filters.** Search results always show current validation state and all bounded matching origins regardless of the selected reporting period or access-tier preset. This avoids hiding a valid credential behind an unrelated dashboard filter.
|
|
|
|
7. **Retain bounded rendering and query failure isolation.** Lookup origins and breakdowns have explicit limits. Query failure rolls back the PostgreSQL transaction and degrades only the affected section.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **Raw pasted credentials traverse the local Streamlit session before hashing** -> Keep loopback-only authority, hash immediately, never log/echo/persist the input, and offer SHA-256 lookup as the safest path.
|
|
- **Broad metadata lookup can be expensive** -> Escape wildcard characters, require a useful minimum length, search indexed exact identities first, and cap returned origins.
|
|
- **Removing advanced pages hides forensic diagnostics** -> PostgreSQL and log files remain available to engineering tools; the operator dashboard intentionally prioritizes clarity.
|
|
- **Existing helper functions may remain temporarily unused** -> Remove the visible routes first and prune only when tests establish that no safety helper depends on them.
|
|
|
|
## Migration Plan
|
|
|
|
1. Add and test the new query/lookup helpers.
|
|
2. Switch `main()` to the single page and disable Streamlit telemetry.
|
|
3. Run focused dashboard tests and a live supervisor-launched desktop/mobile smoke test.
|
|
4. Keep dashboard startup disabled by default. Rollback is a source revert; no persisted data changes are involved.
|
|
|
|
## Open Questions
|
|
|
|
None required for the initial implementation.
|