truf-server/docs/worker-parallelism-validation-2026-09-23.md

# Worker Parallelism Validation, 2026-09-23

## Scope

This report records the production validation performed on `sec` using the local
Windows computer:

- bounded discovery for GitLab, DockerHub, and HuggingFace;
- a native Windows protocol-2 worker with client and server parallelism 3;
- a dual-worker run with native Windows parallelism 1 and the existing trusted
  WSL production worker at cap 1;
- authoritative reconciliation from discovery through queue, reservation,
  receipt, bundle ingestion, scan settlement, projection, and JSONL append;
- restoration of production source/capacity settings and normal controls.

No provider credential, device token, raw target, raw finding, runtime YAML, or
worker command line is included in this report.

## Result

The core validation passed.

- Native Windows reached exact unfinished concurrency 3 and never exceeded 3.
- The dedicated Windows cohort accounted for 180 accepted assignments across all
  three sources.
- Windows and WSL independently reached concurrency 1 at the same time, for exact
  combined concurrency 2, and never exceeded their individual caps.
- All accepted results were ingested, queue-settled, projected, and appended as
  required.
- Final cohort and global lineage checks reported zero violations.
- No pipeline quarantine or failed hold was created.
- Production controls and the WSL worker were restored.

The originally requested 10-hour observation was not completed continuously.
After successful active parallelism-3 work, 14,105 seconds (about 3 hours 55
minutes) of detached idle stability observation completed before the user changed
the objective to the dual-worker test. The active concurrency evidence itself is
complete; the residual limit is soak duration, not functional coverage.

## Baseline

The fresh pre-test snapshot was stored root-only on `sec`:

- Path: `/opt/truf-remote-server/staging/windows-p3-before2.json`
- SHA-256: `469bcd69888cacce55523fef17508ee48191c4c879a6d39f2251abad4aa0efdb`
- Active config SHA-256:
  `e48797689ab0ee7328d21cf5c23d0ffedb979dba492bf49d1da3afe4b075cf5b`
- Config bytes: 37,233
- Controls: revision 56, discovery open, dispatch open, drain normal
- Existing blocker: one stale production assignment, allowed to settle naturally
- Runtime image:
  `sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf`
- Failed hold: absent
- All recorded orphan/mismatch invariants: zero

Baseline high-water values included reservation 929, scan 926, projection job
926, projection append 984, error 1919, finding 127, and zero quarantine rows.

The existing WSL assignment was never canceled or directly mutated. Dispatch was
paused and the lease was allowed to resolve before changing worker caps.

## Windows Package

The trusted run artifact was built from pinned local caches:

- Package directory:
  `build/worker-parallelism3-20260922/package-fixed1`
- Archive:
  `build/worker-parallelism3-20260922/truf-worker-windows-x86_64-fixed1.zip`
- Archive SHA-256:
  `6ca460a47d31606c430e8c6f081443530892332cfc183fc0c6edfe7568a454cc`
- Manifest file SHA-256:
  `cd55820e0a04a04ded9ba340b9c6e5c6259422aab80989180a6ba28ff3b77103`
- Canonical code-manifest SHA-256:
  `e981da19283aad5971fc2865268bb8151b3540d71fbd057ae56d9d2379993374`
- Platform: `windows-x86_64`
- Protocol and bundle schema: v2
- Capabilities: GitLab `exact_git_v1`, DockerHub `docker_direct_v1`,
  HuggingFace `huggingface_space_v1`

The registered Windows v2 manifest was stale relative to the current production
authority files. It was not overwritten. The fixed package was registered as the
new root-owned `windows-worker-package-v3.json` trust authority.

### Reproducibility defect

Two initial package builds differed only in four timestamp bytes in each
pip-generated Windows launcher executable and the corresponding RECORD hashes.
The package builder now normalizes the embedded launcher ZIP timestamps to the
DOS epoch and recomputes the RECORD rows.

Changed local source:

- `app/worker_package_builder.py`
- `tests/test_worker_package.py`

Verification:

- Two post-fix archives were byte-identical.
- `python -m pytest tests/test_worker_package.py -q`: 16 passed.

## Discovery

The temporary discovery candidate was derived from the exact active config.

GitLab used 16 reviewed keyword queries, one API page, one result per page, and
one target maximum per query. DockerHub used 16 reviewed keyword queries, one
repository result per query, and one image per repository.

HuggingFace was deliberately reported differently: the configured implementation
does not perform keyword search. It requests the newest-modified Spaces with a
fixed API limit of 100. `pages: 1` therefore means one real recent page, not one
keyword result. Repeated polls occurred while GitLab and DockerHub rotated their
queries; deduplication bounded the admitted rows.

Final bounded discovery evidence:

- GitLab: 16/16 distinct successful query rotations, 7 new queue rows
- DockerHub: 16/16 distinct successful query rotations, 10 new queue rows
- HuggingFace: 17 successful recent-page cycles, 31 new queue rows
- Discovery failures: 0
- Hard limits: GitLab 20, DockerHub 20, HuggingFace 120; none exceeded

## Native Windows Parallelism 3

An isolated temporary server user and device were created with typed, audited
`AdminService` operations. The one-time token was transferred privately, the
server transfer file was deleted after successful launch, and the native worker
used a dedicated protected `LOCALAPPDATA` state tree.

The first foreground launch exposed a local automation issue: the tool runner
retained the descendant process and did not return. The worker itself survived.
Subsequent starts used `Win32_Process.Create` so the worker was genuinely
detached. No process command line was inspected.

### Capacity findings

Client `--parallelism 3` and server user cap 3 were not by themselves enough to
permit three physical scans.

The first run reached only one active reservation because the production config
had:

- `global.max_active_scans = 1`
- 64 MiB maximum event size
- 256 MiB projection backlog maximum
- 128 MiB projection headroom

The capacity model reserves twice the event maximum per assignment, plus
headroom. Three slots therefore require 512 MiB. The temporary test candidate
changed only:

- `max_active_scans: 1 -> 3`
- `projection_backlog_max_bytes: 256 MiB -> 512 MiB`

A second run still reached only one assignment. Admission evidence showed
`pipeline_capacity_closed` on the keycheck axis. The stable keycheck backlog was
127 items and each assignment reserved up to 2,000 items. The production limit
of 4,096 allowed one reservation but not three. The temporary candidate changed:

- `keycheck_queue_max_items: 4096 -> 8192`

No unrelated limit was increased.

### Successful run

After both capacity corrections:

- Exact max unfinished concurrency: 3
- Never exceeded: 3
- Issued: 180
- Accepted: 180
- Prebundle failures: 0
- Expired: 0
- Unfinished after settlement: 0
- Accepted-not-ingested: 0
- Queue-not-settled: 0
- Projection-not-completed: 0

Source totals:

| Source | Accepted |
|---|---:|
| DockerHub | 63 |
| GitLab | 65 |
| HuggingFace | 52 |

Scan outcomes were transport-successful but not necessarily target-clean:

| Source | Outcome | Count |
|---|---|---:|
| DockerHub | clean | 44 |
| DockerHub | degraded | 18 |
| DockerHub | error | 1 |
| GitLab | clean | 57 |
| GitLab | error | 8 |
| HuggingFace | clean | 43 |
| HuggingFace | error | 9 |

These scan statuses are provider/target results. They are not lost transport,
ingestion, or projection records. All 180 authoritative results settled.

## Stability Observation

After the 180-assignment cap closed dispatch for the test identity, the worker
remained online at cap 0 under detached monitoring.

- Successful checks: 43
- Elapsed observation: 14,105 seconds
- Runtime restarts: 0
- Edge restarts: 0
- Native worker OOM/restart: none
- Pipeline debt: 0
- Quarantine: 0
- Global invariants: 0

A single strict-health probe can observe a discovery producer during its normal
short startup window. Monitors were corrected to require failure across three
attempts with 20-second delays rather than treating one transient sample as a
production defect.

## Dual Worker Validation

The user then requested a second topology:

- native Windows worker: real client parallelism 1 and server cap 1;
- existing trusted production WSL worker: cap 1;
- desired combined concurrency: 2.

The native worker was cleanly stopped at cap 0, relaunched from the same trusted
package and protected state with parallelism 1, and verified by PID/resource
evidence. The production WSL token and command line were never inspected.

### Monitor corrections

Several fail-closed attempts improved the monitoring policy without losing or
canceling work:

- A scanner subprocess can remain active without an HTTP contact until it
  reports. Contact age alone must not fail an identity with unfinished work.
- A worker may honor a prior cap-0 `Retry-After` after caps change. Arming uses a
  separate 600-second threshold; the 180-second idle threshold applies only
  after dispatch has opened.
- Operation IDs are one-use and identity-bound. Every retry used a fresh audited
  operation namespace.
- A transient PostgreSQL connection timeout while reopening a read-only monitor
  connection terminated observability after caps and gates were already safely
  closed. The monitor now retries database opens three times with 20-second
  delays. A separate read-only settlement monitor completed the active run.

### Attempt 1 evidence

The first active dual run already proved max concurrency Windows 1, WSL 1,
combined 2. It fail-closed because of the old contact policy while WSL was doing
a long DockerHub scan.

- Windows: 19 accepted assignments
- WSL: 2 accepted HuggingFace assignments
- WSL: 1 DockerHub reservation naturally expired after its two-hour lease
- No assignment was canceled

### Final attempt 4

The final bounded run used baseline reservation ID 1131.

- Issued: 30
- Windows issued/accepted: 29/29
- WSL issued: 1
- WSL accepted: 0
- WSL expired/refunded: 1 long DockerHub assignment after its two-hour lease
- Max Windows concurrency: 1
- Max WSL concurrency: 1
- Max combined concurrency: 2
- Accepted-not-ingested: 0
- Queue-not-settled: 0
- Projection-not-completed: 0
- Cohort integrity violations: 0
- Global invariant violations: 0
- Failed holds in the cohort window: 0
- Quarantine bytes/items: 0/0

Windows source/outcome totals in the final run:

| Source | Outcome | Count |
|---|---|---:|
| DockerHub | clean | 12 |
| GitLab | clean | 7 |
| HuggingFace | clean | 9 |
| HuggingFace | error | 1 |

The WSL expiry does not invalidate the concurrency result: both independent
clients simultaneously held one real reservation and never exceeded cap. Earlier
in attempt 1, the same WSL path also completed two accepted HuggingFace results.

Authoritative dual evidence:

- `/opt/truf-remote-server/staging/windows-dual-worker-report.json`
- SHA-256:
  `8fd03bf79960393406ef1a0f788d90db3ed3521d32ffa83f802c9d1c2c3509bf`

## Data Reconciliation

The validation distinguished assignment acceptance, bundle ingestion, queue
settlement, and projection completion.

Checks covered:

- accepted reservation has a durable receipt;
- accepted reservation has exactly one matching bundle;
- reservation/bundle/scan event IDs and hashes agree;
- target scan references the same queue row;
- queue completion is `applied`;
- projection job exists, matches the event identity, and is completed;
- required `scan_results` append exists;
- required `found_secrets` append exists when requested by the mask;
- projection append event identity matches the job;
- no orphan bundle, scan, finding, error, or projection job exists;
- JSONL files are present, newline-terminated, and cursor offsets match.

All cohort and global mismatch counters were zero at the final report points.

## Code Defects Fixed

### Integer cap zero

`AdminService._validate_cap()` used `str(value or '')`, converting integer zero
to an empty string even though cap 0 is valid. It now distinguishes `None` from
zero.

Changed local source:

- `app/admin_api.py`
- `tests/test_admin_api.py`

Verification:

- `python -m pytest tests/test_admin_api.py -q`: 62 passed.

The currently deployed runtime image predates this source fix. Run helpers used
canonical string `"0"` through the same typed/audited API; no direct database
mutation was used.

### Operational helpers

Run-specific helpers were added under `build/sec-deploy/` for config lifecycle,
controls, typed worker administration, monitoring, observation, local launch,
safe stop, settlement, and reporting. Python helpers passed `py_compile` and
Ruff; PowerShell helpers passed parser validation before use.

## Configuration Restoration

Temporary active config identities were:

| Stage | SHA-256 |
|---|---|
| Bounded discovery + v3 profile | `ec2ae20f9a23ee9ec423c28f7da0b45db3eaff17abb3f484e1a1f0c9227fada4` |
| Parallel scan/projection capacity | `4fee05f3a9ae001767f2de0388dcd5db75183dd954be8e024e98f47d4b64b6b1` |
| Parallel keycheck capacity | `b3509aacc53ab0bfa1e6b13dd789f889d9fea34daf0f680579d2a54697393bf9` |
| Restored production semantics + v3 trust | `c0966cac4f7f0610a813fa8732e91953f2e3e838ad880f91fd1a9437096925c7` |

The final config restores the original production discovery/source settings,
global scan capacity, projection capacity, keycheck capacity, and supervisor
interval. The only intentional permanent semantic difference from the initial
config is the Windows compatibility profile path changing from stale v2 to the
reproducible/current v3 manifest. Therefore the final config hash intentionally
differs from the initial hash while retaining the same 37,233-byte size.

All four managed config apply operations succeeded and reconciled. The final
apply operation was:

- `a9943967-372e-522a-8386-4f9465cc03f9`

The complete root-only operation export contains 69 operations for the run
actor, all with status `succeeded`:

- `/opt/truf-remote-server/staging/windows-p3-operations.json`
- SHA-256:
  `318aaa56cc2ba348e905b72fd63d7795e24cbd1aa9e84d0a75fa341744ef0acf`

## Final Production State

The temporary native process was stopped only after its local bundle/work trees
and authoritative pipeline were empty.

Typed cleanup completed:

- temporary Windows cap: 0
- temporary device: revoked
- temporary user: disabled
- server plaintext token transfer: absent
- production WSL cap: 1

Controls after restoration:

- revision: 98
- discovery: open
- dispatch: open
- drain: normal
- effective gates: open

The production WSL container was idle but still honoring a previous cap-0 retry
delay. With zero unfinished work it was safely restarted using the same container,
token, and state. It immediately made a fresh server contact and received a new
production assignment, proving restored admission.

At the final snapshot:

- production WSL contact was fresh and one real production assignment was active;
- blocker count 1 was therefore expected, not a drain defect;
- WSL container running, OOM false, restart count 0;
- runtime healthy, restart count 0;
- edge running, restart count 0;
- host-agent, host Caddy, and X-UI active;
- strict runtime health: ACTIVE/READY with DockerHub, GitLab, HuggingFace,
  janitor, JSONL projector, result ingester, and Worker API;
- failed hold absent.

Runtime image remained:

`sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf`

## Final Snapshot

- Path: `/opt/truf-remote-server/staging/windows-p3-final.json`
- SHA-256: `1e3ab6bdb50c5be1416974bf56235c220166a6b731e76a87d8abebb947aae074`
- Bytes: 6,883
- Config SHA-256:
  `c0966cac4f7f0610a813fa8732e91953f2e3e838ad880f91fd1a9437096925c7`
- Reservation high-water: 1165
- Scan high-water: 1159
- Projection job high-water: 1159
- Projection append high-water: 1239
- Error high-water: 1945
- Finding high-water: 127
- Quarantine rows: 0
- All six recorded global orphan/mismatch invariants: 0
- `scan_results.jsonl`: 4,592,879 bytes, cursor exact, newline-terminated
- `found_secrets.jsonl`: 146,366 bytes, cursor exact,
  newline-terminated

Final strict-health evidence:

- `/opt/truf-remote-server/staging/windows-p3-final-health.json`
- SHA-256:
  `68c2b0dcd16591b2407fefcb5c3921aa7f58443011539ff912e2cb19b9fdef13`

## Retained Local Evidence

Per the user's explicit instruction on 2026-09-23, no additional local files were
deleted during final reporting.

The following remain intentionally retained:

- protected local one-time token file;
- dedicated native worker state directory;
- worker stdout/stderr and PID evidence;
- fixed package and both reproducibility builds;
- local run-specific helpers and reports.

The token is no longer usable because the server device is revoked and user is
disabled, but the local protected file remains until the user authorizes deletion.

## Residual Limits

- The continuous 10-hour soak was shortened by the later dual-worker request.
- HuggingFace configured discovery is recent-page discovery, not keyword search.
- Long DockerHub work twice reached the fixed two-hour WSL lease and expired;
  expiry/refund behavior was correct, but those targets did not produce bundles.
- The local source fixes for package reproducibility and integer cap zero are
  tested in this checkout but were not deployed as a new production runtime
  image during this validation.
- Local sensitive/state evidence is deliberately retained and requires a later
  explicit cleanup instruction.