truf-server/openspec/changes/add-web-operations-control-plane/HANDOFF.md

Implementation Handoff

Updated: 2026-09-21

This file preserves implementation context for the active OpenSpec change add-web-operations-control-plane. It is a working handoff, not a normative specification. The OpenSpec artifacts and tasks.md remain authoritative.

User Direction

• Continue implementation autonomously and pragmatically.
• Avoid overengineering. Ask through the question tool only when a real product or architecture choice is unclear.
• Keep changes minimal and scoped to the current OpenSpec task.
• Do not modify or remove unrelated worktree changes.
• Do not touch the production server unless explicitly requested. The only previously approved remote host was sec; prod was explicitly excluded.

Non-Negotiable Provider Architecture

These decisions are also recorded in the repository AGENTS.md.

• The server validates assignment shape and canonical immutable identity.
• Git planning may bind an exact commit.
• Docker planning may resolve a mutable tag to an immutable digest.
• These planning operations are not provider-access proofs.
• The worker is the final authority for real provider access.
• Discovery credentials do not enter direct assignment fields.
• Do not add server-side provider probes, durable public-access proof state, proof TTL/freshness migrations, broad worker environment scrubbing, credential sandboxes, or post-hoc redaction pipelines without explicit user approval and a new OpenSpec requirement/task.
• Worker ambient HOME/XDG/Git/Docker/provider environment belongs to the worker operator.
• Permanent provider outcomes include target-scoped auth/access/not-found.
• Retryable provider outcomes include network/rate-limit/provider 5xx failures.

OpenSpec Progress

• Change: add-web-operations-control-plane
• Schema: spec-driven
• Completed through task 9.4.
• Current count: 50/58 complete.
• Current task: 9.5 (fixed host installation and production wiring).

Major Completed Work

Operations authority and distributed sources

• Added PostgreSQL/SQLite operation-control, operation, and append-only audit authority in migration 29.
• Discovery and dispatch pause gates and drain reconciliation are transactional.
• Server producers are discovery-only for GitLab, DockerHub, and HuggingFace.
• Protocol 2 and package schema 3 support exact GitLab, Docker direct, and HuggingFace Space assignment capabilities.
• New assignment claims are protocol 2; protocol 1 remains completion-only.

Removed rejected infrastructure

• Docker anonymous-access proof/preflight machinery was removed.
• Proof columns, claim gates, resolver proof handling, and migration 30 were removed; migration count remains 29.
• Broad Docker/HuggingFace child-environment sanitization was removed.
• Docker tag-to-digest immutable planning remains.

Worker/package and canary verification

• Windows/Linux packaged-worker E2E covers real GitLab scan/recovery plus synthetic DockerHub/HuggingFace protocol-2 claim-to-ingestion flows.
• Bounded test-only DockerHub canary covers discovery, enqueue, claim, worker-classified permanent/retryable failures, upload, ingestion, projection, replay, expiry, and drain.
• Synthetic test transport is test-only. The live public DockerHub canary is task 10.5.

Runtime documents

• Strict bounded UTF-8 YAML loader rejects duplicate keys, including effective merge collisions, and exposes content-free errors.
• Combined config/secrets validator enforces exact types, bounds, core profile, auth pools/references, package capability evidence, and deployment paths.
• Startup, Supervisor launch, preview, and secrets import use the shared validator without replacing existing lifecycle/path security checks.
• Candidate files use fixed paths:
  • /data/runtime-document-candidates/config.yaml
  • /data/runtime-document-candidates/secrets.yaml
  • /data/runtime-document-candidates/candidate.lock
• Candidate storage uses private descriptor-based reads, raw SHA-256 CAS, durable atomic replacement, bounded config diffs, and aggregate-only secrets diffs.
• Candidate concurrency/error-redaction tests cover stale active/candidate revisions, hardlinks/symlinks, interrupted writes, rollback, lock contention, cancellation locals, and orphan temporary cleanup.

Supervisor and operation services

• Control protocol stays schema 1 for compatibility.
• Legacy textual snapshot/command remains for CLI and health compatibility.
• Web-facing paths use only exact typed actions; they never call the generic parser or shell.
• Structured runtime/source snapshot covers every configured managed child.
• Typed lifecycle/settings support all practical Supervisor-managed children, including dashboard through its separate exact action.
• Bounded log tail accepts only an exact managed-source ID and line count.
• Durable operation methods support apply/restart, managed-source actions, worker-admin mutations, and content-free hash-chained audits.

Web console through task 7.5

• Caddy strips inbound private admin headers and injects authenticated Basic username plus private edge marker.
• Backend trusts operator identity only with the private marker and stores it on request-local state.
• Shared navigation includes Workers / Dispatch, Overview, Search, Supervisor, Logs, Config, and Secrets as pages are implemented.
• Overview uses independently degradable bounded health components.
• Search provides persistent discovery pause/resume and exact producer controls.
• Workers / Dispatch provides dispatch pause/resume, drain controls/progress, package compatibility, users, devices, assignments, and deferred requeue.
• Supervisor page exposes structured state and exact typed controls for all managed children. Logs page exposes bounded allowlisted tails only.
• Every mutation implemented so far receives trusted actor attribution and a durable accepted/terminal audit operation.

Important Web Decisions

• Admin pages are server-rendered and contain no application JavaScript.
• Every response remains no-store with restrictive security headers.
• Paths are relative so the random public Caddy admin prefix is preserved.
• Actor is the raw validated Basic-auth username; no actor form field exists.
• Exact form shapes reject missing, extra, and duplicate fields.
• POST mutations use canonical operation UUIDs embedded by the server.
• Plaintext device tokens appear only once in their direct POST response.
• Config/secrets content must never enter generic file-service scope.
• Worker status/upload/report/receipt endpoints remain independent of admin page health.

Completed Task 7.6

Task text:

Add config and plaintext secrets preview/save/apply pages with no-store rendering, revision/hash conflicts, and no value leakage outside the editor.

Implemented UI and service behavior

• GET /admin-internal/config
• GET /admin-internal/secrets
• POST /{config|secrets}/preview
• POST /{config|secrets}/save
• POST /{config|secrets}/apply
• POST /runtime/apply-both
• Editors load the candidate when present, otherwise the active document.
• Config editor never receives plaintext secrets.
• Secrets plaintext appears only inside the escaped no-store textarea.
• Preview uses the shared validator and candidate pairing rules.
• Config diff now redacts every changed string value.
• Secrets diff is aggregate-only and contains no pool, entry, username, token, hash fragment, or secret length.
• Save stages only a candidate and never activates an active file.
• Production currently has no host-agent apply provider. Apply returns 503 before operation creation. Host apply belongs to tasks 9.x.
• Save operations are durable and content-free:
  • runtime.config.save
  • runtime.secrets.save
• Accepted/terminal audit records contain only document name, hashes, byte counts, written flag, trusted actor, and fixed outcome/category.

Active files

• app/admin_api.py
• app/runtime_document_io.py
• app/runtime_document.py
• app/scanner_db.py
• app/worker_api.py
• tests/test_admin_api.py
• tests/test_runtime_document_io.py
• tests/test_operations_service.py

Completed replay and cancellation-safety fixes

1. Save replay after the candidate changed length

Resolved problem:

• _save_runtime_document_candidate_bytes() previews the current candidate and recomputes candidate_before_bytes before asking the database to replay the existing operation.
• After a successful candidate write, current byte length may differ from the originally accepted before length.
• Reusing the same operation UUID then conflicts even though it is the exact request replay.

Implemented fix:

• Look up runtime_operation(operation_id) before creating a new document operation.
• If it exists, validate actor, action, target kind/ref, submitted active hashes, selected candidate-before hash, proposed candidate-after raw hash and byte count against persisted expected identity.
• Persist and validate the counterpart candidate hash as part of the document operation expected identity. The accepted operation must bind all four CAS hashes, not only the selected candidate.
• For a running replay:
  • active config/secrets and counterpart candidate must still match accepted hashes;
  • if selected candidate still has the accepted before hash, retry the save;
  • if selected candidate already has the accepted after hash, complete the operation without writing again;
  • any other state is a 409 revision conflict.
• Do not recompute accepted before-byte count from post-save state.
• Extend the real ScannerDB operation test and make the admin fake compare the complete immutable identity, so changed-length replay is covered.

2. Preserve recovery identity on save errors

Resolved problem:

• A 503 runtime document completion is pending page renders pre-save state and generates a fresh operation UUID.
• Pressing Save from that page starts another operation instead of replaying the pending one.

Implemented fix:

• _render_runtime_document_page() should accept an optional operation UUID.
• Preview/error responses must keep the submitted operation UUID.
• After save failure/pending completion, reload editor state before rendering so hashes represent the physical candidate now on disk.
• The textarea may retain the submitted document only inside the editor.

3. Apply replay after host-side file changes

Resolved problem:

• request_runtime_apply() verifies current candidate/active filesystem state before checking for an already persisted operation.
• If the host agent changed active files and its response was lost, an exact POST retry can fail revision verification instead of recognizing the durable operation.

Implemented fix:

• Check runtime_operation(operation_id) first.
• Validate actor, action, target kind/ref and persisted expected hash identity against the submitted request.
• Terminal success returns without filesystem verification.
• Terminal failure returns 409.
• Requested/running exact replay redispatches the same operation ID/action without re-verifying post-apply filesystem state.
• A fresh operation still performs full candidate verification before durable operation creation and dispatch.
• Provider dispatch is expected to be idempotent by operation ID; task 9.x host agent will enforce persisted-operation/hash checks.

4. Cancellation traceback plaintext cleanup

Resolved problem:

• asyncio.CancelledError is a BaseException path and can leave raw URL-encoded body, decoded fields, or document_text in traceback locals.

Implemented fix:

• Add try/finally cleanup in _form_fields() for body, decoded text, parsed pairs, field mapping, and current chunk locals.
• Wrap the document branch of _dispatch() in try/finally and clear fields, hashes, editor, preview, operation ID, and document text locals.
• Add a cancellation test that injects cancellation after decoding and asserts the sentinel is absent from all admin_api traceback frame locals.

Task 7.6 Verification

• PyCompile passed for all changed production and test modules.
• Focused runtime-document/admin/operation suites: 96 passed, 1 skipped.
• Broader admin/runtime/operation/worker/edge/query suites: 179 passed, 1 skipped.
• Container unit selection: 37 modules, 810 test definitions, no runner skips.
• Desktop/mobile Chrome checks passed for Config and Secrets: no body overflow, bounded scrollable textarea, prefix-safe relative navigation, no script, local storage or service worker use, and no console errors.
• The secrets sentinel occurred exactly once in serialized HTML and nowhere outside the textarea.
• Strict OpenSpec validation passed.
• git diff --check passed. The checkout still has no useful tracked baseline.
• Live PostgreSQL was unavailable locally; SQLite service tests and SQL-shape checks remain the current database coverage.

Completed Task 7.7

Task 7.7 added durable operation-status and bounded paginated audit pages that remain backed by database state across runtime/admin restarts.

Implementation

• ScannerDB.runtime_audit_events(before_event_id=None, limit=50) returns a deterministic newest-first keyset page using event ID as the cursor.
• Cursor and page limits are strictly bounded. The query fetches at most one extra row to determine whether an older page exists.
• The audit read uses a SQLite transaction or PostgreSQL repeatable-read, read-only transaction and loads referenced operations in one bounded batch.
• Only operation-bound audit rows are exposed. Nullable unlinked storage events are excluded because they have no typed operation identity safe for the web console.
• Every returned event is revalidated against its normalized durable operation: actor, action, target, result/category, canonical expected/resulting identity, byte counts, parent hash metadata, timestamp, and event hash.
• Added explicit no-store server-rendered routes:
  • /admin-internal/operations
  • /admin-internal/operations/{canonical-operation-uuid}
  • /admin-internal/audit
  • /admin-internal/audit?before={positive-event-id}
• Unknown or noncanonical operation IDs return 404 without filesystem or agent probing. Duplicate, extra, or malformed query fields fail closed with 400.
• Operation status renders only normalized persisted safe fields and canonical expected/resulting identities.
• Audit renders only safe operation-bound event fields and hash identities.
• Apply POST redirects to the durable operation URL after successful dispatch.
• Relative URL roots preserve the random external admin prefix: list/audit pages use ., operation detail uses .., and apply redirects use ../operations/{operation_id}.
• Operations and Audit were added to shared navigation.

Task 7.7 tests and review

• Admin tests cover operation list/detail, audit pagination, random-prefix URL resolution, strict query shapes, unsupported methods, security headers, escaping, and absence of editor/Supervisor fixture secrets.
• SQLite service tests cover pagination without overlap, cursor exhaustion, operation completion identities, and close/reopen persistence.
• PostgreSQL integration coverage was added for pagination across ScannerDB reopen and rejection of audit UPDATE, DELETE, and TRUNCATE. All 59 bundled PostgreSQL tests were discovered but skipped locally because PostgreSQL is unavailable.
• An independent review found and then verified fixes for relative URL escape, nullable unlinked audit rows, and missing PostgreSQL coverage. Its final result reported no concrete findings.
• Focused admin/operation suites passed: 54 tests.
• Broader selected admin, runtime-document, operation/control/schema, worker API, edge deployment, and production-query checks passed; the unittest modules reported 183 tests with one expected Windows skip.
• Container unit selection passed: 37 modules, 814 test definitions, no runner skips.
• Desktop and 390px mobile browser checks passed for operation list, operation detail, and audit pages. The document had no horizontal overflow; wide tables scroll inside their bounded container. All links remained under the admin prefix. No scripts, local storage, service-worker controller, fixture-secret leakage, console warnings, or console errors were present.
• Strict OpenSpec validation and git diff --check passed.

Completed Task 7.8

Task 7.8 adds admin API and browser coverage for routes, methods, exact form shapes, actor spoofing, Origin/CSRF, stale forms, random-prefix navigation, CSP, and secret non-retention.

Production fixes already implemented

• _dispatch() rejects query fields before any state read or mutation. Only GET /admin-internal/audit accepts the exact optional before cursor; every other route and method requires an empty query.
• _AdminRoute(Route) upgrades Starlette method-only partial matches and calls the admin endpoint directly, so arbitrary methods such as PROPFIND receive the application's secured 405 response instead of an unprotected framework response.
• Supervisor controls were compacted without changing the typed backend: Dashboard and every managed source use collapsed <details> panels. Exact action routes, CSRF, operation UUIDs, allowlists, and no-JavaScript behavior remain unchanged. No shell, generic command/action field, or parser was added.

API and edge test work implemented

• tests/edge_e2e_client.py now submits canonical operation UUIDs for user creation, expects the current 303 redirect, and verifies PostgreSQL operation plus accepted/succeeded audit rows attribute the authenticated Basic user, not spoofed inbound operator headers.
• A table-driven contract test covers all 45 exact POST routes. Each route rejects the wrong method, incomplete fields, and attacker Origin with secure headers and no side effects.
• All six stale control forms return secured 409 without a transition.
• Config/secrets preview, save, individual apply, and apply-both now have direct valid-route coverage.
• Eighteen stale runtime-document hash cases cover every active, selected candidate, and counterpart candidate hash before mutation/operation creation.
• All rendered Search and Supervisor exact actions/settings have successful dispatch coverage; semantically invalid keychecks loop mode remains excluded.
• CSRF/Origin tests now cover missing/duplicate CSRF and duplicate/wrong Origin, proving only the valid request mutates.
• Static random-prefix crawl covers root, overview, Search, Supervisor, Logs, Config, Secrets, Operations, operation detail, and Audit. Every link/form remains under the simulated private prefix; every response has strict headers and no script/browser-storage API references.
• The fixture secret occurs exactly once inside the Secrets textarea and nowhere outside it or on other pages. Form and textarea autocomplete are off.
• tests/test_admin_browser.py starts a real local ASGI fixture behind a simulated random external prefix and drives Chromium through pinned Playwright (tests/requirements-browser.txt). Its dedicated lane fails rather than silently skipping when Playwright is absent.
• The browser test visits every navigation page, validates exact response security headers and inline-script CSP enforcement, rejects unexpected console errors, checks desktop/mobile body overflow, and proves every link, stylesheet, and form remains under the random external prefix.
• The browser test also proves the plaintext fixture secret exists only in the Secrets textarea, no local/session/Cache/IndexedDB/service-worker persistence exists, and an unsaved sentinel is absent after a network reload. Native browser back-forward memory is not treated as application persistence.

Final verification state

• PyCompile passed for app/admin_api.py, tests/test_admin_api.py, and tests/test_admin_browser.py plus tests/edge_e2e_client.py.
• tests/test_admin_api.py passed: 45 tests.
• tests/test_admin_browser.py passed in real local Chrome: 1 test.
• Broad selected admin/runtime-document/operation/control/schema/worker/edge/ query suites passed: 190 unittest tests, one expected Windows symlink skip.
• Container selection passed: 37 modules, 821 test definitions, no runner skips.
• Full local edge E2E passed with current rebuilt images. It proved durable operation/audit attribution to the authenticated Basic user, spoofed operator header stripping, Origin/CSRF behavior, worker-route independence, fail2ban behavior, cleanup, and unchanged foreign Docker state.
• Desktop and 390px mobile real-browser checks passed with no body overflow, scripts, unexpected console errors, or secret leakage. Wide Audit content scrolls only inside its bounded table container.
• An independent read-only review found no remaining concrete task 7.8 defect.
• Strict OpenSpec validation and git diff --check passed.

Completed Task 8.1

Task 8.1 defines the trusted logical-root policy and exact per-root permissions and limits without opening files or prematurely implementing traversal, file I/O, HTTP Files pages, or mutation audit behavior from tasks 8.2–8.4.

Policy and configuration

• Added app/managed_files.py with immutable typed root, permission, limit, and registry records plus the exact operation enum: list, read, create-replace, and delete.
• Root IDs are bounded lowercase logical names; absolute host/container paths exist only in trusted server configuration and are hidden from root repr.
• Every root must explicitly provide all four permissions and all six limits: relative-path bytes, component bytes, path depth, listing entries, listing bytes, and file bytes. Per-root values cannot exceed fixed hard caps.
• The predefined existing roots are runtime-logs, runtime-keychecks, and runtime-results at their fixed runtime directories. They are forced to list/read only and cannot be renamed or granted mutation permissions.
• runtime-results permits files up to 256 MiB so active and rotated scan_results.jsonl and found_secrets.jsonl generations remain downloadable. Only active names and exact six-digit generation names are visible; locks, databases, ledgers, scan errors, temporary/quarantine directories, malformed generations, and recovery artifacts are excluded. Keycheck and log roots retain the narrower 64 MiB per-file bound, and custom roots cannot opt into the larger result bound.
• Future writable roots are permitted only as immediate children of the isolated /data/managed-files directory. No writable root is enabled by default.
• This narrow allowlist excludes config/secrets, candidates, PostgreSQL, application code, sockets/control state, host-agent metadata, raw result bundles/spool, runtime state/queues/caches/work, and imported archives in both normal and parent-container directions. Projected results and keycheck output are available only through their fixed read-only roots.
• Root mappings are deterministically ordered, duplicate paths are rejected, nested fields are exact, booleans cannot pass integer limits, and errors do not echo submitted IDs or paths.
• Missing roots mean an empty registry. Existing explicit admin: null compatibility remains an empty disabled admin configuration, while a present managed_file_roots: null or any other malformed section fails closed.

Runtime wiring

• app/config.linux.yaml defines the read-only runtime-logs, runtime-keychecks, and runtime-results roots with bounded deployment defaults.
• app/runtime_document.py treats root IDs as a strict dynamic mapping, validates the policy even while admin is disabled, preserves omission compatibility, and includes the new dynamic shape in the pinned template schema hash.
• app/worker_api.py validates roots before DSN/secrets/assignment-builder work and passes the immutable registry into AdminService.
• AdminService accepts only a ManagedFileRootRegistry and otherwise defaults to an empty registry. No /files navigation or route exists yet; that belongs to task 8.4.
• Task 8.1 performs no open, stat, path resolution, directory creation, or network work. Descriptor opening and target traversal remain task 8.2.

Verification

• Added tests/test_managed_files.py; 8 policy/configuration tests passed.
• Runtime-document validation passed: 22 tests.
• Worker runtime wiring passed: 12 tests.
• Admin API passed: 46 tests.
• Runtime-document I/O regressions passed: 25 tests with one expected Windows symlink skip.
• Container selection passed: 38 modules, 832 test definitions, no runner skips.
• Strict OpenSpec validation and git diff --check passed.
• Independent read-only review found no remaining concrete correctness or security findings.

Completed Task 8.2

Task 8.2 adds the Linux descriptor-containment layer. It deliberately does not yet enumerate directory entries, return file bytes, mutate files, add HTTP routes, or write audit events; those belong to tasks 8.3 and 8.4.

Canonical path and descriptor model

• parse_managed_relative_path() requires an exact UTF-8 string and returns unchanged components only after enforcing the configured byte, component, and depth limits.
• Empty, absolute, drive-qualified (including nested drive components), backslash, NUL, repeated-separator, trailing-separator, dot, and dot-dot paths fail before target traversal.
• ManagedFileTraversal is Linux-only for nonempty registries and fails closed unless O_PATH, O_DIRECTORY, O_NOFOLLOW, O_CLOEXEC, descriptor-relative os.open, and the required nonblocking flags are available.
• Each configured absolute root is opened from / one component at a time with dir_fd, O_PATH | O_DIRECTORY | O_NOFOLLOW, and retained for the traversal lifetime. Symlinked root or parent components cannot be followed.
• Each operation duplicates the retained root under a lock before walking. This prevents concurrent close() and descriptor-number reuse from redirecting a request; already-started operations remain anchored after close.
• Intermediate client components use the same O_PATH | O_DIRECTORY | O_NOFOLLOW traversal. A renamed/replaced configured pathname does not change the retained root object.
• Listing opens only a verified directory descriptor. None is the internal root-list sentinel; an empty client path remains invalid.
• Read targets are first opened with O_PATH | O_NOFOLLOW, checked as a single-link regular inode, then reopened through their own /proc/self/fd descriptor and revalidated by type, link count, device, and inode before a readable descriptor is returned. Devices/FIFOs/sockets therefore are not opened for reading before type rejection.
• Opened targets are context-managed; partial root walks, operation walks, ordinary exceptions, and BaseException paths close every descriptor they unambiguously own. close() is lock-safe and idempotent; use-after-close fails closed.
• Errors expose only bounded categories such as invalid_path, unknown_root, operation_not_allowed, not_found, unsafe_target, root_unavailable, filesystem_unavailable, and closed. They never echo client paths, root paths, errno text, or filenames.

Adversarial coverage and packaging

• Traversal tests cover canonical ASCII/Unicode parsing, UTF-8 byte accounting, exact no-follow/dir-fd flags, partial-constructor and operation cancellation, close/open races, permissions, unknown roots, non-Linux behavior, and procfs failure classification.
• Real Linux tests cover nested list/read descriptor opens, root rename/path replacement anchoring, root/intermediate/final symlinks, hardlinks, directory-as-file, FIFO, Unix socket, component swap after intermediate open, idempotent close, and use-after-close.
• .dockerignore now admits only the new managed-file module/test explicitly, and container_unit.py selects the policy, mock traversal, and Linux kernel traversal classes.

Verification

• Native managed-file suite: 17 tests, 13 passed and 4 expected Linux-only skips.
• WSL Linux managed-file suite: 17/17 passed.
• Rebuilt read-only Linux test image: 17/17 managed-file tests passed under UID 10001 and the container audit fences.
• Related runtime-document, worker-runtime, admin, and runtime-document-I/O suites passed: 105 tests with one expected Windows symlink skip.
• Container selection passed: 38 modules, 841 test definitions, no runner skips.
• Strict OpenSpec validation and git diff --check passed.
• Final independent security review found no concrete task 8.2 findings.

Completed Task 8.3

Task 8.3 adds the bounded listing/download and durable file-mutation service on top of task 8.2 descriptor containment. It deliberately adds no HTTP route, operator form, durable operation row, or audit event; those belong to task 8.4.

Listing and download

• list_directory() lazily scans an already verified directory descriptor, counts every encountered entry against the configured work limit, bounds returned UTF-8 name bytes, and sorts accepted logical names deterministically.
• Listings expose only directories and bounded single-link regular files. Symlinks, hardlinks, special files, over-limit files, invalid names, and internal temporary names are omitted without revealing host paths.
• download_file() reuses the O_PATH//proc/self/fd inode-safe open, rejects oversized files before reading, reads at most the configured bound, computes SHA-256 while reading, and verifies stable inode/type/link/size/mtime/ctime metadata before returning content.
• Public result records expose logical names, file/directory kind, bounded byte counts, hashes, and content only on the direct download result. Download content and host descriptors remain hidden from repr.

Durable create/replace/delete

• Client relative components beginning with the reserved internal temporary prefix are rejected; the prefix is also hidden from listings.
• Create/replace accepts exact bytes up to the root file limit. None expected hash means create-only; a canonical lowercase SHA-256 means replace-only.
• Temporary files are created with random same-directory names using O_RDWR | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC, start private, are hardened to exact mode 0600, and must be regular, single-link, and owned by the effective runtime UID.
• Writes handle short writes/interruption, verify exact size and SHA-256, fsync the temporary inode, and retain its descriptor through publication.
• Create publishes atomically with Linux renameat2(RENAME_NOREPLACE), so an existing name is never overwritten. Replace revalidates the expected inode and hash before descriptor-relative os.replace; readers observe complete old or complete new files.
• Every mutation takes both an in-process lock and an advisory exclusive flock on the opened parent-directory inode. Separate traversal instances/processes using this service therefore implement one cooperative hash-CAS authority. Writable roots are dedicated immediate children of /data/managed-files; arbitrary SSH/root writers that ignore the lock are outside this contract.
• Namespace publication and unlink run inside a finally-protected parent directory fsync. A failure after visible mutation is reported only as bounded durability_uncertain/concurrent_change; the service never attempts an unsafe rollback over a later writer.
• Final published files are re-read and verified for expected hash/bytes, original staged inode, effective UID, exact 0600, regular type, and one link. The parent descriptor is then revalidated.
• Replace with identical expected/proposed identity is an idempotent no-write.
• Delete requires the exact current SHA-256, revalidates the named inode, unlinks descriptor-relatively, and fsyncs the parent directory.
• Prepublication failure and BaseException paths unlink and directory-fsync the known temporary name before potentially interrupted descriptor close. Explicit LOCK_UN prevents an interrupted parent close from retaining the mutation lock. Cancellation is never hidden by an ordinary conflict/error.
• Errors remain content/path/errno-free categories such as invalid_hash, invalid_content, limit_exceeded, hash_conflict, concurrent_change, durability_uncertain, and the task-8.2 access categories.

Verification

• Native Windows managed-file suite: 28 tests, 13 passed and 15 expected Linux-only skips.
• WSL Linux managed-file suite: 28/28 passed, including real renameat2, flock, inode/mode/owner checks, fsync, cleanup, and cross-instance concurrency.
• Rebuilt read-only Linux test image: 28/28 managed-file tests passed under UID 10001 with network disabled; 1048 unrelated tests were deselected.
• Cross-instance and independent-process replace/replace and replace/delete probes produced exactly one winner.
• Related runtime-document, worker-runtime, admin, runtime-document-I/O, and operation-service suites passed: 121 tests with one expected Windows symlink skip.
• Container selection passed: 38 modules, 852 test definitions, no runner skips.
• Final independent security review found no concrete task 8.3 findings.

Completed Task 8.4

Task 8.4 exposes the descriptor-safe managed-file service through exact server-rendered admin routes and binds every accepted mutation to a durable, content-free operation and audit lifecycle.

Files pages and forms

• Shared admin navigation now includes Files.
• GET /admin-internal/files renders either the logical-root index or a bounded root/nested listing selected by exact root_id and optional canonical relative_path query fields.
• GET /admin-internal/files/download requires exact logical root/path fields and returns application/octet-stream with no-store/security headers, exact length, SHA-256 ETag, and a logical-leaf attachment filename. Ordinary files retain the bounded in-memory response. An allowed result projection is first copied and hashed in 64 KiB chunks into an anonymous stable snapshot on the same data volume, then streamed from that snapshot with bounded memory. Source revision/size changes retry once and then fail closed; only one result snapshot may exist at a time, and response completion/failure releases it. No host path is passed to a response object.
• Async request cancellation retains cleanup ownership until a background snapshot build finishes, while the streaming response closes the snapshot in a response-level finally; disconnects and ASGI send failures therefore cannot strand the sole result-download permit.
• Exact POST routes are /files/create, /files/replace, and /files/delete. They accept only CSRF, operation UUID, logical root ID, canonical relative path, canonical expected hash where required, and canonical URL-safe Base64 content for create/replace.
• The existing bounded URL-encoded admin body is the explicit web-upload cap; the page displays that bound. No multipart parser, JavaScript, command field, actor field, absolute root, or generic action field was added.
• Root pages expose only logical IDs, permissions, configured limits, safe relative names/kinds/byte counts, and permission-appropriate forms. Private configured absolute roots and arbitrary file bytes never enter HTML.
• Query and form shapes reject duplicate, missing, extra, partial, empty, or noncanonical values before filesystem or database mutation.
• Managed-file errors map only to bounded HTTP outcomes (400/403/404/409/413/ 503) and never expose paths, errno text, content, or raw exceptions.

Durable mutation authority

• ScannerDB recognizes exact actions files.create, files.replace, and files.delete with target kind managed-file and logical root ID as the bounded target reference.
• Canonical expected identity contains only root ID, relative path, expected SHA-256, proposed SHA-256, and proposed byte count. Successful resulting identity contains only before/after hashes and counts, outcome, and written flag. No migration was required; migration count remains 29.
• create_runtime_managed_file_operation() atomically commits a running operation and accepted audit before physical mutation.
• complete_runtime_managed_file_operation() commits one terminal audit and safe resulting identity, or the fixed content-free failure category managed_file_mutation_failed.
• Uploaded bytes, Base64, CSRF/Origin/auth data, credentials, absolute roots, exception text, and rendered content cannot be accepted by the operation or audit APIs.
• List and download remain read-only and create no operation/audit rows.

Replay and execution serialization

• Each mutation holds a per-operation execution claim on a dedicated DB connection across terminal lookup, preflight, acceptance, physical CAS, and terminal completion.
• PostgreSQL uses a session advisory lock keyed by the canonical operation UUID. SQLite uses deterministic process-global lock stripes for local/test connections. Different operation IDs targeting one path remain serialized by task 8.3 parent-directory flock and hash CAS.
• Lock order is operation advisory claim, AdminService local lock, then traversal parent-directory lock. Release and connection close run through unconditional nested cleanup even under BaseException; cancellation remains primary and uploaded payload references are cleared.
• Exact terminal success validates immutable operation identity and returns without requiring current traversal/root availability. Terminal failure requires a new operation UUID.
• A fresh request validates current root permission/path and expected namespace state before accepted audit creation. Create requires absence; replace/delete require the exact expected hash. A fresh CAS loss always fails even if another writer produced matching bytes.
• A running exact replay may retry only from its accepted before state. It may complete without a second mutation only after observing the accepted desired after-state (or stable deletion), fsyncing the parent directory, and revalidating the same named inode/revision or absence after fsync.
• Matching replayed create/replace after-state must also be a private regular single-link file owned by the runtime UID with exact mode 0600.
• Content-bearing parser, dispatcher, service, and traversal frames clear or release URL-encoded, Base64, decoded payload, and memory-view locals on normal, error, and cancellation paths.

Traversal lifecycle

• Worker API lifespan constructs exactly one retained ManagedFileTraversal when admin is enabled and reuses it for all Files requests.
• Root-open failure safely degrades Files to 503 without taking down worker claim/status/upload/report routes and logs only a bounded category/type.
• Nested shutdown cleanup clears app state and closes the traversal exactly once even when assignment-reaper shutdown fails. Admin-disabled apps never open managed roots.

Verification

• Focused host suites passed: operation service 19/19, admin API 54/54, Worker API 35/35, and real-browser admin coverage 1/1.
• Managed-file tests passed 30/30 on WSL Linux; native Windows ran the same 30 with 17 expected Linux-only skips.
• Related runtime-document, runtime-document-I/O, worker-runtime, operation control/schema, edge-deployment, and production-query suites passed; the only skip was the expected Windows symlink case.
• Real Chromium covered Files navigation under the random prefix, security/CSP, no scripts/storage, mobile layout, and console cleanliness.
• Rebuilt read-only/no-network Linux test image passed all 46 selected managed-file/admin/operation/lifecycle tests under UID 10001; 1046 unrelated tests were deselected.
• Container selection passed: 38 modules, 868 test definitions, no runner skips.
• SQLite two-connection execution serialization and cancellation cleanup have dedicated regressions. A PostgreSQL two-session advisory-block/unlock/ session-close integration test is committed and discovered, but skipped locally because bundled PostgreSQL is unavailable.
• Strict OpenSpec validation and git diff --check passed.
• Final independent security review found no concrete task 8.4 findings.

Completed Task 8.5

Task 8.5 closes the managed-file service with adversarial traversal, link, special-file, limit, concurrency, durability, transport-encoding, and forbidden-root coverage.

Production hardening

• Nested listings validate the full logical child path, not only the leaf name, before exposing an entry. Children beyond the configured path-byte or depth bound remain hidden.
• Temporary-file cleanup fsyncs the parent directory even when the allocated temporary name is already absent, preserving durable cleanup evidence.
• Admin dispatch requires byte-valued ASGI raw_path to be the exact canonical ASCII encoding of the decoded fixed route. Missing or percent-encoded route aliases fail with a secured 404; query and form values still decode exactly once through their normal parsers.
• Download stability again requires an unchanged regular single-link inode, including device, inode, size, link count, mtime and ctime. On a concurrent atomic replacement, download performs at most one canonical reopen and strict reread, so callers receive one complete version rather than mixed bytes.

Adversarial coverage

• Policy tests explicitly exclude active secrets, candidate documents, PostgreSQL, host-agent metadata, Docker/runtime sockets, application code, raw bundles, spool/results/state/queues/cache/work, and imported archives.
• Portable tests prove FIFO, socket, character-device, block-device and directory modes are rejected before a readable /proc/self/fd reopen.
• Real Linux tests cover root and final symlink swaps, retained-descriptor anchoring, hardlinks, directories, FIFOs, hidden unsafe listing entries, and rejection by download/replace/delete.
• Exact and one-over tests cover total path bytes, component bytes, depth, listing entry/name-byte limits, nested logical paths, and file download/upload limits.
• Two traversal instances prove one winner for create/create and delete/delete; prior replace/replace and replace/delete tests remain. Reader replacement tests prove complete-version behavior, including a hostile in-place write, restored mtime, and atomic pathname replacement.
• Durability fault tests cover temporary inode fsync, already-absent temporary cleanup, cleanup unlink/fsync failures, and replace/delete parent-fsync uncertainty while asserting the resulting visible namespace.
• HTTP tests cover encoded and mixed-case traversal, backslash, NUL, drive prefixes, exact one-pass double decoding, encoded fixed-route aliases, forbidden logical roots, generic action/command/path/service field injection, and exact/over/streaming request-body limits before decode, traversal, or operation/audit creation.

Writer authority boundary

• Writable roots are dedicated managed-service roots. Cooperating app instances and processes serialize mutations with the parent-directory advisory flock and hash CAS.
• Arbitrary SSH/root writers that deliberately ignore that flock are outside this authority contract; POSIX has no atomic primitive for replace/unlink iff current content hash equals X against such writers.
• The service still detects bounded concurrent drift wherever possible and never follows a swapped symlink target.

Verification

• Managed-file tests passed 37/37 on WSL Linux; native Windows ran the same 37 with 23 expected Linux-only skips.
• Native scoped review reported 71 passed plus the expected Linux skips, and the admin API adversarial suite passed.
• Broad runtime-document, operation/control/schema, Worker API/runtime, production-query, edge-deployment, and real-browser suites passed; the only broad skip was the expected Windows symlink case.
• Container selection passed: 38 modules, 878 test definitions, no runner skips.
• Rebuilt truf-worker-test:task-8-5; a read-only, no-network Linux run under UID 10001 passed all 55 selected managed-file/admin/operation/lifecycle tests, including all 37 Linux managed-file tests; 1047 unrelated tests were deselected.
• Independent final security review found no concrete in-scope correctness or security findings.
• Strict OpenSpec validation and git diff --check passed before artifact closure.

Completed Task 9.4

Task 9.4 adds one fixed automatic rollback attempt, durable terminal evidence, and a global failed-hold fence without adding request-selected lifecycle or file surfaces.

Byte-identical rollback

• Stopped-runtime proofs are operation-bound, purpose-bound (forward or rollback), uniquely issued, and single-use. Forward proof cannot authorize restoration and rollback proof cannot authorize candidate publication.
• HostApplySession retains authoritative original snapshots separately from observed active/candidate state and classifies publication as original, partial, or candidate.
• Exact mixed old/candidate replay is recovery-only. It never reapplies the candidate deployment.
• restore_backups() accepts no paths or payloads. It rereads fixed root-owned operation backups, accepts active bytes only when exactly original or exact candidate, stages byte-identical originals, rechecks CAS, atomically restores, adopts fixed runtime ownership/mode, fsyncs, and verifies the complete original pair. Backups and candidates remain as evidence.
• Exact root-owned originals left by interruption between rollback rename and ownership adoption are recognized and safely adopted on replay. Any unknown active bytes fail closed without overwrite.
• Publication state becomes partial before the first restoration mutation, so a failed second restore cannot be misreported as a complete candidate state.

One-attempt recovery and failed hold

• Lifecycle execution retains a private mutable attempt before the first stop, including the original attested image/config identities and every observed stop/remove/recreate milestone.
• Any failure after lifecycle mutation quiesces only the exact fixed edge and runtime containers, restores selected backups, and recreates/verifies the original attested runtime and edge exactly once with the same strict aggregate health gates.
• Successful recovery emits rolled_back with both original active hashes and the bounded forward failure category. It never reports apply success.
• Recovery failure writes the global failed-hold marker before one containment stop, preserves operation/backups/candidates, emits failed_hold, and performs no third deployment attempt, restoration loop, or forced authority release.
• A global marker fences every different operation. The same operation may only finish a missing failed_hold phase/result; it skips document preparation and cannot resume forward or rollback lifecycle work.
• Pre-mutation validation/identity failure records failed without downtime or rollback.

Durable phase and result evidence

• New app/host_agent_state.py uses only fixed operation/result/hold paths under /var/lib/truf/host-agent, canonical bounded JSON, no-follow stable reads, root metadata checks, same-directory atomic replacement, file fsync, directory fsync, and exact-byte idempotent replay.
• Closed phases are prepared, forward_started, rollback_started, succeeded, failed, rolled_back, and failed_hold; transitions bind the exact operation UUID, action, four request hashes, publication state, bounded category/detail, and containment evidence.
• The terminal phase is persisted before the immutable seven-field ScannerDB- compatible result. Missing results are deterministically reconstructed from a terminal phase without lifecycle mutation.
• A write that renamed phase evidence but could not confirm parent durability is classified uncertain. Terminal uncertainty never rolls back or contains an already healthy deployment; nonterminal rollback uncertainty durably fences and contains instead of retrying.
• KeyboardInterrupt and SystemExit propagate directly before mutation. After mutation, the one rollback/failed-hold safety path runs first and the original cancellation is still propagated, including result-write and phase-rename uncertainty windows.

Verification

• Lifecycle tests passed 32 selected cases in the final Linux container and native Windows passed with four expected POSIX-only skips.
• Host apply passed 23/23 under WSL root; native passed 18 with five expected root/POSIX skips. Durable state passed 6/6, protocol passed 15/15.
• Broad operations, container-runtime, runtime-document/I/O, admin, Worker API, worker-runtime, and edge-deployment regressions passed.
• Container selection passed: 43 modules, 968 test definitions, no runner skips.
• Rebuilt truf-worker-test:task-9-4; a read-only, no-network container run as UID 10001 passed 76 selected host-agent tests with two expected distinct-root- ownership skips; 1119 unrelated tests were deselected.
• Independent security/correctness review completed repeated crash/cancellation review rounds and returned no findings after terminal ordering, early fencing, replay, uncertainty, and cancellation fixes.
• Strict OpenSpec validation and git diff --check passed before artifact closure.

Completed Task 9.1

Task 9.1 establishes the authenticated, closed privileged-agent request boundary without implementing file replacement, restart, health, or rollback.

Protocol and client

• app/host_agent_protocol.py defines a strict canonical UTF-8 JSON protocol framed by a four-byte big-endian payload length.
• Requests are bounded to 1024 payload bytes and contain exactly operation UUID, one of apply-config, apply-secrets, apply-both, or restart, and the four active/candidate hash fields used by ScannerDB expected identity.
• Canonical nonzero UUID, lowercase SHA-256 values, and the action-specific candidate-null/hash matrix are enforced. Duplicate, missing, extra, reordered, whitespace-altered, non-finite, or incorrectly typed fields fail closed.
• No path, service, unit, command, argv, environment, Docker/Compose argument, timeout, actor, content, or generic options field exists.
• Each connection carries exactly one request and one response. The client half-closes its write side; the server requires EOF, rejecting trailing or pipelined bytes. Absolute monotonic deadlines prevent trickle extension.
• app/host_agent_client.py has no configurable socket path and uses only /run/truf/host-agent.sock. It requires a root-owned socket pathname and kernel SO_PEERCRED proving the connected server UID is root; there is no retry loop.

Root agent boundary

• app/host_agent_server.py checks kernel peer credentials before reading any body and permits only the fixed runtime UID 10001. GID is intentionally not part of the identity contract.
• The server validates an inherited systemd socket as exact AF_UNIX, exact SOCK_STREAM, listening, fixed-path, socket-typed, and root-owned.
• deploy/host-agent/truf_host_agent.py requires root, no command-line arguments, and exactly one systemd-activated listener.
• The production task-9.1 handler always returns unavailable. It never claims accepted before task 9.2 has verified the persisted operation and durably assumed ownership.
• Systemd units, host path installation and socket permissions remain task 9.5; the agent executable is ready for that fixed activation contract but no unit was added early.

Runtime integration boundary

• Admin apply provider calls now pass the persisted expected identity fields: active config/secrets hashes and candidate config/secrets hashes, in addition to operation ID and action.
• Production Worker API still installs no apply provider. An absent agent therefore preserves the existing 503 before operation creation. Wiring is deferred until persisted verification/execution exists; no request can currently trigger privileged lifecycle work.

Verification

• Portable protocol/client/server tests passed 15/15, including exact schemas, candidate matrix, framing bounds, duplicate/extra fields, absolute trickle deadlines, response pipelining, cancellation cleanup, unauthorized-before- read ordering, listener type/listening checks, and inherited-FD cleanup.
• Real Linux tests passed 2/2 under fixed UID 10001 using kernel SO_PEERCRED.
• Rebuilt truf-worker-test:task-9-1; a read-only, no-network container run as UID 10001 passed all 17 selected host-agent tests, including the real Linux peer path; 1102 unrelated tests were deselected.
• Broad admin 57, operations 19, Worker API 35, worker-runtime 12, edge-static, and provider-hash-binding regressions passed.
• Container selection passed: 40 modules, 895 test definitions, no runner skips.
• Independent security review found no concrete task-9.1 correctness or security findings after exact SO_TYPE/SO_ACCEPTCONN hardening.
• Strict OpenSpec validation and git diff --check passed before artifact closure.

Completed Task 9.2

Task 9.2 adds a fail-closed execution core for one persisted runtime apply under fixed host paths. It remains deliberately unwired until task 9.3 can stop the runtime before publication and task 9.5 installs protected DB/path authority.

Persisted operation claim

• ScannerDB.claim_runtime_operation_execution(...) atomically binds the canonical operation UUID, action, runtime-deployment target, and exact four active/candidate hashes to the requested-to-running transition.
• The method uses the existing SQLite write transaction or PostgreSQL control and operation row locks. The host session separately requires PostgreSQL; SQLite support exists only for deterministic database tests.
• Requested/pending operations can be claimed once. An exact running/running operation replays; unknown, mismatched, terminal, or incoherent rows fail.
• Claim requires exactly one canonical accepted audit for the operation and verifies its actor/action/target/identity/time, global predecessor link, the predecessor's own payload hash, and the accepted event hash before transition.
• Schema-valid unlinked audit predecessors remain supported. No schema migration or new authority table was added.

Fixed host apply session

• app/host_agent_apply.py defines only fixed constants: active config/secrets, candidate config/secrets, root apply lock, operation-ID backup directory, and the trusted packaged config template. Socket requests cannot provide paths, commands, services, environment, or options.
• HostApplySession takes the root-private nonblocking singleton filesystem lock before claim and keeps it through validation, backup, publication, and later task-9.3 lifecycle insertion points. The lock survives PostgreSQL shutdown.
• The session requires PostgreSQL authority, claims the exact persisted request, stable-reads regular single-link active/candidate/template files with nofollow, enforces fixed owner/group/mode and raw hash CAS, selects the action-specific effective pair, and calls the shared strict runtime-document validator.
• Package capability evidence is host-supplied trusted input, never a socket field. Production remains unwired until task 9.5 provides fixed protected package/DB/path mappings.

Backup, publication, and replay

• Selected active bytes are backed up byte-for-byte under the canonical operation UUID using exclusive creation, root-only metadata, file fsync, reread/hash verification, and parent-directory fsync. Existing backups are accepted only when their bytes and identity exactly match.
• All selected replacements are staged and fsynced before publication. Stages remain root-owned until same-directory atomic rename, then receive fixed runtime ownership/mode and are fsynced again; the runtime UID cannot mutate a predictable stage before publication.
• Active and candidate snapshots are rechecked immediately before publication. On POSIX, the displaced active inode is retained and reread after rename, so a late in-place writer becomes a conservative partial result rather than silent data loss.
• Candidates and backups remain as evidence. A stale fixed stage from the same operation is durably removed before retry.
• Exact running replay distinguishes not-published, completely-published, and partially-published state. Complete replay requires exact backups, candidates, and active bytes, recovers a root-owned post-rename/pre-adoption file, and revalidates again at replace(). Partial replay requires valid backups and fails closed for task 9.4 recovery.
• Each config/secrets rename is atomic; the pair is not a filesystem transaction. A failure after any publication is categorized partial; task 9.2 does not invent rollback or failed-hold behavior ahead of task 9.4.
• restart still claims and validates the active pair but creates no backup and publishes no file.

Production boundary

• deploy/host-agent/truf_host_agent.py was not wired to this core and still returns unavailable. Current deployment has no protected host PostgreSQL endpoint, fixed mounts/permissions, or safe stop/recreate sequence.
• Task 9.3 must guarantee stopped-runtime publication and perform fixed recreation/health checks. Task 9.4 owns rollback. Task 9.5 owns installation, host DB/path authority, systemd units, and permissions. Task 9.6 owns the full crash/fault matrix.

Verification

• Host apply tests passed 17/17 under WSL root with distinct root/runtime ownership; native Windows passed with four expected POSIX-only skips.
• Operation tests passed 25/25, including exact claim/replay, mismatch, accepted/predecessor corruption, valid unlinked predecessor, missing, and terminal cases.
• Broad protocol 15, runtime-document 22, runtime-document I/O 25, admin 57, Worker API 35, worker-runtime 12, and edge deployment suites passed.
• Container selection passed: 41 modules, 918 test definitions, no runner skips.
• Rebuilt truf-worker-test:task-9-2; a read-only, no-network container run as UID 10001 passed 33 selected host-agent tests with one expected root-only ownership-recovery skip; 1108 unrelated tests were deselected.
• Independent security/correctness review found no findings after root-owned staging, displaced-inode detection, durable replay, and predecessor-hash fixes.
• Strict OpenSpec validation and git diff --check passed before artifact closure.

Completed Task 9.3

Task 9.3 adds a fixed, fail-closed runtime/edge lifecycle and bounded aggregate health verification. It remains deliberately unwired until rollback and installation authority are complete.

Fixed lifecycle authority

• app/host_agent_lifecycle.py accepts no production paths, services, commands, environment, or timeouts. It uses only /usr/bin/docker, project truf-docker, the two fixed Compose files, fixed edge environment, and the runtime and edge services.
• Preflight validates the fixed Compose model, captures immutable runtime/edge image IDs, resolves exactly one container per service, and inspects only a bounded selected projection. It never reads container environment, full state, health logs, or host mount source paths.
• Attestation binds IDs, images, users, entrypoint/command, runtime healthcheck, Compose labels/config hash, capabilities/security options, read-only and privileged state, namespaces, restart policy, resource limits, stop policy, log driver, destination-only mounts, ports/tmpfs, and absence of device access.
• Edge stops first with a fixed 30-second grace; runtime then receives the fixed 600-second coordinated stop. Each captured container must prove exited, PID 0, exit 0, non-OOM, non-restarting/non-dead, and unchanged restart count.
• Immediately before publication, lifecycle re-resolves both service IDs, revalidates their stopped state and immutable image tags, and issues a private operation-bound stopped proof. HostApplySession.replace() rejects absent, forged, or wrong-operation proof.
• The executor removes only the stopped edge and runtime containers, recreates runtime with --no-deps --no-build --pull never --force-recreate, requires a new exact identity, waits under one bounded deadline, and recreates edge only after strict runtime health. Edge likewise requires a new exact identity, fixed network namespace, successful fixed Caddy validation, and bounded stable running observation.
• No down, kill, build/pull, volume/orphan, provision, systemd, fail2ban, request-selected action, automatic recreation retry, rollback, or terminal result reconciliation was added.

Document and execution ordering

• execute_fixed_forward() keeps the task-9.2 singleton lock across backup, deployment preflight, pre-stop CAS, stop proof, publication, recreation, and health verification.
• HostApplySession.revalidate_for_stop() rechecks active/candidate identities after backup/preflight and before downtime; replace() repeats validation after clean stop and immediately before publication.
• Restart follows the same fixed stop/recreate/health sequence but does not back up or publish documents. Complete publication replay recreates and verifies once rather than treating file presence as runtime health evidence.

Bounded aggregate health

• container_runtime.py health --require-worker-api preserves all existing authenticated Supervisor ACTIVE, PostgreSQL READY/schema/cutover/PG16/storage, and instance-bound ingester/projector lease checks, while requiring Worker API to be explicitly enabled and running.
• Strict Worker API health sends a fixed non-secret invalid Bearer credential to loopback /api/v1/worker/claim and requires the exact bounded 401 Bearer response. Authentication performs the normal ScannerDB lookup before rejection, proving the API-to-PostgreSQL path without a real credential or mutation.
• Ordinary Compose health remains unchanged; the strict flag is valid only for health/status. Production cannot pass strict mode until task 9.5 installs the intended enabled Worker API configuration.

Command and deadline hardening

• Docker subprocesses use a fixed minimal environment, no shell, closed stdin, suppressed stderr, a new process session, bounded 16 KiB stdout, and monotonic phase deadlines.
• Timeout, output overflow, reader failure, cancellation, and a descendant that retains stdout terminate only the owned process group with bounded waits. Successful commands never signal a stale process-group ID, and the raw output descriptor has single-owner close semantics.
• Stop/recreate/publication commands run at most once. Only bounded state/health observation polls.

Verification

• Lifecycle tests passed 18/18 on WSL, including real timeout, output-overflow, retained-descendant, and successful-process cleanup behavior; native Windows had four expected POSIX-only skips.
• Host apply tests passed 18/18 on WSL and protocol tests passed 15/15. The full container-runtime suite passed with strict Worker API positive and hostile response coverage.
• Broad operations 25, admin 57, Worker API 35, worker-runtime 12, runtime-document 22, runtime-document I/O 25, and edge deployment suites passed.
• Container selection passed: 42 modules, 943 test definitions, no runner skips.
• Rebuilt truf-worker-test:task-9-3; a read-only, no-network container run as UID 10001 passed 52 selected protocol/apply/lifecycle tests with one expected root-ownership skip; 1119 unrelated tests were deselected.
• The selected real Docker inspect format was exercised locally and parsed all 49 expected keys; a legacy container with an extra bind/import command was correctly rejected.
• Independent security/correctness review found no findings after stopped-proof, pre-stop CAS, attestation, Worker API DB-path, deadline, process-group, and FD ownership fixes.
• Strict OpenSpec validation and git diff --check passed before artifact closure.

Known Residual Boundaries

• Apply cannot execute in production until crash verification from task 9.6 is implemented. Returning 503 before operation creation remains correct when the fixed host-agent socket or result authority is unavailable.
• A process crash exactly after an external side effect and before terminal DB persistence is a distributed boundary. Stable operation IDs and idempotent replay are the intended mitigation; do not add a general coordinator here.
• Worker API self-stop/restart can terminate the admin request before terminal operation completion. Full external lifecycle reconciliation belongs to the host operation architecture, not task 7.5/7.6.
• Managed log pages display bounded child output as written. Do not add a post-hoc redaction pipeline without user approval.
• Candidate and active file handoff across privileged apply will be revalidated by the host agent; task 7.6 must not implement active replacement itself.

Known Environment/Test Issues Unrelated to Current Work

• Several broad Supervisor safety tests fail in this checkout because external runtime authority fixture files such as runtime/check-openrouter-keys.ps1 are absent or legacy fixture config is incomplete. Focused clean runs exclude only the documented affected classes; do not weaken production code for them.
• Bundled PostgreSQL integration tests skip when initdb/test DSN is not available.
• Windows symlink tests may skip when the process lacks symlink privilege.
• The worktree has no useful commit baseline and contains many pre-existing untracked/dirty files. Never perform broad resets or cleanup.
• Caddy 2.10.2 does not implement SIGUSR1 reload while production fail2ban documentation assumes it. Edge E2E uses an admin-enabled test reload path. This predates task 7.1 and needs a separate Caddy-upgrade/reload decision.
• Post-fail2ban-reload generic 404/401 responses in edge E2E do not reliably retain global security headers. Authenticated admin pages and mutations still enforce strict headers; the harness isolates the unrelated generic response behavior.

Important Successful End-to-End Evidence

• Packaged Windows/Linux protocol-2 E2E passed with GitLab plus synthetic DockerHub/HuggingFace claim-to-ingestion.
• Full container E2E passed after strict runtime-document integration.
• Real edge E2E passed twice after rebuilding edge, fail2ban and runtime images; final safe result proved authenticated Basic username attribution, private marker acceptance, worker header stripping, worker availability, and fail2ban behavior.
• Tasks 7.2–7.8 received desktop/mobile browser checks with real Starlette routes and trusted headers; task 7.8 also has committed real-Chromium coverage.

Task 9.5 Completion

• Added the fixed systemd socket/service, tmpfiles layout, root-only installer, installed /usr/lib/truf-host-agent entrypoints, graceful host-agent shutdown, PostgreSQL peer authority, async operation dispatch, and durable result reconciliation.
• Runtime Compose now uses exact long-form mounts with host-path creation disabled. Host config and secrets use the read-only /etc/truf/runtime to /data/config authority, while immutable package manifests use the separate root-owned /etc/truf/worker-packages to /data/worker-packages authority. Candidates, result handoff, host-agent socket, and PostgreSQL socket mounts retain their fixed access. Runtime-generated initialization state, lock, and PostgreSQL password remain in the private writable /data volume.
• Lifecycle inspection validates exact bind sources and the named data volume. The installer parses bounded rendered Compose JSON and independently requires the same source/target/read-only/create-host-path projection.
• Package manifests are stable-read as root-owned 0644 files beneath a fixed descriptor-walked root. Every descendant directory is root-owned and not group/other writable; host and container readers consume byte-identical files.
• The installer validates the full root-owned application tree, installed bytes, executables, Docker and agent sockets, units, tmpfiles, directories, and file modes. Repeat install stops active socket/service units before replacement, reloads systemd, enables the socket, and explicitly restarts it. Existing active config and secrets are never overwritten.

Task 9.5 Verification

• Native focused runtime-document/security/host deployment suites passed 67 tests with seven platform skips; the wider host deployment/lifecycle focused suites passed independently.
• WSL root stdlib suites passed 109/109. systemd-analyze verify accepted both units; warnings were limited to Windows-mounted source-file permissions.
• Canonical combined base+edge Compose rendered JSON passed the exact installer projection validator. Base Compose validation also passed under WSL Docker.
• The deny-by-default test image rebuilt successfully. Its UID 10001, read-only, no-network run passed all task-9.5 host-agent/deployment selections. Four of five initially failed broad tests passed alone after packaging docker/verify.py; the remaining deep-JSON hostile-input assertion belongs to task 9.6.
• Container selection is AST-clean with 47 modules and 994 test definitions.
• Three independent review passes found no remaining task-9.5 blocker after application-tree, mount-source, cleanup, manifest-root, Compose-projection, and repeat-install fixes.

Task 9.6 Completion

• Host-result JSON rejects nesting deeper than 64 before parsing, using an iterative string-aware scanner so hostile depth is deterministic across Python versions without recursive traversal.
• Candidate validation after a durable database claim now terminalizes as an exact failed/validation_failed result before acceptance. A failed result publication is replayable, and non-original prepared state is never rewritten.
• Crash-boundary coverage now proves interrupted combined-backup replay, retained rollback evidence, restart-specific rollback, and durable rollback failure.
• Protocol coverage rejects deeply nested unknown request fields at the real server boundary without invoking the privileged handler.

Task 9.6 Verification

• Native task-matrix suites passed 106 tests with nine platform skips; the WSL root stdlib matrix also exited successfully.
• Targeted UID 10001/Python 3.12 container tests passed 102 tests with two expected root-identity skips.
• The full hardened, read-only, no-network container suite passed 1219 tests with ten platform skips and one deprecation warning.
• Container selection is AST-clean with 47 modules and 1000 test definitions.
• Strict OpenSpec validation and diff checks passed. A final independent audit found no remaining task-9.6 blocker.

Production Completion Status

• The authorized production host now uses the exact root-selected shared-host-edge-v1 profile. Existing host Caddy remains the sole owner of ports 80/443, the managed Truf edge binds only 127.0.0.1:18766, and the host agent has no lifecycle authority over host Caddy, X-UI, or another unrelated service.
• Task 10.4 production execution is complete: protocol-2 package, runtime, admin, edge, and agent checks passed; one reconciled restart succeeded; one health-triggered automatic rollback restored the original documents and reconciled as rolled_back; only then were production controls reopened.

Task 10.1 Completion

• The production server image and lifecycle authority are remote-only and Git-only; TruffleHog remains confined to worker and test images.
• Explicit code authority covers the runtime-document, managed-file, and host-agent modules plus the core-profile launcher.
• Existing marked volumes gain only the fixed private /data/managed-files directory; all other incomplete or unsafe layouts still fail closed.
• Production runbooks now require the fixed host-agent installer, protected edge environment, active documents, package manifests, socket, and combined runtime/edge Compose deployment in the correct order.
• Container selection includes the operations, discovery-only, core-profile, multisource, and direct-source fixtures. The edge fixture advertises the exact GitLab, DockerHub, and HuggingFace profile and real-Caddy coverage traverses every new admin route plus operation detail.

Task 10.1 Verification

• Focused native coverage passed 401 tests with eleven platform skips; the final edge deployment suite passed 32 tests with one skip.
• Container selection is AST-clean with 54 modules and 1058 definitions.
• A hardened container run passed 1282 tests before fixture-only corrections; all five corrected image/static/real-CLI cases then passed in the rebuilt image.
• A built production runtime image was verified to contain executable Git and no /usr/local/bin/trufflehog.
• Two independent reviews found no remaining task-10.1 blocker. Strict OpenSpec validation and diff checks passed.

Task 10.2 Completion

• The offline rollback test removes the protocol-2 worker and operations authorities, applies the real stopped-runtime migration, and confirms all additive tables are restored.
• A live remote reservation and pre-commit bundle prevent drain completion; after both resolve, the authoritative state advances to drained with no blockers.
• A raw previous-image enqueue contract then opens the same database and writes using only legacy columns. The new control state, operation/audit evidence, and protocol-2 tables remain intact.
• The complete operations-control suite passed 13 tests on Windows and 13 under WSL. Container selection is AST-clean with 54 modules and 1059 definitions; strict OpenSpec validation and diff checks passed.

Task 10.3 Completion

• Focused unit, browser, packaged-worker, edge, PostgreSQL, and formal verification gates completed without a remaining failure.
• The PostgreSQL fixture now supports an explicitly selected POSIX PostgreSQL binary directory while preserving its existing bundled-Windows default.
• Real PostgreSQL execution exposed and closed one exact local-admission recovery identity omission plus stale protocol-2 fixture limits and bundle path evidence.

Task 10.3 Verification

• The real PostgreSQL 16 integration suite passed all 60 tests as UID 10001 in a read-only, no-network, capability-free container. Native Windows collected the same 60 tests and skipped them because the optional bundled PostgreSQL is absent.
• Packaged Windows and Linux worker verification passed after rebuilding the protocol-2 artifacts. The real edge E2E passed with 28 baseline responses, the exact core source profile, authenticated Caddy routes, and fail2ban restart, unban, and expiry evidence.
• Browser coverage passed. The focused task-10.1 matrix passed 401 tests with eleven platform skips, and the operations-control suite passed on Windows and WSL.
• Container selection is AST-clean with 54 modules and 1059 definitions. Strict OpenSpec validation and diff checks passed.

Task 10.4 Completion

• The production cutover used shared-host-edge-v1 while both explicit gates were paused and drain was authoritative with zero blockers. Root-owned profile, package, Compose projection, loopback edge, runtime, admin, agent, Caddy, and X-UI service checks passed without granting the Truf lifecycle authority any control over shared-host services.
• Reconciled restart operation 08dc7e8b-551f-5c64-b803-de905f040420 completed succeeded/succeeded with no safe failure category, healthy runtime and edge, and no failed hold.
• Reconciled rollback operation 6d09ea58-2c48-5366-b5ee-951c341fcd6f deliberately applied a valid candidate with Worker API disabled. Strict forward health classified the failure as health_check_failed; the agent restored the byte-identical original config and secrets exactly once and completed rolled_back/rolled_back with no failed hold.
• Audited, revision-checked operations then canceled drain, resumed discovery, and resumed dispatch in that order. Controls advanced from revision 19 to 22 and are now discovery open, dispatch open, and drain normal.
• After reopening, the active production worker contacted the server and received new DockerHub work. Lifecycle preflight, strict runtime health, and host-agent/Caddy/X-UI service checks remained healthy.

Production Architecture Corrections

• Real candidate validation found that private writable runtime configuration could not also be the root-owned immutable package authority. Worker package manifests now use the separate read-only /data/worker-packages authority, while initialization, lock, and PostgreSQL password state use private files directly beneath /data.
• PostgreSQL maintenance and offline migration use the discovery-only server authority profile and therefore do not require the worker-only TruffleHog executable. The offline cutover reconciled two stale leases, migrated the production database, and established audited discovery and dispatch pauses.
• Production validation also corrected Worker API HTTP concurrency from one to eight while retaining the real assignment cap of one.
• DockerHub discovery now configures account metadata without creating scanner Docker directories or requiring scanner runtime initialization. Scanner execution keeps the original full-runtime guard and Docker config behavior.

Task 10.4 Production Evidence

• The active thin-v6 runtime image is sha256:dd55500ee2f947c0088a57061d1689025653fc0b08ed7fb22baea6816fb93270 under tag truf-local:runtime; the managed edge image is sha256:6ec35cae4f4cf4bcd2bb1fe427b9cd2f16f8b5c2577442eae0941d806d362529 under tag truf-local:edge.
• Canonical health reports Supervisor ACTIVE, PostgreSQL READY, and healthy DockerHub, GitLab, HuggingFace, janitor, projector, ingester, and Worker API processes. Lifecycle attestation confirms host-network runtime with no published ports and a loopback-only edge.
• Root-owned protocol-2 Windows and Linux manifests advertise only GitLab, DockerHub, and HuggingFace. Candidate initialization passed against the real production volume before cutover.
• Public checks after rollback returned 401 for an invalid Worker API token, 401 for the unauthenticated protected admin route, and 404 for an unrelated Truf path. Host Caddy, X-UI, and the Truf host agent remained active.
• Shared-host restart validation exposed a real Docker Compose identity nuance: network_mode: service:runtime changes the edge Compose config hash whenever runtime is recreated. Lifecycle now attests all fixed edge metadata before observing and pinning each forward or rollback edge hash; it never weakens the fixed topology checks.
• Strict runtime-health command failures are normalized to lifecycle health failures. This made the intentional rollback drill reconcile with the exact health_check_failed safe category rather than generic apply_failed.
• Root-only evidence for two diagnostic failed holds remains preserved under /opt/truf-remote-server/staging/failed-hold-recovery-20260922 and /opt/truf-remote-server/staging/failed-hold-recovery-v2-20260922. Both holds were cleared only after strict health, identity, and evidence checks.
• The exact pre-v4 unit, previous runtime image and configuration, verified custom-format PostgreSQL backup, failed-hold evidence, and reconciled operation results remain available for rollback and audit. No X-UI or unrelated host service was modified.

Task 10.4 Verification

• The final focused host-agent matrix passed 107 tests with fourteen platform skips. Edge deployment and container-runtime pytest coverage passed 221 tests with one platform skip.
• Runtime-document coverage passed 22 tests and worker-package coverage passed 15 tests. Changed lifecycle and drill scripts compile successfully.
• Both standalone base+edge and shared-host base+edge Compose projections passed config --quiet with non-secret synthetic validation values; required edge variables still fail closed when omitted.
• Strict OpenSpec validation and diff checks passed after production evidence was reconciled and controls were reopened.

Task 10.5 Production Evidence

• A single immutable DockerHub digest was processed independently by the packaged Windows worker and the hardened WSL worker with assignment caps and parallelism fixed at one. Both reservations produced accepted protocol-2 bundles and acknowledged ingestion/projection records.
• A completed assignment replay returned the original accepted receipt without rescanning. A separate claim-only assignment expired at its real wall-clock deadline, refunded queue capacity and attempts, and replayed the same durable expiry receipt over the public Worker API.
• An audited drain advanced from draining to drained with zero live remote assignments and zero pre-commit bundles. Audited cancel restored normal while both explicit gates remained paused.
• Temporary canary devices were revoked, their token file was overwritten and removed, the normal 7200-second assignment TTL was restored, and no unresolved canary assignment remained.

Task 10.6 Production Evidence

• Only after the DockerHub canary gates passed, audited operations enabled production discovery and dispatch and created one production WSL worker user and device with an active assignment cap of one.
• Real production cycles completed successfully for GitLab with gl_1, public HuggingFace with hf_1, and DockerHub with its real account pool. The final DockerHub cycle reported eleven available accounts and exited with code zero.
• The active server and both protocol-2 compatibility profiles contain exactly gitlab, dockerhub, and huggingface; GitHub remains excluded.
• Current controls are revision 22 with discovery and dispatch open and drain state normal. The production WSL worker is running detached with parallelism one; its audited user is enabled, its device is not revoked, and it contacted the server after the task-10.4 rollback gates reopened.
• Rollback evidence retains the previous image, exact units, previous active configuration, and verified database dump. The v4 unit and image identities were checked before and after activation.

Managed Files Production Extension (2026-09-22)

• Production Files now exposes exactly three configured logical roots: runtime-logs, runtime-keychecks, and runtime-results. All are list/read only. Logs and keychecks retain the 64 MiB per-file limit; only the fixed result root permits 256 MiB files.
• The result root exposes only active scan_results.jsonl and found_secrets.jsonl projections and their exact six-digit generations. Internal locks, databases, ledgers, scan errors, malformed generations, recovery files, and temporary/quarantine directories remain unavailable.
• The deployment paused discovery and dispatch and allowed the existing remote DockerHub assignment to resolve naturally. Drain reached drained with zero blockers before any runtime or configuration activation; no assignment was cancelled.
• Runtime image sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf was activated with a fail-safe Compose transition. The preceding image is retained as truf-local:runtime-pre-managed-files-v1.
• Official restart operation 1978f770-c4a5-537d-b474-c3c9428cb79c completed succeeded/succeeded, reconciled without a safe category or failed hold, and proved the full fixed lifecycle on the new immutable image before the root configuration changed.
• Official Preview/Save operation 87b78ff6-d8eb-5fb4-a897-83b52c3697e5 changed only the managed-root mapping. Apply operation 7a572971-08d2-5e51-be78-4fb7039a93d4 completed succeeded/succeeded, reconciled without rollback or failed hold. The active configuration identity is e48797689ab0ee7328d21cf5c23d0ffedb979dba492bf49d1da3afe4b075cf5b.
• Live runtime traversal verified all three exact policies, an empty keycheck listing, two allowlisted active result projections, stable snapshot length/hash, one-snapshot serialization, permit reuse, safe rejection of an internal result artifact, and denied result mutation.
• Live Worker API HTML rendered all three root IDs and the read-only keycheck and result pages. A permitted 146366-byte projection download matched its exact source SHA-256 and length through the streaming route; an internal result name returned 404. No file contents were emitted during validation.
• Final audited controls are revision 56 with discovery and dispatch open and drain normal. Canonical Worker API plus discovery-producer strict health and lifecycle preflight pass; failed hold is absent; host agent, host Caddy, and X-UI are active. Public checks remain 401 for invalid Worker authentication, 401 for unauthenticated admin access, and 404 for an unrelated path.