sashatrask/truf-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Initial server source import

Browse Source
This commit is contained in:
sashatrask
2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/build-dependencies/README.md
+273
View File
@@ -0,0 +1,273 @@
# Runtime Dependency Build
This directory records the public build inputs and pip-tools generator. It is not
an application configuration directory and must never contain credentials.
## Pins
| Input | Pin |
| --- | --- |
| Python image | `python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254` |
| Python version | `3.12.14` |
| Linux amd64 manifest | `sha256:9c47360a2a0355e2da18516d0b1c2126ec22c195d2185e97347c9d98398c5bef` |
| Linux arm64/v8 manifest | `sha256:d04f49f5882f49a3b91f874e75e19f0c265f7222da8659741a9d7eab148f22a9` |
| Debian and Debian security snapshots | `20260914T000000Z` |
| Git and git-man | `1:2.39.5-0+deb12u3` |
| tini | `0.19.0-1+b3` |
| CA certificates (already present in the pinned base) | `20250419~deb12u1` |
| PGDG server, client, libpq | `16.15-1.pgdg12+2` |
| PGDG common and client-common | `293.pgdg12+1` |
| PGDG signing-key fingerprint | `B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8` |
| PGDG signing-key SHA-256 | `0144068502a1eddd2a0280ede10ef607d1ec592ce819940991203941564e8e76` |
| TruffleHog | `3.97.4` |
| TruffleHog Linux amd64 archive SHA-256 | `dc24007c2f233bd61c05beabeb44aa27ea9b43288166279209abe0458c5ce76b` |
| TruffleHog Linux amd64 archive size | `34970205` bytes |
| TruffleHog Linux arm64 archive SHA-256 | `7e65e771d2a247964056aa5edba0f8ae3945895e5dce867fe0ffbc7b0128239a` |
| TruffleHog Windows amd64 archive SHA-256 | `6ce9a957ac62bfb19463048333d9e8481327dbbf5bdc0c43f5ab5327b9631fb9` |
| Windows embeddable Python | `3.12.10`, SHA-256 `4acbed6dd1c744b0376e3b1cf57ce906f9dc9e95e68824584c8099a63025a3c3` |
| Windows MinGit | `2.47.1.windows.1`, SHA-256 `50b04b55425b5c465d076cdb184f63a0cd0f86f6ec8bb4d5860114a713d2c29a` |
| pip-tools | `7.6.1` |
| Generator pip | `26.2.1` |
| pytest | `8.4.2` |
| httpx (test target only) | `0.28.1` |
| zstandard | `0.23.0` |
| pandas | `3.0.5` |
| plotly | `7.0.0` |
| streamlit | `1.63.0` |
| psycopg and psycopg-binary | `3.3.5` |
| boto3 and botocore | `1.43.94` |
TruffleHog's expected hashes were checked against the public release's
[`trufflehog_3.97.4_checksums.txt`](https://github.com/trufflesecurity/trufflehog/releases/download/v3.97.4/trufflehog_3.97.4_checksums.txt).
The PGDG key's primary OpenPGP fingerprint was independently calculated from the
hash-pinned public key and matches the fingerprint above.
The Dockerfile pins the base index, download digests, PGDG package versions, and
Debian snapshot. Apt verifies Debian signatures with its shipped archive keyring
and PGDG signatures with the separately hash-pinned, repository-scoped armored
key. Full GnuPG is not installed. Expired `Valid-Until` checks are disabled only
for the immutable Debian snapshots, never signature verification. The official
PGDG archive retains older package versions; apt preferences exclude every PGDG
package except the five exact pins above.
The complete Debian Git package and HTTPS helper are retained. Native executable
symlinks in the Python/Git tool directories, and PG client version-wrapper links,
are replaced with root-owned regular hard links. Dependencies stay in the
interpreter's `/usr/local/lib/python3.12/site-packages`, not a virtual environment
or user site. Installation requires hashes and binary wheels and disables
bytecode generation. The application lock includes both manifests, optional
dashboard packages, and pytest in one environment. The test target layers its
separate hash lock containing Starlette TestClient's `httpx` dependency; the
runtime target does not contain `httpx`. The separate worker lock contains only
Requests, PyYAML, zstandard, and their four transitives; it excludes PostgreSQL,
server, dashboard, test, and detailed-keycheck dependencies.
`docker/worker-package-pins.json` is the canonical public build-input record for
Linux and Windows worker artifacts. Worker manifests hash every application,
dependency, scanner, detector-policy, Python-runtime, and complete Git-runtime
file. The package grants no provider or detailed keycheck authority and contains
no server or database credentials.
PostgreSQL service starts and automatic cluster creation are disabled during
installation. No database is initialized by the build. Generated distribution
snakeoil TLS keys are removed in the same layer. Package pins make dependency
selection repeatable; this is not a claim of byte-for-byte identical OCI images
across BuildKit versions or package-maintainer timestamp generation.
## Generate Locks
All four requirements lock files are generated by pip-tools on Linux with the pinned
Python 3.12.14 interpreter. Do not edit any generated lock by hand. The initial
compiler bootstrap installs only public `pip==26.2.1` and `pip-tools==7.6.1`, then
resolves and hashes the compiler's entire dependency closure. Subsequent compiler
installs use that generated hash lock.
The generator also copies the existing locks, so normal regeneration retains
valid pins. Use pip-compile's `--upgrade` only for an intentional dependency
refresh, then rebuild and revalidate the dependency image.
Run these Docker commands from the isolated source checkout (`D:\truf-workers` for
this change), using WSL's
`sudo -n docker -H unix:///var/run/docker.sock` in place of `docker` on this host.
The generator receives only the three public application manifests and compiler
inputs. There are no host bind mounts.
```sh
docker build --target lock-generator -t truf-lock-generator:py3.12.14 .
docker run --name truf-runtime-lock truf-lock-generator:py3.12.14
docker cp truf-runtime-lock:/src/docker/requirements.lock docker/requirements.lock
docker rm truf-runtime-lock
docker run --name truf-test-lock truf-lock-generator:py3.12.14 \
python3 -m piptools compile --generate-hashes --allow-unsafe \
--resolver=backtracking --strip-extras --no-emit-index-url \
--no-emit-trusted-host --index-url=https://pypi.org/simple \
--pip-args=--only-binary=:all: \
--output-file=docker/requirements-test.lock docker/requirements-test.in
docker cp truf-test-lock:/src/docker/requirements-test.lock docker/requirements-test.lock
docker rm truf-test-lock
docker run --name truf-worker-lock truf-lock-generator:py3.12.14 \
python3 -m piptools compile --generate-hashes --allow-unsafe \
--resolver=backtracking --strip-extras --no-emit-index-url \
--no-emit-trusted-host --index-url=https://pypi.org/simple \
--pip-args=--only-binary=:all: \
--output-file=docker/requirements-worker.lock docker/requirements-worker.in
docker cp truf-worker-lock:/src/docker/requirements-worker.lock docker/requirements-worker.lock
docker rm truf-worker-lock
docker run --name truf-compiler-lock truf-lock-generator:py3.12.14 \
python3 -m piptools compile --generate-hashes --allow-unsafe \
--resolver=backtracking --strip-extras --no-emit-index-url \
--no-emit-trusted-host --index-url=https://pypi.org/simple \
--pip-args=--only-binary=:all: \
--output-file=docker/build-dependencies/requirements.lock \
docker/build-dependencies/requirements.in
docker cp truf-compiler-lock:/src/docker/build-dependencies/requirements.lock docker/build-dependencies/requirements.lock
docker rm truf-compiler-lock
```
## Build Targets
```sh
docker build --target dependencies -t truf-dependencies:py3.12.14-pg16.15-th3.97.4 .
docker build --target runtime -t truf-runtime:local .
docker build --target test -t truf-test:local .
docker build --target worker -t truf-remote-worker:linux-x86_64 .
```
## Remote Worker Artifacts
The `worker` target is independent of the server runtime. It uses the pinned image
Python and includes the worker authority, required shared DB-free modules, worker
dependency lock, detector policy, TruffleHog, CA roots, tini, and a package-local
complete Git helper tree. The final build executes the scanner and Git version
checks and verifies the full schema-3/protocol-2 capability package as unprivileged UID 10001. PostgreSQL
tools, server runtime/control authority, test code, `httpx`, provider and detailed
keycheck authority, server credentials, and database credentials are absent.
Build the Windows amd64 portable directory and deterministic ZIP from the same
isolated checkout:
```powershell
New-Item -ItemType Directory -Path dist -Force | Out-Null
python -B app/worker_package_builder.py windows `
--project-root . `
--output dist/truf-worker-windows-x86_64 `
--archive dist/truf-worker-windows-x86_64.zip `
--cache build/worker-cache
```
The builder downloads the hash-and-size-pinned Python 3.12.10 embeddable runtime,
MinGit 2.47.1, and TruffleHog 3.97.4, installs the cross-platform worker lock with
hash checking, verifies the complete package, and writes adjacent release JSON.
After extracting the ZIP, run `prepare-worker.ps1` once to replace inherited ACLs,
then use `run-worker.cmd`. Local files rely on private OS ACLs rather than
application-layer encryption; the client has no TLS-verification bypass.
Build a complete distributable release (Windows ZIP, Linux image archive,
Docker bundle, manifests, README, Compose file, and SHA-256 lists) with one
PowerShell command:
```powershell
.\build_worker_release.ps1 -ReleaseName release-YYYYMMDD-vN
```
The script uses native Docker when available and otherwise uses the Docker Engine
in `Ubuntu-24.04` through WSL. Use `-WslDistro NAME` for another distribution.
The destination must not already exist; a failed build remains on disk for
inspection and is never published as a partial replacement.
While the main context allowlist is pending, the dependency build can use this
explicit two-file context from WSL. It does not send any application code, secret
files, findings, state, or original checkout directories to the builder:
```sh
tar -C /mnt/d/truf-docker -cf - Dockerfile docker/requirements.lock \
| sudo -n docker -H unix:///var/run/docker.sock build --file Dockerfile \
--target dependencies -t truf-dependencies:py3.12.14-pg16.15-th3.97.4 -
```
`dependencies` stops before copying application code. `runtime-base` holds the
production filesystem and launch configuration; `test` inherits those exact
contents and adds private test sources. The last/default target, `runtime`, is a
direct alias of `runtime-base` and contains no test tree. The test entrypoint uses
the same interpreter isolation flags and the real `child_bootstrap.py` dependency
path loader, without processing `.pth` files or starting the application.
Application and test copies are owned by `10001:10001`; their directories are
`0700` and regular files `0600`. Build-time checks reject symlinks, special files,
and cached bytecode in these controlled copies. No recursive permission changes
are made to host paths or runtime-mounted data. The default user is `10001:10001`;
the image precreates private `/data` and `/data/home` directories.
## Verified Results
Validated on 2026-09-15 using Ubuntu 24.04 under WSL2, stock Docker Engine 29.8.0,
the local Unix socket, and `sudo -n`. Builds and generation were polled without
printing full dependency logs. A temporary WSL keepalive prevented idle shutdown
during detached lock generation; no host configuration changes were made.
| Result | Value |
| --- | --- |
| Dependency image tag | `truf-dependencies:py3.12.14-pg16.15-th3.97.4` |
| Local dependency image ID | `sha256:3fbdc2ea6fd1199aa742f54fb653e433d79ad3f1a5e4bfeed8b413dc9f704eb5` |
| Built and executed platform | `linux/amd64` |
| Runtime lock | 49 exact package pins, 1059 SHA-256 wheel hashes |
| Runtime lock SHA-256 | `82c69394b116fd8a762c3dea481682045af87c013974fcb78ac0529892188537` |
| Compiler lock | 8 exact package pins, 8 SHA-256 wheel hashes |
| Compiler lock SHA-256 | `0172b08004c6f2b0702ea9a472300cc63243492df2d3d782fd4dbaa612497fd2` |
| Lock replay in the hash-locked generator image | Both files byte-for-byte identical |
| `pip check` | No broken requirements |
- The `dependencies` and `lock-generator` targets built successfully with hash-required, binary-wheel-only installs.
- All 49 package import checks passed normally, then with `-I -S -B` through the actual `child_bootstrap.py` loader for all 10 child kinds. Application entrypoints and providers were not executed.
- Isolated `sys.path` contained only the interpreter ZIP path, standard library, `lib-dynload`, and `/usr/local/lib/python3.12/site-packages`. Neither the working directory nor the user site was added.
- 14,202 dependency-tree permission checks found root ownership, no group/world write access, and no symlinks or bytecode files. The unprivileged UID could not write these dependencies.
- Eleven selected native executables were regular root-owned files. Version and `ldd` checks passed for Python, Git, the Git HTTPS helper, tini, TruffleHog, and the six PG16 tools. TruffleHog is static; other checked ELF binaries had no missing shared libraries.
- TruffleHog global, Git, filesystem, Docker, and Hugging Face help flags were checked, including the scanner's legacy `--local-dev` and `--log-level` options. No scans or provider requests were made.
- No PostgreSQL `PG_VERSION` file or initialized cluster was present. No PostgreSQL server, application, or provider was started. Full GnuPG is absent.
- Minimal bootstrap-only runtime and test fixtures exercised the actual copy stages: `10001:10001`, directories `0700`, files `0600`. Generated in-memory contexts containing a symlink or `.pyc` file were rejected by the build.
- The test target's real tini/Python/bootstrap entrypoint reported `pytest 8.4.2` with networking disabled, a read-only root, and a UID-10001 private `/tmp` tmpfs. Pytest capture needs writable temporary storage even for `--version`.
The temporary runtime/test fixtures were deliberately incomplete and were used
only for dependency and filesystem-policy verification. They are not deployable
application images. No application test suite, PostgreSQL initialization test,
provider integration, or arm64 build was run. The arm64 base and TruffleHog assets
are pinned, but that platform still requires a native or emulated build/test.
## Integration Boundary
The main integration owns `.dockerignore`, `app/container_runtime.py`, Compose,
and tests. The context allowlist must include the exact Dockerfile, dependency
input/lock paths, this metadata file, the new runtime entrypoint, and the intended
test files. Do not replace the deny-by-default context rules with directory-wide
or wildcard exceptions. Keep `.env`, credentials, findings, original runtime
directories, and generated caches excluded.
New exact metadata/input exceptions required in the main-owned `.dockerignore`:
```text
!docker/requirements.in
!docker/requirements.lock
!docker/build-dependencies/requirements.in
!docker/build-dependencies/requirements.lock
!docker/build-dependencies/README.md
```
The Dockerfile is already allowlisted. Main must separately allowlist its
`app/container_runtime.py` and intended test source files once they exist. The
test target copies only `app/` and `tests/`; repository tests that read root-level
Compose, Docker, or PowerShell fixtures still need main-owned fixture integration.
Compose must select the runtime target, enforce a read-only root filesystem,
provide newly initialized Linux writable volumes and a temporary filesystem as
needed, and preserve the unprivileged UID/GID. Runtime startup, PostgreSQL
initialization, provider execution, and existing-data integration are explicitly
outside dependency-build validation.
For a read-only test image, provide private temporary storage, for example
`--tmpfs /tmp:rw,nosuid,nodev,noexec,size=64m,mode=0700,uid=10001,gid=10001` for
version/import checks. Main must choose temporary-storage size and execution
policy appropriate for its full tests. After the entrypoint, allowlist, fixtures,
and Compose changes are integrated, rebuild complete `runtime` and `test` images
and perform the separately authorized integration checks. No files outside the
Dockerfile and `docker/` build inputs were edited, and nothing was staged,
committed, or pushed.
docker/build-dependencies/requirements.in
+3
View File
@@ -0,0 +1,3 @@
# Toolchain only; these packages are not copied from the generator into runtime.
pip==26.2.1
pip-tools==7.6.1
docker/build-dependencies/requirements.lock
+40
View File
@@ -0,0 +1,40 @@
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# See docker/build-dependencies/README.md for the pinned Python 3.12.14 pip-tools generation command.
#
--only-binary :all:
build==1.6.1 \
--hash=sha256:ecd351a4be9d35a9eaaba244a7687143c9c7d4aea6ac964e7e7ddab20cbcf4e7
# via pip-tools
click==8.5.0 \
--hash=sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360
# via pip-tools
packaging==26.3 \
--hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c
# via
# build
# wheel
pip-tools==7.6.1 \
--hash=sha256:6111c8b4b07fd14b7223ca921485b0e96cf66e20bf94da95eeed9845f510cb8f
# via -r docker/build-dependencies/requirements.in
pyproject-hooks==1.2.0 \
--hash=sha256:9e5c6bfa8dcc30091c74b0cf803c81fdd29d94f01992a7707bc97babb1141913
# via
# build
# pip-tools
wheel==0.48.0 \
--hash=sha256:3217dcc807155e45db462d7ef2431f5ddda0d7273b700d05a67b271ceb1287ab
# via pip-tools
# The following packages are considered to be unsafe in a requirements file:
pip==26.2.1 \
--hash=sha256:71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
# via
# -r docker/build-dependencies/requirements.in
# pip-tools
setuptools==84.0.0 \
--hash=sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670
# via pip-tools