Self-host the trace worker

The API never computes activations itself. A worker, python/trace-worker (provenance-trace-worker), runs beside the model, hooks every mlp.down_proj, and serves readings, completions and attribution graphs over HTTP. The same image runs on our GPU pod for hosted inference and, for Enterprise, beside your own model.

Hardware

Qwen3-8B in bf16 needs about 16 GB of GPU memory — one 24–48 GB card. Larger models (Qwen3-32B) are a configuration change on a larger card. Managed inference APIs can't be used: they don't expose activations.

Run it

docker build -f python/trace-worker/Dockerfile --build-arg TORCH_INDEX=cu124 -t provenance-trace-worker .
docker run --gpus all -p 8090:8090 \
  -e TRACE_WORKER_BACKEND=hf \
  -e TRACE_MODEL=Qwen/Qwen3-8B \
  -e TRACE_WORKER_TOKEN="$(openssl rand -hex 32)" \
  provenance-trace-worker

Locally, docker compose --profile gpu up trace-worker starts it next to the stack (the default docker compose up is unchanged). Without a GPU, TORCH_INDEX=cpu TRACE_WORKER_BACKEND=fixture replays the sandbox fixtures.

GET /health answers {"status": "loading" | "ready", "backend", "models"}. While the model loads, every other route returns 503 {"error":"cold"}, which the API surfaces as trace_backend_cold.

Connect the API

TRACE_BACKEND=remote
TRACE_WORKER_URL=https://trace-worker.internal.example:8090   # https + public host in production
TRACE_WORKER_TOKEN=<the same token>
TRACE_MASK_SALT=<≥16 random chars>                           # required in production

The worker URL passes the same SSRF guard as outbound webhooks: in production it must be https and must not resolve to a private or loopback address. TRACE_BACKEND=fixture is refused in production, and none makes every trace route answer 503 trace_backend_not_configured while the registry stays readable.

Deployment options

• Dedicated GPU pod (recommended). RunPod, Lambda or Vast — the model stays loaded; no cold starts.
• Serverless GPU. Modal or RunPod serverless. Cold starts take minutes while weights load. Set TRACE_KEEP_WARM_INTERVAL_MS (live posture only) and the API pings /health on that interval to keep the model resident. Passing a trace_id from an earlier traced inference never waits on the GPU.

Regenerating the sandbox fixtures

pip install -e 'python/trace-worker[hf]'
python python/trace-worker/scripts/gen_fixtures.py   # Qwen/Qwen2.5-0.5B-Instruct on CPU, ~10 min

Everything it writes is measured on the model — activations, canary thresholds and held-out precision/recall, completions, attributions and ablations. Only the labels are heuristic, and the labeler is recorded on every neuron.