The repository ships everything the service needs to run — a Dockerfile, a
Compose file, config/tenants.example.json, and scripts/hash-secret.mjs —
and no runbook, because docs/ is not published. This page is that runbook.
Running your own instance makes you the operator. Nobody issues you
credentials; you issue them. That is the part Get started
assumes is already done, and it is two things: a server HMAC key and a
tenants.json holding at least one tenant.
Hand it to an agent
Copy this into an agent that has a shell on the server, or SSH access to it: Claude Code, Codex, Cursor, or anything else that can run commands and read back what they printed. It asks you for the handful of values it cannot know, does the work, and stops rather than guessing.
Agent prompt
Deploy TEE Docker to a Linux server
Preflights the host, provisions a tenant, wires the domain, boots the container, and proves it is actually up — not merely running.
You are deploying tee-docker, a self-custody wallet API, onto a Linux server I
control. Work on that server, directly or over SSH. The reference runbook is
https://tee.hypetrade.xyz/deploy/index.md — fetch it if you can; this prompt is
self-contained if you cannot.
Before you touch anything: read the INPUTS, ask me for every one that is blank,
then echo back a plan and wait for me to say go.
## INPUTS
REPO = https://github.com/HypeTradeXYZ/tee-docker
MODE = bare-docker | dokploy # is Dokploy running on this host?
API_HOSTNAME = # e.g. tee-api.example.com; DNS must already point here
HOST_DIR = /opt/tee-docker # absolute host path for everything persistent.
# MODE=dokploy: must stay /opt/tee-docker. The
# shipped docker-compose.yml writes that path
# LITERALLY into its volumes, so no environment
# variable can move it - only editing the file.
TENANT_ID = # lowercase slug, up to 63 chars, e.g. acme
## RULES - not suggestions
1. Never invent a value above. Ask me.
2. You generate the secrets. Print each one to me EXACTLY ONCE, in a clearly
marked block, and never again - not in a file, not in a later summary, not
in your final report. Say plainly that the service stores only a hash and
can never give a secret back.
3. Never pass a secret as a command-line argument. Pipe it on stdin. An argv
secret is in `ps` output and in shell history permanently.
4. Exactly one instance may hold a state directory, and the service never
reclaims that lock automatically. Do not raise replicas above 1, do not
point a second stack at the same HOST_DIR, and never delete
state.json.lock while any container might still be running.
5. If a step fails, stop and show me the real log output. Do not set
TEE_ALLOW_UNSAFE_KDF, do not widen file permissions to make an error go
away, and do not remove the health check.
6. Do not go looking for docs/DEPLOY.md. It is not in the repository.
## STEPS
1. PREFLIGHT - report, fix nothing yet:
uname -m (expect x86_64 or aarch64)
nproc; free -m (below ~2 GB RAM, warn me the build may be OOM-killed)
docker --version; docker compose version
getent hosts $API_HOSTNAME (must resolve to this server's public IP)
MODE=dokploy: docker network inspect dokploy-network
2. Clone REPO into a build directory and cd into it.
3. PERSISTENT DIRECTORY. The container runs as uid 1000, and on first boot it
chmods its own state directory to 0700 - which requires ownership. A
root-owned bind mount fails boot with EPERM instead of anything readable:
sudo mkdir -p $HOST_DIR/state $HOST_DIR/data
sudo chown -R 1000:1000 $HOST_DIR
sudo chmod 700 $HOST_DIR/state $HOST_DIR/data
4. SERVER HMAC KEY. This value can never be changed once tenants exist.
It is the HMAC root for every tenant secretHash in tenants.json. Change it
and every tenant fails authentication with 401, permanently, until you
recompute their secretHash values under the new key.
node -e "console.log(require('node:crypto').randomBytes(32).toString('hex'))"
Print it to me once. Tell me to store it in a password manager now, because
the secretHashes you deploy alongside a restored data/ directory were
computed under it. Keep it in a shell variable for step 6 only.
VERIFY IT IS EXACTLY 64 HEX CHARACTERS before going further. The parser
decodes hex only when the value is all-hex AND of even length, and silently
falls back to UTF-8 otherwise - so a 63-character value from a truncated
paste is accepted as a completely different 63-byte key, clears the 16-byte
minimum, boots normally, and gives every tenant 401. "Fixing" that later,
after workspaces exist, IS a rotation:
if [[ ${#HMAC} -eq 64 && $HMAC =~ ^[0-9a-f]+$ ]]
then echo "key OK"; else echo "KEY IS WRONG - stop, do not deploy"; fi
5. TENANT CREDENTIALS. Generate:
apiKey - a public identifier, at least 16 characters, e.g. ak_live_<16 hex>
apiSecret - 32 random bytes, base64url
Print both to me once.
6. SECRET HASH - stdin only, never argv:
printf %s "$API_SECRET" | TEE_SECRET_HMAC_KEY="$HMAC" node scripts/hash-secret.mjs
The output is exactly 64 hex characters. hash-secret.mjs uses only
node:crypto, so it needs no install step.
7. TENANTS.JSON. Build it from config/tenants.example.json: drop the _comment
array, replace every placeholder, keep limits (maxWorkspaces, maxWallets).
Install it, then delete your working copy:
sudo install -o 1000 -g 1000 -m 600 tenants.json $HOST_DIR/tenants.json
It must EXIST before first start: Docker creates a DIRECTORY at a missing
single-file mount target, and the service then fails on operator config it
cannot read.
8. SERVE IT.
MODE=dokploy - create a COMPOSE service, not an Application: an Application
is a Swarm service whose rolling update starts the replacement before
stopping the old one, meets the held state lock, and crash-loops on every
deploy. In the Environment tab there are exactly two variables worth
setting: TEE_SECRET_HMAC_KEY (required - nothing else is) and
TEE_API_HOST=$API_HOSTNAME. Do NOT invent a variable for the host paths:
the shipped compose file writes /opt/tee-docker literally into its
volumes, because Compose resolves volume paths when it PARSES the file,
before any container exists. TEE_API_HOST is the trap here: it DEFAULTS to
the upstream's own hostname rather than refusing to start, so leaving it
unset comes up healthy while Traefik routes a host we do not control.
Check tls.certresolver=letsencrypt matches the resolver this Dokploy
install actually uses. Deploy from the dashboard.
MODE=bare-docker - there is no Traefik and no dokploy-network. Write
docker-compose.self.yml (the deploy page has the full file) that keeps the
build, volumes, healthcheck, stop_grace_period and resource limits, drops
the labels and the external network, and publishes 127.0.0.1:3000:3000 -
loopback, not 0.0.0.0. Keep stop_grace_period at 30s: a KDF unlock or a
batch wallet derive can outlast Docker's default 10s grace, so a redeploy
mid-operation SIGKILLs the process and strands the state lock. Put TEE_HOST_DIR and
TEE_SECRET_HMAC_KEY in a chmod 600 .env beside it - TEE_HOST_DIR is a
variable this self-hosted file introduces, not one the service reads.
Then:
docker compose -f docker-compose.self.yml up -d --build
Terminate TLS in a reverse proxy in front of it. Tenant secrets and
workspace tokens travel these requests; do not serve them over plain HTTP.
9. VERIFY - running is not up:
curl -fsS http://127.0.0.1:3000/v1/health # bare-docker, on the host
curl -fsS https://$API_HOSTNAME/v1/health # through the proxy
Expect {"status":"ok"}. In the logs expect
native KDF backend verified (probe NNNms)
If instead you see "KDF readiness check failed (probe_slow); refusing
startup", this host is too slow or too throttled for its probe budget:
raise TEE_KDF_MAX_PROBE_MS and restart. The service default is 1000ms, but
the shipped docker-compose.yml already overrides it to 2500 - so check
which number is actually in effect before assuming it is 1000. The ceiling
is 60000. If it says backend_unsafe, the image is not the shipped glibc one
- do not work around it, and never set TEE_ALLOW_UNSAFE_KDF; tell me.
10. PROVE THE CREDENTIALS AGREE with the HMAC key - the only check that
actually tests tenants.json:
curl -X POST "https://$API_HOSTNAME/v1/workspaces" \
-H "X-Api-Key: $API_KEY" -H "X-Api-Secret: $API_SECRET" \
-H "content-type: application/json" \
-d '{"slug":"smoke","password":"<a strong password you print to me once>"}'
A 201 is success. A 401 bad_api_key means secretHash was computed under a
different key than the container is running with.
11. REPORT: the health response, the KDF probe line, the smoke result, every
file you created with its owner and mode, and a checklist of every secret
you printed so I can confirm I stored each one. Confirm you deleted the
build directory's copy of tenants.json.Fill these in before you paste it
| Input | What it is | If you get it wrong |
|---|---|---|
MODE |
dokploy if Dokploy manages this host, otherwise bare-docker |
The wrong networking. The shipped Compose file publishes no host port and expects Traefik on dokploy-network |
API_HOSTNAME |
The hostname you will call, with DNS already pointing at this server | Certificate issuance fails, and the domain 404s at the proxy |
HOST_DIR |
One absolute path holding state/, data/ and tenants.json. On Dokploy it must stay /opt/tee-docker — the shipped Compose file hard-codes that path |
Everything persistent lands inside the container and dies with it |
TENANT_ID |
A lowercase slug for your first tenant, up to 63 characters | Boot fails on schema validation |
What the deployment actually is
One container, one persistent directory, one configuration file you write by hand.
| Piece | Where it lives | Why it is separate |
|---|---|---|
tenants.json |
$HOST_DIR/tenants.json, mounted read-only |
Holds every tenant’s secret hash. Never baked into an image |
state/ |
$HOST_DIR/state, 0700, uid 1000 |
Machine-owned ledger. Never hand-edited |
data/ |
$HOST_DIR/data, 0700, uid 1000 |
Workspace and wallet material |
TEE_SECRET_HMAC_KEY |
Environment only | The key every secretHash is computed under. A data/ backup is unusable without it |
errors.json |
Baked into the image | Ships with the code, so an upgrade cannot leave a stale copy behind |
Stand it up by hand
Prepare the persistent directory
The container runs as the unprivileged node user, uid 1000. On first boot
the service creates its state directory and chmods it to 0700 — and chmod
requires ownership, so a root-owned bind mount fails boot with EPERM rather
than anything that explains itself.
sudo mkdir -p /opt/tee-docker/state /opt/tee-docker/data
sudo chown -R 1000:1000 /opt/tee-docker
sudo chmod 700 /opt/tee-docker/state /opt/tee-docker/dataA named volume does not need this — the image creates those directories as
node, so the volume inherits the right owner. Only bind mounts, which keep
the host directory’s own ownership, need the chown.
Generate the server HMAC key
node -e "console.log(require('node:crypto').randomBytes(32).toString('hex'))"Store it before you use it. Anything under 16 bytes is refused at boot; there is no default, because a per-process random key would break every configured hash and a fixed fallback would be a published credential.
Check it is exactly 64 hex characters before the first boot. The parser
decodes hex only when the value is all-hex and of even length, and silently
falls back to UTF-8 otherwise. Drop one character in transit and the
63-character value is accepted as a completely different 63-byte key — it
still clears the 16-byte minimum, so the service boots normally and every
tenant gets 401.
if [[ ${#TEE_SECRET_HMAC_KEY} -eq 64 && $TEE_SECRET_HMAC_KEY =~ ^[0-9a-f]+$ ]]
then echo "key OK"; else echo "KEY IS WRONG - do not deploy"; fiThat check matters more than it looks, because of the rotation trap above:
deploy with a mistyped key, create workspaces, discover the 401s, then “fix”
the key — and you have just performed a rotation.
Hash your first tenant’s secret
The API key is a public identifier — at least 16 characters. The API secret is yours to generate; the service only ever stores its HMAC.
API_SECRET=$(node -e "console.log(require('node:crypto').randomBytes(32).toString('base64url'))")
printf %s "$API_SECRET" | TEE_SECRET_HMAC_KEY=the-key-from-the-previous-step node scripts/hash-secret.mjsPipe it on stdin. Passed as an argument it lands in ps output and in your
shell history, and it cannot be un-leaked afterwards. hash-secret.mjs uses
only node:crypto, so it runs on a checkout with nothing installed.
The output is exactly 64 hex characters — that is secretHash.
Write tenants.json
Build it from config/tenants.example.json, dropping the _comment array.
{
"tenants": [
{
"id": "acme",
"apiKey": "ak_live_0123456789abcdef",
"secretHash": "the 64 hex characters from the previous step",
"limits": { "maxWorkspaces": 5, "maxWallets": 200, "maxUnlockedWorkspaces": 8 }
}
]
}If a browser will call the tenant-credential routes, that tenant needs an
origins array — see Letting a browser call it.
A browser app that uses a scoped API key needs nothing; those routes are open.
Install it before the first start, owned by uid 1000 and readable by nobody else:
sudo install -o 1000 -g 1000 -m 600 tenants.json /opt/tee-docker/tenants.jsonServe it
The shipped docker-compose.yml publishes no host port and expects Traefik on
an external dokploy-network. On a plain host, keep it untouched — so git pull stays clean — and write a sibling file instead.
# docker-compose.self.yml — plain Docker host, own reverse proxy in front.
services:
tee-docker:
build:
context: .
dockerfile: Dockerfile
image: tee-docker:local
init: true
restart: unless-stopped
# Must outlast the longest in-flight operation. The service never calls a
# chain RPC, so the longest drain is a KDF unlock or a batch wallet
# derive/seed, while Docker's default stop grace is only 10s — so a redeploy
# mid-operation SIGKILLs the process mid-drain. SIGKILL cannot be caught, so
# the fatal-exit lock release does not cover it and the state lock is
# stranded for a human to clear. Do not lower this.
stop_grace_period: 30s
# Loopback only. Terminate TLS in a proxy in front of this; tenant
# secrets and workspace tokens travel these requests.
ports:
- "127.0.0.1:3000:3000"
environment:
TEE_SECRET_HMAC_KEY: "${TEE_SECRET_HMAC_KEY:?set it in .env beside this file}"
# Pinned, not just inherited from the image. NODE_ENV=test makes the
# dependency run REDUCED KDF iterations and makes the boot probe skip
# itself with only an info log. A literal here beats a stray value in
# the .env file, so a typo cannot quietly weaken key derivation.
NODE_ENV: "production"
# The service default is 1000; the shipped docker-compose.yml uses 2500.
TEE_KDF_MAX_PROBE_MS: "${TEE_KDF_MAX_PROBE_MS:-2500}"
volumes:
- "${TEE_HOST_DIR:?set TEE_HOST_DIR to an absolute host path}/state:/var/lib/tee-docker/state"
- "${TEE_HOST_DIR:?set TEE_HOST_DIR to an absolute host path}/data:/var/lib/tee-docker/data"
- "${TEE_HOST_DIR:?set TEE_HOST_DIR to an absolute host path}/tenants.json:/app/config/tenants.json:ro"
healthcheck:
test:
[
"CMD",
"node",
"-e",
"fetch('http://127.0.0.1:3000/v1/health').then(r=>r.json()).then(j=>process.exit(j.status==='ok'?0:1)).catch(()=>process.exit(1))",
]
interval: 30s
timeout: 5s
start_period: 60s
retries: 3
deploy:
resources:
limits:
cpus: "1.0"
memory: "1024M"Put the two variables in a .env beside it and lock it down — it holds the
HMAC key in plaintext. .dockerignore already keeps .env* out of the build
context, so it cannot end up inside the image.
printf 'TEE_HOST_DIR=/opt/tee-docker\nTEE_SECRET_HMAC_KEY=%s\n' "$HMAC" > .env
chmod 600 .env
docker compose -f docker-compose.self.yml up -d --buildThen point your proxy at 127.0.0.1:3000. Caddy is two lines:
tee-api.example.com {
reverse_proxy 127.0.0.1:3000
}Create a Compose service, not an Application. An Application runs as a Swarm service, and a Swarm rolling update starts the replacement task before stopping the old one — it meets the held state lock and crash-loops on every single deploy until someone deletes the lock file by hand. Compose recreates a changed service stop-then-start, which hands the lock over cleanly.
The same reasoning pins deploy.replicas to 1. Do not raise it.
Point the service at your fork or clone. The Traefik router is declared in the Compose file’s labels and reads its hostname from the environment, so the domain needs no edit to the file — but the host paths do, as below.
In the Environment tab set:
| Variable | Value | Required |
|---|---|---|
TEE_SECRET_HMAC_KEY |
the key you generated | Yes — the only one |
TEE_API_HOST |
tee-api.example.com |
No, but see below |
TEE_TLS_CERT_RESOLVER |
letsencrypt |
No — set it if your Traefik resolver differs |
The environment block is otherwise literals: TEE_KDF_MAX_PROBE_MS is pinned
to 2500 there rather than the service default of 1000, and a set of optional
operational knobs sit at their service defaults — rate limits
(TEE_MINT_RATE_LIMIT, TEE_WORKSPACE_CREATE_RATE_LIMIT), the recreate cooldown
(TEE_WORKSPACE_RECREATE_COOLDOWN_SEC), and capacity ceilings
(TEE_MAX_UNLOCKED_WORKSPACES, TEE_MAX_TOKEN_LEASES_PER_WORKSPACE); change them
only with reason, and docker-compose.yml holds the current values. The only ${VAR} interpolations are TEE_API_HOST and
TEE_TLS_CERT_RESOLVER, both in the Traefik labels.
The Traefik label tls.certresolver defaults to letsencrypt; if your Dokploy
install names its resolver something else, set TEE_TLS_CERT_RESOLVER in the
Environment tab to match rather than editing the label. A mismatch presents the
same way as a forgotten TEE_API_HOST — a Traefik 404, not a certificate error.
Everything else in the file has a working default. Nothing goes in the Domains tab — the labels already declare the router.
Prove it is up, not merely running
curl -fsS https://tee-api.example.com/v1/health{ "status": "ok" }Then prove tenants.json and the HMAC key agree, which nothing else tests:
curl -X POST "https://tee-api.example.com/v1/workspaces" \
-H "X-Api-Key: $API_KEY" \
-H "X-Api-Secret: $API_SECRET" \
-H "content-type: application/json" \
-d '{"slug":"smoke","password":"a-strong-workspace-password"}'A 401 bad_api_key here means the secretHash was
computed under a different key than the container is running with. Fix it now,
before any workspace exists — once one does, changing the key is no longer
recoverable. From here, Get started picks up at minting a
token.
Environments that need a second look
A small or shared VPS
Password derivation is Argon2, which is deliberately memory-hard. Startup runs
one real derivation before the port opens, and refuses to continue if it
takes longer than TEE_KDF_MAX_PROBE_MS.
The service default is 1000. The shipped docker-compose.yml already
raises it to 2500, so on a Compose or Dokploy deployment that is the number in
force — check which one you are actually running under before concluding the
budget is 1000.
The raise is not caution for its own sake. The same probe measured 559 ms and 811 ms on two runs of the same idle Apple-silicon laptop: run-to-run variance alone nearly breaches the 1000 ms default on fast hardware, before a throttled shared core is involved.
environment:
TEE_KDF_MAX_PROBE_MS: "2500"The ceiling is 60000, and the check itself is not optional.
TEE_ALLOW_UNSAFE_KDF=1 exists as an emergency operator acknowledgement — it
still runs and reports the probe, and it logs
UNSAFE KDF ACCEPTED BY OPERATOR (<reason>). It must be exactly 0 or 1; any
other value stops the boot. Reaching for it because boot failed is choosing to
run password derivation you have not verified.
Its predecessor TEE_SKIP_KDF_CHECK was removed. Setting it to anything at
all, including an empty value, is now a hard boot failure naming its
replacement — so an old .env carried across from a previous version stops the
service rather than silently skipping the check.
Memory matters twice over. At runtime, an OOM-kill strands the state lock and
costs you a manual recovery, so keep the memory limit at 1024M or above. At
build time, the builder stage installs a toolchain and compiles TypeScript; on
a 1 GB instance that is the step most likely to be killed without a useful
message. Build somewhere larger and push the image, or size the box up.
arm64 and x86_64
node:22-bookworm-slim — Debian, glibc — is a deliberate choice, not a default.
Password derivation goes through @node-rs/argon2, whose linux-x64-gnu and
linux-arm64-gnu prebuilds load on glibc with no C toolchain present. Startup
refuses to continue unless that native backend is the one actually selected, so
a base image that silently falls back to a slower implementation is a boot
failure, not a slow build. It presents as:
KDF readiness check failed (backend_unsafe); refusing startupBoth architectures are fine. What is not fine is a mismatch: building the image
on an Apple-silicon laptop and pushing it to an x86_64 server ships
linux-arm64-gnu binaries that cannot load there. Both Compose files here build
on the host, which sidesteps it entirely — if you build elsewhere, build for the
target platform explicitly.
Swapping the base to node:22-alpine fails the same way. musl is not glibc, and
the prebuilds do not load.
Dokploy
Covered in the tab above, and worth restating because each one costs a deploy cycle to discover:
- Compose, never Application. Swarm’s rolling update overlaps the old and new tasks, and two instances against one state directory is exactly what the lock refuses.
replicas: 1is a correctness constraint, not a capacity decision.- The certresolver name defaults to
letsencrypt. If your install names it something else, setTEE_TLS_CERT_RESOLVERto match, or the domain 404s. - Nothing goes in the Domains tab. The router is declared in the labels.
Bare Docker behind your own proxy
The shipped Compose file deliberately publishes no host port — Traefik reaches
the container over dokploy-network. Without that network, nothing reaches it
at all. The docker-compose.self.yml above publishes 127.0.0.1:3000 instead:
bound to loopback, so only a proxy on the same host can reach it. Publishing
0.0.0.0:3000 puts a wallet API on the public internet with no TLS in front of
it.
If you are on Podman rather than Docker, the uid-1000 ownership step works
differently under rootless mode — the container’s uid 1000 maps to a different
host uid through your subuid range. Either use a named volume, or work out the
mapped uid before you chown.
When it will not come up
Every one of these is a hard boot failure by design: the service refuses to start half-understood rather than run in a state you did not intend.
The config rules below tightened after origins shipped, so
a file that started an older build can refuse a newer one — the trigger being a
restart rather than the edit. Run pnpm check-config against your tenants file
before pulling a new image, and it will name any entry that would now be refused.
| What you see | What it is | What to do |
|---|---|---|
TEE_SECRET_HMAC_KEY is not set. |
No key in the environment | Set it. There is no default, deliberately |
TEE_SECRET_HMAC_KEY is too short |
Key under 16 bytes | Generate a 32-byte one |
TEE_SKIP_KDF_CHECK was removed |
An old .env still sets it. Any value, including empty, is fatal |
Delete the line. Use TEE_ALLOW_UNSAFE_KDF=1 only as a deliberate acknowledgement |
TEE_KDF_MAX_PROBE_MS must be a positive safe integer at most 60000 |
Out of range, zero, negative, or not a number | Set an integer between 1 and 60000 |
cannot read operator config at /app/config/tenants.json |
The host file was missing at first start, so Docker created a directory there | sudo rm -rf $HOST_DIR/tenants.json, write the real file, restart |
EPERM / EACCES during boot |
Bind mount not owned by uid 1000 | sudo chown -R 1000:1000 $HOST_DIR |
ZodError naming tenants.N.origins.M and expected an origin like https://app.example.com |
An origin carrying upper case, a path, a wildcard, credentials or a trailing slash. A browser could never match it, so it is refused rather than left looking effective | Write the origin exactly as a browser sends it — lower-case scheme and host, no trailing slash. Then pnpm check-config before restarting |
ZodError naming tenants.N.id and tenant id must not be a reserved name |
A tenant id of con, prn, aux or nul. These are refused as directory names, and a tenant using one can create no workspace and mint no token |
Rename the tenant. Note the config is a list, so this one entry stops every other tenant from starting |
KDF readiness check failed (probe_slow); refusing startup |
The derivation exceeded its budget on a slow or throttled host | Raise TEE_KDF_MAX_PROBE_MS, or give the container more CPU |
KDF readiness check failed (backend_unsafe); refusing startup |
The native Argon2 backend is not the one selected — usually a swapped base image or an architecture mismatch. This is the boot-time face of the kdf_backend_unsafe condition, which is why you see it in the logs rather than as an HTTP response |
Use the shipped node:22-bookworm-slim build, built for the target architecture |
KDF readiness check failed (probe_failed); refusing startup |
The probe threw rather than finishing. It derives a throwaway workspace under the container’s temp directory, so this is usually no writable temp space — or the native module failing to load at all | Check the temp directory and its free space; if that is fine, treat it as the backend_unsafe case |
service state is already locked. |
Another instance holds it — or one died without releasing it | See below. Do not delete anything yet |
| Traefik 404 at the domain | Certresolver name mismatch, or DNS not resolving | Check the label against your Traefik config, and the A record |
401 bad_api_key on a correct-looking call |
secretHash computed under a different HMAC key — often a key truncated in transit and silently accepted as UTF-8 |
Recompute every tenant hash under the running key. If the key changed rather than the hashes, stop and read the rotation warning above — every tenant stays locked out until their hashes are recomputed |
Reading the logs
The service logs to stdout, so the container runtime holds them. There is no log file to find.
docker compose -f docker-compose.self.yml logs -f tee-dockerOn Dokploy, the same stream is the service’s Logs tab.
Output is colourised even when it is not a terminal, which makes saved logs
noisy. Set NO_COLOR=1 in the environment for plain text before you pipe logs
anywhere.
A healthy boot says so explicitly:
[Nest] 1 - 08/20/2026, 5:25:20 PM LOG [KdfCheckService] native KDF backend verified (probe 559ms)
[Nest] 1 - 08/20/2026, 5:25:20 PM LOG [bootstrap] tee-docker listening on :3000Failed key exports
Export is the highest-consequence route in the service, so it is the one
operation where when did this key leave has to be answerable afterwards.
Every attempt is logged — successes and failures alike — as a structured
record under the event name key_export.
An export that fails after passing the scope check logs outcome: 'FAILURE'
at WARN, from ExportController:
[Nest] 1 - 08/20/2026, 5:25:20 PM WARN [ExportController] Object(8) {
event: 'key_export',
outcome: 'FAILURE',
tenantId: 'acme',
workspaceSlug: 'demo',
accountSlug: 'treasury',
target: 'privateKey',
walletId: 1,
vm: 'evm'
}An export refused because the token lacked the export scope never reaches
the controller. It is logged separately by the scopes guard as
outcome: 'DENIED', and it is the record that carries a requestId you can
match against the client’s error body:
[Nest] 1 - 08/20/2026, 5:25:20 PM WARN [ScopesGuard] Object(7) {
event: 'key_export',
outcome: 'DENIED',
tenantId: 'acme',
workspaceSlug: 'demo',
target: 'privateKey',
required: [ 'export' ],
requestId: '01J...'
}Both are pretty-printed across several lines, so a bare search finds the header and drops the fields. Ask for trailing context:
docker logs tee-docker 2>&1 | rg -A 10 key_exportReading the outcomes:
| Sequence | What happened |
|---|---|
ATTEMPT then SUCCESS |
Sealed material was returned. Note the time |
ATTEMPT then FAILURE |
The caller had the scope; the operation failed — wrong account kind, unknown wallet, or a VM the wallet has no address for |
DENIED alone, no ATTEMPT |
The token lacked export. Nothing was decrypted, nothing was read |
ATTEMPT with no terminal record |
The process died mid-operation. Treat it as unresolved |
The record deliberately never contains key material, a mnemonic, or the sealed blob — only who asked, for what, and how it ended.
The super-admin tier
Above every tenant sits a super-admin tier, authenticated by a single
X-Admin-Key header. It is off by default — it exists only when you set an
admin-key value in the environment, at least 16 characters, treated like the HMAC
key: generate it once and keep it in a secret store. Today it raises a tenant’s
maxWorkspaces or maxWallets ceiling, raise-only, via
POST /v1/admin/tenants/{id}/limits, alongside three read-only
operational endpoints — diagnostics, activity, and metrics. The lift is persisted to
state.json, so it survives a restart even though tenants.json is mounted
read-only.
The exact variable name is kept out of the API surface by design; it is set as an
environment variable on the service, the same way as TEE_SECRET_HMAC_KEY. Leave
it unset and the tier simply does not exist.
Redeploys and backups
An ordinary redeploy is safe. Compose stops the running container before starting its replacement, so the lock is released and retaken in order. There is a brief gap with no service: this deployment cannot do zero-downtime releases, by design, because the alternative is two writers on one workspace.
What is never safe is anything that leaves two containers against one state
directory — raising replicas, a second service pointed at the same host
directory, or a manual docker compose up alongside a Dokploy-managed stack.
Back up the whole of $HOST_DIR with the service stopped. data/ holds
wallet material and state/ holds the ledger describing it; a backup that
catches the two at different moments may not restore cleanly. Exclude
state.json.lock — it is process state, not data — or delete it from the
restored copy before starting.
The HMAC key belongs in your secret store, not in the filesystem backup. A
restored data/ directory is unusable without it, and no part of the backup
contains it.
What to collect from a developer
Once the service is up, onboarding someone onto it is one value in their tenant
entry in tenants.json. It is edited by hand, takes effect only on restart, and
has no endpoint behind it.
| Ask them for | Lands in | Without it |
|---|---|---|
| The origins a browser calls the tenant-credential routes from — only if a page hits those routes directly | origins |
Those tenant-route browser calls fail while curl works. A browser app using a scoped API key needs nothing here — those routes are open |
That is the one thing about a tenant a developer supplies rather than you deciding, and it strands them in a way that looks like a bug in your deployment rather than missing configuration.
Letting a browser call it
Everything above assumes a server-side caller — curl, or your own backend.
A browser is different, and the service splits browser access by route.
Scoped-key routes are open to any origin. Accounts, wallets, signing, export, and the rest of the bearer surface answer a browser on any domain and let its script read the response — because a scoped API key is meant to live in an end user’s browser on a domain you cannot know in advance. You configure nothing for this; it is on by default.
The six tenant-credential routes stay allowlisted. Token mint, scoped-key mint, workspace create / list / delete, and quota let a browser read a response only if the calling origin is listed on a tenant — those credentials are server-side only, so the allowlist is a backstop for the rare page that calls them directly.
That allowlist is a per-tenant origins array in tenants.json:
{
"tenants": [
{
"id": "acme",
"apiKey": "ak_live_0123456789abcdef",
"secretHash": "the 64 hex characters from earlier",
"limits": { "maxWorkspaces": 5, "maxWallets": 200 },
"origins": ["https://app.example.com", "http://localhost:4321"]
}
]
}CORS is not an access-control boundary
Worth being blunt about, because opening the scoped routes to every origin sounds more permissive than it is.
Open or allowlisted, CORS grants nothing on its own. Every route still
requires a valid scoped API key or workspace token — or, on the tenant routes,
X-Api-Key + X-Api-Secret — and the service never sends
Access-Control-Allow-Credentials, so nothing reads a cookie and no page rides
an ambient session. CORS decides only which page may read a response it was
already authorized to receive. A leaked key is what would let another site act;
the key’s account/wallet scoping and the
durable-key model are the containment, not CORS.
What each entry must look like
Exactly scheme://host with an optional port, http or https, and nothing
else — because that is all a browser ever sends as an Origin.
https://app.example.com ✓
http://localhost:4321 ✓
https://sub.domain.example.com:8443 ✓Everything below is rejected at boot — the service refuses to start rather than carry an entry that could never match:
| Rejected | Why |
|---|---|
* |
Not an origin. There is no wildcard mode |
https://*.example.com |
No wildcard subdomains either — list each one |
https://app.example.com/ |
A trailing slash. A browser never sends one |
https://app.example.com/path |
Origins have no path |
https://user:[email protected] |
Origins carry no credentials |
app.example.com |
No scheme |
ftp://app.example.com |
http and https only |
https://APP.example.com |
Upper case. A browser lower-cases the scheme and host |
"" |
Empty string |
Failing closed is the point: an entry a browser can never match would otherwise
sit in the allowlist looking effective. * is rejected here too — the allowlist
is exact-match only; the scoped routes are open by the service’s own behaviour,
not by a wildcard you write.
Casing is worth calling out because the rule is not obvious: a browser
lower-cases the scheme and host when it builds an Origin, and the allowlist is
matched as an exact string, so https://APP.example.com could never match
anything. It is refused at boot rather than silently accepted and never used.
The table above is not exhaustive, and the entries it does not catch are the
ones to watch. An explicit default port — https://app.example.com:443 or
http://app.example.com:80 — is accepted at boot and can never match either,
because a browser omits the default port from an Origin. Write the origin
exactly as a browser would send it: scheme, host, and a port only when it is
non-default.
What CORS advertises
The origins allowlist governs the tenant-credential routes only. Leave it
off every tenant and those routes allow no browser reads; the scoped routes stay
open regardless. Set it on any tenant and the allowlist is the union across all
tenants — a preflight arrives before authentication, so there is no tenant to
scope it to yet, and that union decides readability on the tenant routes only.
Both sets advertise the same methods, headers, and cache:
| Value | |
|---|---|
| Methods advertised | GET, POST, PUT, DELETE — plus OPTIONS, which is the preflight itself |
| Request headers accepted | authorization, content-type, x-api-key, x-api-secret, x-request-id |
| Response headers readable from script | x-request-id, retry-after. Nothing else |
| Credentials | Off. Access-Control-Allow-Credentials is never sent — auth is header-based and nothing reads a cookie, so a browser client must set its auth headers explicitly |
| Preflight cache | 600 seconds |
The published route surface is 36 routes. Preflight is handled as
middleware, not as a route, so no OPTIONS entries appear in the reference.
When it does not work
A scoped-key route failing in a browser is not a CORS problem — those routes are open to every origin, so look to the key, the token, or the request itself.
CORS failures live on the tenant-credential routes, with the familiar symptom “curl works, the browser does not”:
- A preflight from an origin no tenant registered returns
404— not204, not403. The refusal falls through to the unroutedOPTIONS, and the console shows only the generic No ‘Access-Control-Allow-Origin’ header is present on the requested resource. - An empty allowlist and a missing origin look identical — both
404on a tenant-route preflight, with nothing in the response saying which. Telling them apart means readingtenants.json.
A scoped route, by contrast, answers a preflight from any origin with 204 and
Access-Control-Allow-Origin: * — GET /v1/health is the simplest place to
watch that happen.
Nothing is logged operator-side when an origin is refused, and the Origin
value is not recorded anywhere — so there is no log line to go and find. The
diagnosis is manual, and it is short: ask the developer for their exact origin
string, and compare it against the tenant’s array by eye, scheme, case and port
included. https://app.example.com and https://app.example.com:443 are
different strings to the matcher, even though a browser treats them as the same
page. Casing cannot be the cause here — an upper-case entry is refused at boot,
so a running service has none — but a default port written explicitly can be,
and that one is still accepted.
Next
- Get started — mint a token against the instance you just deployed
- Safe usage — handling the credentials you now issue
- Scopes and permissions — including why
exportis gated three separate ways - Errors — every code the service returns