Gotchas worth remembering

Things that cost real debugging time here, kept so they cost it once. This is engineering knowledge about this codebase — not a style guide, and not a list of everything that could go wrong. If something bit you and the cause was non-obvious, it belongs here.

Testing has its own set: see Testing, which covers mutation-testing every guard and the fixture traps that come with a real Docker daemon.

API and data shapes

• decodeJSON uses DisallowUnknownFields() (internal/api/respond.go), so a request body must contain only struct-declared fields. Read-only fields such as hasPassword have to be stripped client-side — see smtpPayload / ldapPayload for the pattern.
• Image and object refs contain : and /, so pass them as query params, never as chi path segments: chi will not decode %3A.
• Go nil slices marshal to JSON null, not []. The SPA then crashes on x.length / .map. Initialise API-returned slices (make / []T{}) so empty means [], and still guard with ?? [] on the TypeScript side. This bit us via ResourceOverview.Containers when no containers were running.
• ORDER BY cannot be a bound parameter. Anything sortable from the UI must map its sort key through a fixed whitelist (see AlertQuery.orderBy), or you are building SQL out of a query string.
• time.Duration is int64 nanoseconds, so days * 24 * time.Hour overflows above ~106,751 days. Asking for a 200,000-day token expiry produced a date in 1989 — a credential dead the moment it was issued — and larger values wrapped to arbitrary dates. Validate the number of days against a ceiling before it ever becomes a Duration.
• SQLite LIKE has no default escape character. Escaping % and _ only works together with an explicit ESCAPE '\' clause — without it the backslashes are matched literally and the filter silently searches for the wrong string.

Docker behaviour

• Reading a Dockerfile line by line gets it wrong. A # escape= directive changes the line-continuation character, and a heredoc body (COPY <<EOF … EOF) can contain a line that looks exactly like FROM. A hand-rolled reader that picks the registries a build needs missed both and sent the build’s daemon a credential it had no use for. BuildRegistries therefore uses BuildKit’s own parser (moby/buildkit/frontend/dockerfile/parser), the one dependency in the tree that’s there for one function. It is the only importer, so it can be swapped for a smaller parser later: the table tests in dockerfile_registries_test.go and the header pentest in build_auth_pentest_test.go define the behaviour a replacement must keep.

• docker stats CPU is per-core: 100% is one core. A container busy on four cores reads ~400%, so any fixed threshold or dashboard built on it is wrong on a multi-core host unless divided by the core count. The engine exposes the count as CPUCores for exactly this.

• docker compose up -d builds only a missing image. Without --build, the second deploy after editing a Dockerfile or its context silently keeps running the old image and reports Container Running.

• A build context is uploaded by Docker itself; a bind mount is not. That is why remote deploys need bind seeding but nothing for build:. Conflating the two put a phantom item on the roadmap for months.

• Alert-rule cooldown: docker stop emits several events (kill → die → stop), so a 1s cooldown can double-fire. Defaults are 60s.

• Container network stats are cumulative counters that reset on recreate. Store the raw counters and derive rates at read time; a delta computed across a reset is negative, and rendering it produces either a phantom spike or a negative rate. applyNetRates skips the delta when the new counter is lower than the previous one, so a reset reads as a gap at zero.

• /containers/{id}/stats is keyed by interface, not by network. The API’s endpoint_id field exists but the daemon fills it in on Windows only, which is why docker stats itself shows a single aggregate NET I/O column. Sum across interfaces and say so; anything per-network needs MAC/namespace inspection via netlink (Linux-only, hostile to remote hosts).

• On older engines /containers/json lags inspect. After stopping a stack, Engine 24 reported a container as running for ~250 ms while inspect already said exited. The app polls, so it is only a problem for tests that assume instantaneous consistency.

• COMPOSE_PROFILES has three sources, and they don’t merge the way you’d guess. Verified empirically against a real docker compose up (Compose v5.4.0): when --profile is passed on the command line at all, Compose uses only the --profile flags and ignores COMPOSE_PROFILES entirely — no union. COMPOSE_PROFILES (from the shell/process environment, or the project’s own .env file, which compose auto-loads from its working directory) only takes effect when no --profile flag is given at all — exactly the “no profiles selected” deploy case. And between the two COMPOSE_PROFILES sources, process/shell env wins over .env: setting it to the empty string in the subprocess’s env (last entry wins in exec.Cmd.Env) suppresses a project’s own .env setting too, it isn’t only a shell-level override. ComposeUpFiles relies on exactly this: it always sets COMPOSE_PROFILES= on the subprocess so its explicit --profile list is the sole source, closing the gap where an operator’s env or a project’s .env could silently activate more than what gets persisted as “last deployed profiles”.

• A non-streaming ContainerStats call needs IncludePreviousSample, or CPU is a lifetime average. The moby client sends one-shot=true unless you set it. The daemon then returns an empty precpu_stats, and the usual (cpuDelta / sysDelta) * cpus * 100 is taken against zero. A container using four cores shows as about 0.1%. Memory and network look fine, so it is easy to miss. The old SDK call took the previous sample by default; the migration (#256) lost that. The test in internal/docker/stats_sample_test.go fakes a daemon that answers one-shot requests the way the real one does.

HTTP timeouts and streaming

• A handler runs from the moment the HEADERS arrive, not the body. With no ReadTimeout a client can send headers and dribble the body indefinitely, holding a handler open — and any check the handler makes before reading the body is made against a state the client can then outlive. That is why the authorisation for pairing a second factor lives in the INSERT rather than in a check above it.
• Re-arming a read deadline after the body ends cancels the request. When a handler finishes reading, net/http clears the deadline itself and starts a background read to notice the client leaving (startBackgroundRead); any failure of that read is taken as “client gone” and cancels the request context. With Content-Length, the final read returns bytes and io.EOF together, so a naive “extend on n > 0” re-arms it at exactly the wrong moment. Every long docker build died two minutes after its context finished uploading.
• Clearing that deadline on any error is the opposite mistake. If the handler returns without draining the body, net/http drains the remainder inline, inside chunkWriter.writeHeader, before the response headers go out — unbounded if the deadline is gone. A client that declares a body, sends two bytes and goes quiet without closing pins a goroutine and an fd for the life of the process, and never receives the timeout the handler wrote. Clear on io.EOF only.
• Connection: close skips that drain, so a reproduction that closes the connection will not show the bug. Test stalled-body behaviour over keep-alive.
• Hijacked connections are off the clock: net/http clears deadlines on hijack, so WebSocket streams are unaffected by ReadTimeout — but an idle keep-alive connection now closes after it, because Server.idleTimeout() falls back to ReadTimeout when IdleTimeout is unset.

Authorization

• A sequential id is not an access control. Any endpoint or tool that takes an integer id (project, alert, container-metrics series) must resolve the host that record belongs to and authorize against it — checking the section alone leaves the id space walkable, since ids are consecutive. Three MCP tools shipped checking their section against host 0 while acting on someone else’s record. The tell is machine-detectable: an integer id argument with no host_id.
• A missing record and an out-of-reach one must answer identically. Otherwise the error itself confirms the record exists, and the endpoint becomes a way to map what runs on hosts you cannot see.
• Over-tightening is a bug too, not a safe default. Host reach is derived from grants across all sections, so also demanding the hosts section for a host-scoped alert route hid alerts from users who legitimately reached that host. Guards need a test for what they must still allow.

Deployment

• systemd ProtectHome=true breaks docker compose plugin discovery. The shipped unit runs as a dedicated user with ProtectHome=true, which makes that user’s home inaccessible (EACCES, not ENOENT). The docker CLI bases plugin discovery on its config dir (~/.docker) and treats EACCES there as fatal, so docker compose reads as an unknown command: ComposeAvailable() returns false and Projects’ Deploy/Down are disabled. It works in a shell and not under the service, which is what makes it confusing. Fixed by Environment=DOCKER_CONFIG=/var/lib/dockercmd/.docker in deploy/dockercmd.service; the install-* scripts cover each OS.

• The running server cannot obtain a certificate through a local Pebble instance — two independent, stacked reasons. DC_ACME_DIRECTORY_URL pointed at Pebble (Let’s Encrypt’s own ACME test server) does not work as an end-to-end way to exercise this app’s real HTTPS-serving path, unlike Let’s Encrypt’s staging directory, which does (its own API endpoint has an ordinary trusted TLS cert; Pebble’s is locally-generated and untrusted).

  1. Directory discovery fails closed. internal/acme.NewManager never sets a custom trust root or InsecureSkipVerify on the *acme.Client it builds — correctly, since that client also targets the real Let’s Encrypt directory by default. Against Pebble’s self-signed cert this means the very first request fails: tls: failed to verify certificate: x509: certificate signed by unknown authority. Confirmed by actually starting the server with DC_ACME_DIRECTORY_URL pointed at a local Pebble and watching the handshake fail this way.
  2. Even past that, autocert.Manager.GetCertificate’s order-polling breaks against Pebble specifically. Post "": unsupported protocol scheme "": its internal CreateOrderCert polls the order via the URI from the finalize response’s Location header, but Pebble doesn’t repeat Location on that response (only the original order-creation response does, and RFC 8555 §7.4 doesn’t require it to). The finalize POST and the actual certificate issuance still succeed server-side (visible in Pebble’s own logs: “Issued certificate…”) — only the client’s subsequent poll breaks, 100% reproducibly, independent of PEBBLE_VA_ALWAYS_VALID/PEBBLE_VA_NOSLEEP. Real Let’s Encrypt (Boulder, staging included) doesn’t hit this — if it did, it would break autocert for effectively everyone.

  Both are specific to Pebble, and neither is a bug in this app to fix: (1) is internal/acme correctly refusing an unverifiable server, the same as it should; (2) is Pebble’s own response shape, not something a caller of autocert controls. internal/acme/pebble_integration_test.go still uses Pebble — but only for developing this package itself, driving the lower-level acme.Client methods directly (with an explicit, test-only InsecureSkipVerify client) and polling the order via its own real URI (known from AuthorizeOrder, unaffected by gap 2) instead of going through the convenience wrapper the real server relies on. Document -acme-directory-url as staging-or-a-real-alternate-CA only; don’t point a user at Pebble for it.

Secrets

• net/http transport errors embed the full request URL. *url.Error’s message includes it, so logging or storing err.Error() from a failed webhook call writes any token in that URL wherever the error goes. Report the wrapped cause instead (redactURL in internal/monitor/webhook.go).
• A backup archive is equivalent to the plaintext of every stored secret, because the at-rest encryption key is a row in the same database. Hence 0600 and the optional passphrase.

Destructive operations

• NEVER call a host-global prune (Prune{Networks,Images,Volumes}) from an integration test. The integration tests run against the developer’s real local daemon, and a global prune removes EVERY unused network / image / volume — not just the test’s. This is exactly how a network lifecycle test once wiped a developer’s networks. Test create / connect / remove on test-owned resources only, and exercise prune by hand on a throwaway host.
• A plain file write follows a symlink at the destination. Anywhere the app writes next to a user-controlled path, write to a temp file and rename — rename replaces the link instead of writing through it.

Frontend build

• A local make ui rebuild of web/dist is not guaranteed to match CI’s. Same web/src, same web/package-lock.json, same Node/npm version (checked by installing CI’s exact node -v via nvm) — and the committed CSS still came out smaller locally than CI’s own rebuild: CI’s included 10 extra Tailwind v4 utility classes (transform, transition, underline, blur, isolate, list-item, invert, ease-in/ease-in-out/ease-out) that a local rebuild pruned. Root cause not fully pinned down — suspected is Tailwind v4’s automatic content-candidate scanning behaving differently based on something environment-specific beyond Node/npm version — but the practical fix is procedural: when a web/dist gate failure looks surprising, trust CI’s own rebuild over a local one. Download it (gh run download <run-id> -n <artifact> after a temporary actions/upload-artifact step on web/dist, or just let a pushed branch’s own CI run be the thing you commit from) rather than assuming your local build is right and re-pushing it repeatedly.
• ci.yml‘s push trigger only covered main+release/v*; the web/dist gate had never actually run against several already-merged commits on release/v1.6.1. Every individual PR’s pull_request-triggered check validated that PR’s own diff correctly, but the merge commits combining them (a manual conflict-resolution merge, a forwarded Dependabot merge) were never independently re-checked until push-triggered CI was extended to release/v* — which is exactly when the above discrepancy surfaced. A merge commit needs its own CI run, not just trust in its parents’ green checks.