On our public launch morning — August 18, a few hours before the announcement was supposed to go out — the rendering pipeline did something worse than crash. By every signal we had, it ran perfectly. It rendered nothing at all.

Containers: up. Worker heartbeat: green. Database: healthy. The API accepted requests, queued jobs... and returned 504s, because no job was ever picked up. Every render sat in the queue in created state, forever, while the worker supervising that queue kept reporting that things were going great.

This is the postmortem. The bug sits three layers deep. Each layer looks reasonable on its own, and each one waved the problem through without a sound.

The symptom

Our launch-day smoke test was deliberately boring: hit the production API, render one real page, look at the picture. The picture never came — the request died at the gateway with a 504.

First guess: the renderer was crashing on that particular URL. But the logs showed no render had even started. docker compose ps said everything was Up. The heartbeat was beating. It was the queue table that told the truth: jobs piling up in pg-boss created state, zero transitions to active. The worker had become a spectator at its own job queue.

Layer one: Compose passes '' where you expect nothing

compose.yml forwards an env var so worker concurrency can be tuned on the server without rebuilding the image:

environment:
  - WORKER_CONCURRENCY=${WORKER_CONCURRENCY:-}

That :- default means: when the server's .env doesn't set the variable, pass it through as an empty string. Not unset. Empty. The distinction feels academic right up until it becomes the entire incident.

Layer two: ?? doesn't catch what you think it catches

The worker read the value like this:

const CONCURRENCY = Number(process.env.WORKER_CONCURRENCY ?? 2)

Nullish coalescing handles exactly two values — null and undefined. An empty string is neither. So '' ?? 2 evaluates to '', and the fallback that existed specifically to protect this line stepped politely out of the way. Had Compose passed nothing at all, ?? would have caught it. Compose passed the other kind of nothing.

Layer three: Number('') === 0

The JavaScript coercion table nobody has memorized at the moment it matters:

Number(undefined)  // NaN  ← would have blown up loudly downstream
Number('nonsense') // NaN  ← same
Number('')         // 0    ← ...oh no

Number('') is zero. Not NaN — which would have poisoned every comparison and almost certainly crashed startup — but a clean, plausible integer that sailed into pg-boss as "run this worker with concurrency 0." pg-boss took it without comment. Zero is a number, and nobody said it couldn't be this one. The worker registered, started its heartbeat, and settled in to consume exactly nothing, precisely as configured.

Three layers, then. Compose turned "not set" into ''. The ?? guard let '' through. And Number('') produced the one wrong value that doesn't crash anything.

Why every monitor lied

The heartbeat did its job — the job we gave it. It proved the worker process was alive: event loop turning, database reachable, timer firing. All true. The process was in excellent health while performing zero units of its actual purpose.

Inside this specific bug hides the general lesson: liveness is not throughput. "Container is Up" and "heartbeat is green" are claims that a process exists. They say nothing about whether work flows through it. Our monitoring answered "is it running?" — the question that mattered was "is it working?"

The fix

Once the cause was visible, the code fix took five minutes. Parse defensively; treat empty, invalid, and non-positive as one category meaning "use the default":

const rawConcurrency = Number(process.env.WORKER_CONCURRENCY || 2)
const CONCURRENCY =
  Number.isInteger(rawConcurrency) && rawConcurrency >= 1
    ? rawConcurrency
    : 2

Note the || where ?? used to be — falsiness is the correct test here, since '' and 0 are both values we refuse. The integer and range check then turns away '3.5' and '-1' for good measure.

The process fix is the one that matters. Our launch checklist now carries a line that would have caught this an hour earlier — and has caught one other issue since: the smoke test must render something real, end to end, and a human must look at the image. Not health endpoints. Not container status. A real render. "Up" is a rumor; output is a fact.

Lessons, in the order we'd hand them to past us

  1. Parse every numeric env var like it's hostile. Empty strings, whitespace, floats, negatives — pick the policy at the boundary and fall back loudly or safely. Never accidentally.
  2. ?? guards against unset; || guards against empty. Know which nothing you're defending against. Compose's ${VAR:-} hands you the second kind.
  3. Liveness is not throughput. A monitor that can stay green while business output is zero measures the wrong thing. At least one check must assert that work happened, not that a process exists.
  4. Smoke tests exercise the money path. Ours now does: one real URL, one real render, one human looking at one actual image before any announcement goes anywhere.

We mentioned elsewhere that we do this full time and still get humbled — this was the incident we meant. The renderer itself never missed a beat that morning. The bug lived in the two characters between "not set" and "set to nothing", and it cost a launch-morning hour we'd happily pay again for the checklist line it bought.

Running headless browser infrastructure yourself? The SSRF post covers the security side of the same discipline — and if you'd rather not run it at all, that argument is here.