The warm-pool rewrite cold starts forced in a four-day AI build

I’ve been building a product with real users, and this pipeline turns what they upload into playable audio streams. A load test against its Kubernetes executor caused enough memory pressure to OOM and take down nodes on the shared cluster. I could have reached for a node autoscaler — my provider wasn’t well set up for that, and I hadn’t put in the time to work around it — but what I wanted was a pipeline that could scale to zero, and for a second reason too: idle nodes standing around between bursts of traffic cost real money for no work done. That want produced the first design: a fresh container per job, torn down right after. I’d already put the executor behind an interface, so swapping Kubernetes for Cloudflare later was a contained change, not a rewrite of the calling API.

I scaffolded a new repository from a Cloudflare template, and four days later a single config value flipped a production API from Kubernetes Jobs to that new service. I’ve since fixed the OOM problem cluster-side too, but by then I was committed to the new pipeline, and its cold starts had become the next problem. Four days, two repos, two languages, and the architecture I shipped on day one wasn’t the one that reached production — real traffic rewrote it on day two.

I want to be specific about how this got built, because the pace is part of the story. I worked this the way I’ve been working most infrastructure builds lately: with an AI coding agent doing the actual typing, driven module by module through RED-then-GREEN TDD, while I reviewed diffs, made the architecture calls, and decided what to test next. That combination is the only reason four days was a real estimate and not a joke. Roughly 400 tests got written across two repos and two languages — 185 in the Rust container, 155 in the Worker and scheduler, 72 on the API-side executor — and every one of them exists because the cycle was: describe the behaviour, watch it fail, write the minimum to pass, move to the next module. An agent that can hold “what does RED look like for this module” in its head while typing at that speed changes what’s possible in four days. It also means you hit real-traffic edge cases inside day two instead of week three.

The two sides

The pipeline splits across two services. On the calling side, the API — TypeScript on Bun — has an executor interface, ITranscodingJobExecutor, selected by an environment variable at boot (TRANSCODING_EXECUTOR, one of bun | k8s | docker | stub | cloudflare, defaulting to k8s). The API doesn’t touch a container directly. It POSTs a signed job and forgets; results come back later as callbacks.

On the runner side is a new service: a Hono-based Cloudflare Worker, a scheduler Durable Object, and a pool of FFmpeg-running containers written in Rust. The flow, once both sides existed:

Two independent HMAC-SHA256 channels secure it — one API→Worker, one container→API callback — each with its own secret, both shaped t={timestamp},v1={base64} over "{timestamp}.{body}".

Building the pipeline module by module

Day one was scaffold-and-strip: pull the Cloudflare template, delete the boilerplate, then build the Rust pipeline in the order the audio actually moves through it — types, then config, then download, analyze, encrypt, transcode, upload, callback, worker orchestrator — each module RED before GREEN. The same day, the executor landed on the API side with unit tests for signature determinism, timestamp sensitivity, and base64 encoding, and staging got flipped over to the new executor through the config flag.

The pipeline itself, stage by stage, with the failure codes each stage reports on the way back to the API:

Progress callbacks fire at each stage, and the temp work directory gets cleaned up on completion.

Real traffic, day two

The next morning, staging put the design under real load, and it didn’t go cleanly: AWS SigV4 signing on the download step needed fixing; the download step moved from buffering the whole source file in RAM to streaming it to disk, since the first version worked on small fixtures and fell over on real file sizes; FFmpeg’s stderr had to stream instead of buffer; a multi-variant HLS encode was picking the wrong audio stream index; and bitrate fields had to convert to bits-per-second to match the API’s wire format.

The one I’d flag as the most interesting bug of the whole build: the EBU R128 loudness parser was matching the first I: (integrated loudness), LRA:, and Peak: values in FFmpeg’s output instead of the last one. FFmpeg emits these multiple times during a two-pass loudness analysis, and the first-pass numbers look completely plausible on their own — they’re real loudness values, just not the final ones. Nothing crashes. Nothing errors. You get a wrong number that looks exactly like a right number, and the only way to catch it is checking the output against a known-good reference, not trusting that a clean run means a correct one. That’s the kind of bug that ships silently in an unaudited system, and it surfaced only because staging was already taking real audio, not fixtures.

A retry backoff timer, sleepAfter, got retuned three times in the same day — 10 minutes, then 2 hours, then settled at 5 minutes — as I worked out how long a stuck job should sit before the scheduler gave up on it. Callback retry got exponential backoff, itself built RED then GREEN like everything else.

The reversal: cold starts killed the per-job model

I won’t smooth this into a tidy line: it wasn’t a refinement, it was a rewrite. The first design, shipped day one, spun up a fresh container per job — launch an instance, run it, tear it down. That’s wrong under real load: launching a Cloudflare Container costs a cold start, and a pipeline fielding jobs continuously was paying that cost on every single one.

A day later, that model got replaced, not patched, with a fixed warm pool: POOL_SIZE=8 reusable containers, each handling one job at a time, with a FIFO queue in front capped at MAX_QUEUE_SIZE=100. The Worker’s job changed shape along with it — it stopped being “launch an instance” and became “load-balance over a pool that’s already running.” Cancelling a job now kills the process group inside the container, not the container itself, so the container comes back into rotation immediately instead of costing a fresh cold start on the next job. Container instance sizing settled at standard-1, up to 32 instances.

The scheduler Durable Object took on real supervisory weight here: containers can’t report their own deaths. It watches for and reports DISPATCH_FAILED, JOB_TIMEOUT, CONTAINER_CRASHED, and JOB_LOST, with a hard MAX_JOB_DURATION_MINUTES=60 ceiling. Status reporting also went one-way: a getStatus() method on the executor deliberately always returns "unknown", because the real status lives in the database and gets written only by callbacks arriving from the pipeline — polling the container for its own state was never wired up, on purpose.

Container config took two attempts too: the first read configuration from Worker bindings lazily, fetching the value the first time something asked for it, which doesn’t survive how the Containers runtime actually initializes bindings. The fix was reading config once, eagerly, at construction time — the kind of thing only the real runtime teaches you, not a mental model of it.

One config field is worth calling out on its own: callbackProfile, set to "production" or "staging", selects which set of environment variables a container reads — separate storage buckets, callback URLs, CDN base, key base, and encryption flag per environment — so one Worker deployment serves both. It’s the only thing standing between a staging transcode job and a callback landing in the production API.

The gap that broke the first production deploy

The original design’s weakest point was callback delivery: if a container finished a job while the API was down, the result was lost silently, with nothing to retry it. That got fixed on both sides in a single day. On the Worker side, a thin /callback-events route now pushes into a Cloudflare Queue, consumed in batches of 25 with a 5-second batch timeout and up to 10 retries, delivered to a batch endpoint on the API. After max retries, a message lands in a dead-letter queue with about four days of retention for manual, redacted-log replay. The scheduler Durable Object drains its own outbox into the same queue rather than holding undelivered results in memory. On the API side, a new idempotent batch-ingest endpoint and worker process went in, with a design doc written alongside the code.

Then the first production deploy attempt broke anyway, for a reason unrelated to the code: the deploy tooling doesn’t create message queues on its own. It got documented and scripted before the next deploy, but the first attempt still ate the mistake in production.

A few smaller hardening passes came from review, not load: Worker errors truncate to 4096 characters so a bad upstream response can’t flood the logs; encryption keys get redacted from callback failure logs; 409 Conflict counts as success with alreadyExists: true, since resubmission is idempotent; and cancel() is best-effort — a non-OK response logs but never throws, and a 404 doesn’t even warrant a warning.

Cutover, and the plan for being wrong

Production cut over — one config value, TRANSCODING_EXECUTOR, moved from k8s to cloudflare. The old Kubernetes-Jobs executor is still in the codebase, still works, and reverting is the same single-value change in reverse. That was decided up front: if the new pipeline broke under production traffic the way the per-job model broke under staging traffic two days earlier, the way back out needed to be as cheap as the way in.

That’s the thing I keep coming back to about this build. The interesting failures weren’t in the parts anyone reviewed carefully on a whiteboard — HMAC schemes, queue architecture, the executor interface. They were in the parts that only show up once real jobs are moving through the system: a loudness parser reading the wrong occurrence of a log line, a container model that looks correct until you count how often it pays a cold-start tax, a queue that nobody remembered to provision. None of those show up in a design doc. All of them showed up within four days, because the pace was fast enough to reach real traffic before the design cooled into something nobody wanted to touch again. It started with an OOM crash under real load, and every rewrite since has been the same lesson: architecture that looks right on a whiteboard until real traffic says otherwise.

This is in production now, working as expected, but it isn’t the final form. I’d like to explore a different messaging facility for callback delivery, Kafka most likely. The justification already exists; I just don’t think it’s the right time to take on that complexity.