I once shipped a distributed job system where the single hardest bug had nothing to do with the jobs. The jobs ran fine. They finished, wrote their results, exited clean. The UI just never found out — it sat spinning until it gave up. Everything worked, and the user saw a failure. Those are the ones that stick with you.
Here's the story, because the fix taught me something I now reach for constantly: when a message can't get through, stop hardening the messenger and change the road it travels.
First, why it was async at all
The jobs were long — minutes, sometimes. The original design held one HTTP request open for the entire run: client asks, server runs it inline, response comes back when it's done. That works until it meets a load balancer, which idles connections out at ~60 seconds. The job kept running; the client got a gateway timeout. You've paid for the work and told the user it failed.
So I moved to submit-and-poll, which is the correct shape for anything that can outlive an idle timeout:
- —Submit returns
202 Acceptedwith ajob_id, immediately. - —The work runs on a background worker pulling from a queue.
- —The client polls a status endpoint every couple of seconds and only gets a result once the job is terminal.
That fixed the timeout and set up the real puzzle.
The completion signal that couldn't be delivered
The workers that ran the jobs were intentionally locked down — a sandbox running untrusted code, no credentials, no identity the rest of the platform would accept. When one finished, it had to announce "done, here's the output."
My first attempt was an internal HTTP callback with a shared secret so the credential-less worker could still authenticate. Sensible. It didn't work, and when I dug in, the reason was almost funny: the gateway in front of everything only forwarded a fixed set of URL path prefixes. The callback's path matched none of them, so it was dropped at the edge before it reached any handler. I'd carefully solved authentication for a request that was never going to arrive.
The unlock was to stop asking "how do I let this worker make an HTTP call?" and ask "what can this worker already reach?" It was already long-polling a queue for its work. So instead of calling home, it just puts a completion event on the same queue it reads from:
# on successful exit enqueue({ kind: "job_completion", job_id: JOB_ID, tenant_id: TENANT_ID, output_ref: OUTPUT_REF, })
And the worker's existing receive loop grows one branch:
def handle_message(msg):
ctx = restore_context(msg) # more on this below
if msg["kind"] == "job_completion":
return handle_completion(ctx, msg)
return dispatch(ctx, msg)
No new network surface. No new auth story. Completion rides the exact channel that dispatch already trusts. The fix wasn't clever code — it was noticing that the whole callback idea was fighting the network shape of the system, and the queue had been sitting there the entire time as the one road that was already open.
At-least-once: assume every message runs twice
The queue was a standard at-least-once queue, which means redelivery is normal — on a crash, on a lease expiry, sometimes for no reason you'll ever see. That redelivery is the point: it's what keeps work from vanishing when a worker dies. The price is that every handler has to be safe to run more than once.
def handle_completion(ctx, msg):
job = get_job_with_retry(ctx, msg["job_id"])
if job is None:
# read-after-write lag: the completion event
# beat the job's own record. DON'T ack — let
# redelivery try again once the write lands.
raise RetryLater()
if job.is_terminal(): # a duplicate delivery
cleanup_index(job.id) # idempotent
return ack(msg)
job.mark_completed(load_output(msg["output_ref"]))
save(job) # commit the effect...
return ack(msg) # ...THEN remove the message
Two things I'd underline for anyone building on an at-least-once queue:
The ack is sacred. You delete a message only after its effect is durably committed. If a duplicate shows up and the job's already terminal, do nothing and ack. And if you can't process it right now, do not ack it to make it go away — leaving it on the queue means it redelivers later, and the queue becomes your retry buffer for free. Acking away a message you failed to handle is how you silently lose data.
Read-after-write lag is real. The completion event and the job record are written by different code paths, and a distributed store doesn't guarantee the record is readable the instant the event is. So I retry the lookup a few times over a couple of seconds, and on a persistent miss I keep the message on the queue rather than acking it into the void. (I also lost an afternoon to a not-found error that didn't survive an RPC boundary as a typed error — it arrived as a plain string, so my is-not-found check returned false and quietly disabled the entire retry path. Match your sentinels by message when they cross a wire.)
One more multi-tenant trap
The data was partitioned by tenant, but the workers ran tenant-less — they're just draining a queue, no user context. A tenant-less reader looks in the default partition, doesn't find a job that lives in a customer's partition, and reports it missing. Records that exist but "don't exist."
The fix is to treat tenant as part of the message contract, not an ambient property: stamp it on the payload at submit time, and restore it before any read or write. The subtle part is remembering to do it on the worker's own re-enqueue paths too — retries, backoffs — or a redelivered message comes back tenant-less and loops forever against the wrong partition.
The lease you're holding without realizing it
Receiving a message doesn't remove it; it hides it from other consumers for a visibility window. That window is a lease: "mine for N seconds." Finish and delete, or crash and let it lapse so someone else takes over. If your work can outlast the window, you extend the lease on a heartbeat — otherwise the message reappears while you're still working on it and a second worker starts the same job.
I shipped this with an off-by-one-interval bug: the heartbeat waited a full tick before its first extension, but the initial visibility window was shorter than a tick. Long tasks let the lease lapse and got redelivered mid-run. The fix is trivial once you see it — extend once immediately, then on every tick:
def start_heartbeat(receipt):
extend = lambda: queue.change_visibility(receipt, LEASE_SECONDS)
extend() # up front, not after one interval
while not done:
sleep(HEARTBEAT_INTERVAL)
extend()
Reaping the dead so the living can run
Workers die mid-job — deploys, OOMs, cancellations — and leave a job stuck in Running forever. Harmless, until you have a concurrency limiter that counts running jobs to decide whether it can start another. Enough stuck-Running orphans and the limiter thinks it's permanently at capacity: the whole dispatcher deadlocks and nothing new ever starts.
The cure is a reaper that runs before the capacity check, fails any job whose last heartbeat/update is older than a generous ceiling (marked failed with a reason, not deleted — keep the audit trail), and then invalidates whatever cached count the limiter reads. Reap and un-throttle on the same pass, or the freed slot stays invisible until the next cycle.
The shape, end to end
What to steal from this
- —Anything that can outlive a load-balancer idle timeout should be submit-and-poll.
202+ a status endpoint beats a held connection. A held request doesn't remove the failure; it relocates it somewhere worse. - —When a message won't route, change the channel, not the messenger. Deliver over infrastructure the producer already reaches. My locked-down worker couldn't make the HTTP call — but it could always talk to the queue it was already reading.
- —At-least-once means idempotent handlers and a sacred ack. Duplicate → clean up and ack. Can't process it → don't ack; let redelivery retry. The queue is your retry buffer; acking to silence a message you failed on is how data disappears.
- —If you partition by tenant but run workers without a tenant, put the tenant in the message — on submit and every re-enqueue — or you'll chase ghosts in a partition nobody queries.
- —If a limiter counts in-flight work, reap the dead before you count — and invalidate the cache the limiter reads, or a handful of zombies will deadlock everything.
The meta-lesson I keep coming back to: I spent real effort making a callback authenticate before I checked whether it could even arrive. Verify the road exists before you pave it.