The worst bugs I've hit weren't the ones that crashed. Crashes are honest — they page you, they leave a stack trace, they demand attention. The worst ones were features I'd finished: written, reviewed, merged, deployed. The system reported success. The dashboard stayed green. And the thing the feature existed to do simply never happened.
I've collected a few of these across different jobs and layers — a full disk, a browser stream, an RPC across a process boundary, a shell-out — and they all rhyme. In every case there was no error to find, because the error had already been made to disappear. Here are four, each as symptom → wrong guess → root cause → fix → lesson. The pattern's at the bottom.
1. The job that "succeeded" and produced nothing
A batch job that generated a large output file started coming back with exit code 0, no error, and no file. Just gone.
My first guess was the generator itself — some input it couldn't handle. I went input-diffing. Wrong tree entirely.
It was a full disk. The job wrote its main output fine, but the next write — a small metadata file — hit ENOSPC. And the error handler looked like this:
def report_failure(exc):
try:
write_log(f"failed: {exc}") # writes to the full disk...
set_status("failed")
except Exception:
pass # ...and this eats the second ENOSPC
def main():
try:
do_work()
except Exception as exc:
report_failure(exc) # throws ENOSPC in here, swallowed
sys.exit(0) # reports success
The failure reporter tried to write to the disk that was full, threw its own ENOSPC, and except: pass swallowed it. The process exited 0 with status stuck at its default. Whatever ran downstream saw "done, no problem" and moved on. A completed job's output, silently discarded.
(The disk had filled for its own dumb reason — a path pattern that was quietly scooping up every previous run's leftovers into each new run, growing a little each time until it tipped over. But that's a footnote; the cover-up was the real bug.)
The fix had two layers. Immediately: reorder so the important output is written before any fragile bookkeeping, and give the disk headroom. Properly: stop hoarding old runs' data. But the lesson is narrower and meaner — a try/except: pass inside the error handler blinds the exact code path that only runs when things are already broken.
2. The stream that was secretly dead
A UI had a live progress indicator fed by a long-lived Server-Sent Events stream. The design was "stream first, fall back to polling if the stream errors." Reasonable. The indicator would update for a while, then freeze — while the backend job finished perfectly.
My first guess: we're not emitting enough events. The events were fine. The stream was dead — it just hadn't told anyone.
Here's the thing about EventSource: a connection that drops mid-stream, especially a half-open TCP connection sitting behind a proxy or load balancer that timed it out, does not reliably fire the error handler. To the browser, "the connection quietly stopped delivering bytes" is nearly indistinguishable from "the server is just being quiet right now." So:
const es = new EventSource(url);
es.onmessage = applyProgress;
es.onerror = () => fallBackToPolling(); // never fires on a silent half-drop
onerror never fires → the fallback never triggers → the UI waits forever for the next message on a connection that's already gone. Our whole "poll fallback" existed and had never once run, because the condition it waited for wasn't observable.
The fix: make it poll-primary. Polling has no long-lived connection to silently die — you just re-ask on an interval, and a couple seconds of latency is invisible for a coarse progress bar. (Where you genuinely need push, you can fix the transport — sane proxy idle timeouts, a real heartbeat event, and a client watchdog that falls back on stall, not just on error. But default to the design with no silent-failure mode.)
Lesson: "detect the failure and fall back" is only as good as your ability to detect the failure. If the failure is invisible, your fallback is decorative.
3. The call that no-op'd because it used the wrong transport
Two services talked across a process boundary through a small broker. One direction's calls — status updates, some control messages — produced nothing at all. And yet the main work still completed, which made it look flaky and intermittent and maddening.
My first guess: the broker's racy. I went looking for a concurrency bug in the broker. There wasn't one.
The client picked its transport from an environment variable holding a socket path. In one deployment that var was set and everything worked. In another — a more isolated runtime — the var was unset, and the client did this:
def call_broker(tool, payload):
sock_path = os.environ.get("BROKER_SOCKET", "")
if sock_path:
return unix_call(sock_path, tool, payload)
# no else — falls through and returns None
Empty path → skip the call → return nothing. Every call in that environment was a silent no-op. The tell, once I knew to look: the broker logged that it started every run but recorded zero incoming requests. Not some — zero. The main work still finished only because it went over a different channel; the broker-dependent stuff was the only thing that was 100% dead, and it happened to be the least visible.
The fix was to add the other transport branch and use it when the socket path is empty — connect over the runtime's actual control channel instead of falling through.
Lesson: an if configured: do it with no else is a silent no-op waiting for a new environment. If a call is mandatory somewhere, make its absence loud — throw, or at minimum log — don't quietly skip it.
4. The shell-out that returned empty on the server but worked by hand
A service shelled out to a command, parsed the output, and used it. On the server it kept coming back empty. SSH in, run the identical command by hand: works instantly, prints exactly what's expected.
My first guess: timing — the thing it was looking for wasn't ready yet. It was ready the whole time.
The code captured only stdout and treated any problem as an empty result:
out = subprocess.run(cmd, capture_output=True).stdout.decode()
return parse(out) # "" whether the command errored or genuinely matched nothing
So "the command failed" and "the command ran fine and found nothing" collapsed into the same empty string — and they need opposite responses. The moment I captured stderr and split them into two distinct log lines — one for a non-zero exit (with stderr), one for a clean run with no match — the answer was right there. It was an environment difference between the service's process and my interactive shell: a PATH/permissions delta, not a missing target. Invisible as long as errors and empty-results looked identical.
Lesson: when a shell-out returns empty, capture stderr first, and never let "it errored" and "it succeeded with nothing" be the same value.
What to steal from this
Four layers, one disease: successful nothing. The transferable rules:
- —Never let a catch-all eat an error — least of all in the error handler.
except: pass(and its cousins in every language) around your reporting path is how a disk-full turns into a green checkmark. The code that runs when things break should be the most observable code you own. - —
exit 0is a claim, not a fact. Verify the artifact — the file exists, the row landed, the event arrived — not the return value. Every one of these bugs passed its exit-code check. - —Prefer failure modes you can see. A stream that silently half-drops, an env-gated call that falls through, a shell-out that returns
""on error: all "successful nothing." Given a choice, pick the design whose failure is loud — poll instead of a droppable stream, throw instead of fall through, two log lines instead of one empty string. - —Don't build fallbacks on signals you can't trust. Half of these were fallback logic waiting on a signal that never arrived. A fallback is only as good as the failure detection under it.
Distrust green dashboards. The best question you can ask a "successful" system is a rude one: prove it actually did the thing.