← writing

Streaming · Go

Why my LLM stream died at exactly 45 seconds — and why I eventually stopped streaming

I spent a good chunk of a year building the streaming layer for an LLM app — the part that pushes tokens and progress from a Go backend to a browser while a model thinks. I went in believing Server-Sent Events were obviously correct for the job. I still think they're the right shape. But two production bugs rearranged my priorities, and the second one ended with me deleting the streaming code and shipping a polling loop instead. Here's the whole story, because the lessons transfer to any long-lived HTTP stream.

Pick the right transport for the shape of your data

When you're pushing model output to a browser, the data flow is one-directional: server → client, token after token, plus the occasional progress event. You are not having a conversation over the wire in both directions at once. That shape has a natural fit.

WebSocket is full-duplex and a different protocol with an upgrade handshake, its own load-balancer headaches, and no built-in reconnect or framing — everything you'd hand-roll. SSE is just an HTTP GET whose body never ends: Content-Type: text/event-stream, write UTF-8 events, flush after each one. The browser consumes it with EventSource, which gives you automatic reconnection and event IDs for free. For one-directional streaming, it's the least machinery. So I used it.

The 45-second guillotine

The bug reported itself with unnerving precision: only long, tool-heavy requests failed, always at about 45 seconds, always with context canceled in the logs — never deadline exceeded.

That one-word difference was the whole investigation. context canceled means something upstream killed my request's context. deadline exceeded would mean my own timeout fired. My app-level timeout was five minutes, so this wasn't me. Something external was cancelling the request at 45 seconds flat.

It was Go's http.Server.WriteTimeout, and it's a trap because it doesn't mean what it sounds like:

srv := &http.Server{
    ReadTimeout: 10 * time.Second,
    // Resets on activity, only covers idle time BETWEEN requests. Not the bug.
    IdleTimeout: 120 * time.Second,
    // A whole-response, wall-clock cap. Starts when the request is read and
    // does NOT reset when you write. For a stream, it's a fuse on a timer.
    WriteTimeout: 45 * time.Second,
}

WriteTimeout is not "abort if one write blocks." It's a fixed budget for the entire response, and it does not reset when you flush bytes. IdleTimeout is the one that resets on activity — and it only governs idle keep-alive between requests, not a response in flight. For a normal handler that returns in milliseconds you'll never notice. For a stream meant to live for minutes, WriteTimeout chops it off at the deadline regardless of how much data is actively flowing.

When it fired, the server cancelled the request context, and that cancellation cascaded down through every layer that had derived a context from it — the stream loop, the model client, the in-flight read of the model's event stream — all of them unwound at once. context canceled, not deadline exceeded. Exactly consistent with a parent cancel.

The fix is per-handler, not global

The tempting fix is to raise the global WriteTimeout to something huge. Don't — that removes the protection from every ordinary route to rescue a couple of streaming ones. Since Go 1.20 there's a targeted tool: http.ResponseController reaches the underlying connection so you can clear the deadline for just the response you're serving.

func streamHandler(w http.ResponseWriter, r *http.Request) {
    flusher, ok := w.(http.Flusher)
    if !ok {
        http.Error(w, "streaming unsupported", http.StatusInternalServerError)
        return
    }

    // Clear the write deadline for THIS response only. The zero time means
    // "no deadline." WriteTimeout never resets on writes, so leaving it set
    // guillotines any stream that outlives it.
    rc := http.NewResponseController(w)
    if err := rc.SetWriteDeadline(time.Time{}); err != nil {
        log.Printf("could not clear write deadline: %v", err) // best-effort
    }

    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("X-Accel-Buffering", "no") // don't let the reverse proxy buffer us
    w.WriteHeader(http.StatusOK)

    ctx := r.Context()
    tick := time.NewTicker(15 * time.Second)
    defer tick.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-tick.C:
            fmt.Fprint(w, ": ping\n\n") // heartbeat comment; keeps bytes moving
            flusher.Flush()
        case tok, open := <-tokens:
            if !open {
                return
            }
            fmt.Fprintf(w, "data: %s\n\n", tok)
            flusher.Flush()
        }
    }
}

One gotcha: ResponseController can only clear the deadline if it can Unwrap() its way down to the real connection. Most routers wrap ResponseWriter; the good ones implement Unwrap() so the controller walks the chain. If yours doesn't, you get ErrNotSupported back — which is why I check the error instead of assuming it worked. After the fix, liveness was still bounded by the right things: my five-minute application context and the heartbeat, not a transport-layer fuse that has no idea whether a stream is legitimately in progress.

The harder lesson: EventSource won't tell you the stream died

That fixed the token stream. A sibling feature — streaming progress for a multi-minute background job — taught me something that reversed my whole position on streaming.

Through the production proxy chain, the long-lived SSE stream dropped silently a couple of minutes in. I'd built it defensively: stream first, fall back to polling on error. The fallback never fired, and I learned why the hard way:

EventSource does not reliably fire its error event on a half-open drop. Clean teardown, sure. But when an intermediary quietly stops forwarding bytes while the socket still looks alive, EventSource just sits there — no error, no reconnect, no signal at all. My fallback was waiting on an event that never came, so the UI froze: no progress, no error, no result, even though the job had finished.

I did the standard hardening first — a 15-second heartbeat so bytes keep flowing under the proxy's idle timeout, X-Accel-Buffering: no so the proxy streams instead of buffering everything into one burst at the end, the right content type. All necessary. None sufficient. The stream still died through that chain.

So I deleted the streaming client and went poll-primary:

async function pollJob(id, onProgress, signal) {
  while (!signal.aborted) {
    const res = await fetch(`/api/jobs/${id}`);
    const job = await res.json();
    if (job.progress) onProgress(job.progress); // each poll advances the step list
    if (job.state === "done")  return job.result;
    if (job.state === "error") throw new Error(job.error);
    await new Promise(r => setTimeout(r, 2000));
  }
}

The move that kept it feeling live: the status endpoint returns the latest progress event on every poll, and both the old stream and the new poll feed the same monotonic step reducer — so progress only ever moves forward, and a 2-second poll advances the visible steps just fine. Polling has no long-lived connection, so there's nothing to half-drop and no phantom-error to wait on. Through a proxy chain that buffers and silently kills long responses, the boring transport was simply the reliable one.

What to steal from this

• —Learn your server's timeouts precisely. In Go, WriteTimeout is a whole-response wall-clock cap that never resets on writes; IdleTimeout is the one that resets, and it only covers idle time between requests. For streaming handlers, clear the write deadline per handler with http.NewResponseController(w).SetWriteDeadline(time.Time{}) and keep your real upper bound at the application layer.
• —context canceled ≠ deadline exceeded. Canceled means a parent killed you; exceeded means your own timer fired. Reading which one you got tells you where to look before you open a single file.
• —Never build a fallback that depends on EventSource.onerror. It may not fire on a half-drop. If "fall back on error" is your reliability story, it has a hole; add a heartbeat and an independent liveness timer, or don't rely on the fallback.
• —SSE is the right shape for one-way streaming, but the transport has to survive your real network. Set X-Accel-Buffering: no, send heartbeats — and if a proxy chain still buffers or silently kills the stream, make polling primary and fold both paths onto one monotonic progress model. Sometimes the least impressive transport is the one that actually works.