Most of what I know about production systems came from problems nobody wrote down. A queue that drained fine in staging and backed up under a holiday load. A retrieval pipeline that answered confidently because the chunker had split the citation away from the claim. A migration that was correct in every test and wrong in every timezone.

Each of those cost an afternoon and was explained in a chat window that no longer exists. This blog is where I keep the explanation.

What belongs here

Three kinds of notes, roughly:

  • Failure modes. The shapes of bugs that survive code review: ordering assumptions, retries that multiply load, caches that hide the write path.
  • Design trade-offs. Why a pipeline is shaped the way it is, and what it would take to change it. Including the options that were rejected.
  • Delivery details. The unglamorous parts — how something is deployed, how it is observed, what the on-call person sees at 3am.

I am less interested in a tidy narrative than in the constraint that made a decision obvious in hindsight.

Why publish instead of keeping notes

Private notes are where ideas go to stay vague. A sentence you would never show another person can stay ambiguous for years, because nothing forces it to resolve. Publishing applies just enough pressure to make me finish the thought and check the claim.

It also makes the record searchable. Half the value of an incident postmortem is that the next person finds it before repeating the incident. A blog does that without anyone needing to be in the right channel at the right time.

On AI systems specifically

A lot of current writing about LLM features is either a demo or a warning. Neither is much help when you have to decide whether to cache a retrieval result, how to evaluate a change to a prompt, or what to do when the model returns something plausible and wrong.

Those are ordinary engineering questions with unfamiliar failure shapes. They deserve the same treatment as any other production system: measure, bound, degrade gracefully, and write down what happened when it did not.

What to expect

Posts will be uneven in length and probably in quality. Some will be a few paragraphs pinned to a specific incident. Some will be longer attempts to work out a design. I would rather publish the short version now than the polished version never.

If something here is wrong, or if you solved the same problem differently, email is on the contact page. Corrections are the best reason to put this on the internet at all.