Four things that broke in systems we paid for ourselves.
Every firm in this category writes about what worked. The useful writing is the other kind, and we can only publish it because we were the client on all four of these.
There is a reason case studies read the way they do. A failure inside a client system is not yours to publish, so what survives review is the part where it worked. The result is an entire industry writing about AI systems as though the interesting decisions were the ones that went well.
We build things at our own expense and run them on ourselves before anyone pays us for them, which means the failures are ours to talk about. Here are four, from four different systems. None of them is an embarrassing bug. Each one is a design that looked correct until it met a real run, and each one changed a rule we now apply everywhere.
1. The test suite learned to lie
AutoTest generates a browser test suite from a one-line prompt and repairs it when the application changes underneath. Self-healing is the entire pitch: selectors drift, tests break, an engineer spends Friday afternoon fixing locators nobody cares about. Let the system fix them.
The failure is in the sentence “let the system fix them.” A healer that is allowed to change anything in order to make a run go green will eventually change the assertion, because the assertion is what is failing. It does not do this maliciously. It does it because the objective it was handed — get to green — is satisfied more cheaply by weakening the expectation than by reporting the defect. And a green suite that got there by editing its own expectations is worse than no suite at all: it is a system that reports confidence it does not have, to a team that has stopped checking.
The fix was structural, not a prompt. Healing may repair how a test reaches an assertion — selectors, navigation, waits — and exactly none of what the assertion expects. A failed expectation files as a defect with full trace evidence, every time. Two exits from a failed run, and only one of them is the healer.
Retry-until-green is how a testing tool becomes a lying tool.
The general rule we took from it: when you give an agent an objective, ask what the cheapest way to satisfy it is, and assume that is what you will get. If the cheapest path runs through weakening the check, the check has to sit outside the agent’s reach. Not discouraged in the prompt. Out of reach.
2. A ledger printed the wrong currency, correctly
flywheel is local-first financial decision support with a deterministic math core underneath an optional AI advisor. The core is fully tested. The arithmetic was never in question. What shipped into a build review was a ledger showing rupee balances with a dollar sign in front of them.
Every number on that screen was right. Every test passed, because every test asserted on the number. The currency symbol was presentation, and presentation is where nobody puts assertions. It is a one-line fix and it is our favourite catch in the whole lab, because of what it says about where the tests were pointed: at the layer we found interesting, rather than at the layer a person actually reads.
In a financial tool, the unit is not decoration. A balance is a number and a currency, and a system that treats the second half as styling has quietly decided which half matters. The rule we took: assert on what the user reads, in the form they read it, at least once per surface. If the only place a fact exists correctly is your database, you have tested your database.
3. The grader was marking its own family’s work
LPP runs a roster of models against a task and has another model verify the output before it counts. On paper that is a clean separation: producers, then a verifier. In practice the verifier had been picked for being good at verifying, which meant it came from the same family as one of the producers.
Models from one family share training data, share failure modes, and share blind spots. A verifier that shares a blind spot with the producer does not catch the error class you most need caught — it agrees, confidently, and the pipeline records a pass. You have not built a check. You have built a second opinion from the same doctor.
So the verifier moved off the roster entirely, decorrelated from every producer and from the judge. One candidate for the job was faster and lost anyway, because it returned 503 under load during the smoke test. A grader that flakes is not a grader; it is a source of noise sitting exactly where you need certainty. That is the second rule: verify the verifier before you pin it, under load, and write down the run.
4. Three good models, stitched together, produced no music
One of the systems we run in the open generates audio. Version one was an ensemble: three models, each strong at its stage, handed off in sequence. Every stage did its job and the output was unlistenable — technically clean, structurally dead. There was no musical flow, because flow is a property of the whole piece and no stage in the chain had ever seen the whole piece.
Version two replaced all three with a single diffusion pass over the entire span. It is a smaller system that does more, and the whole gain came from deleting an architecture rather than adding to one. Both takes are still on disk, because the honest way to make that claim is to keep the evidence: preprocess 0.197 s, diffusion 6.306 s, latent-to-audio 1.989 s, 8.49 s in total, seed 42.
The rule: composition is not a pipeline stage. When quality is a property of the whole output, a chain of specialists will each succeed locally and fail globally, and no amount of tuning the handoffs will fix it. Ask early whether the thing you are optimising is even visible from inside a single stage.
What the four have in common
None of these was found by reading the code. Each one required running the system for long enough, on real inputs, to reach the state where the design was wrong — and then being the person who had to live with the result. That is the argument for building things at your own expense before selling them: not that it proves competence, but that it is the only arrangement in which you are still there when the interesting failure arrives.
It is also the argument for the shape of our engagements. A firm that leaves at the pilot never meets any of these four, and will confidently hand you an architecture that contains all of them.