Field Note: five failures that returned success
In one evening across four systems, every defect that mattered exited clean. A download job that moved nothing, a classifier that dropped a third of its input at the buzzer, an email that addressed its recipient in the third person. Notes on the dominant failure mode of model-assisted building, the three defenses that catch it ranked by what they cost, and one wrong diagnosis of my own.
By Russ. Field Note, 2026.
The existing approach
I run several systems at the same time. A demand-research pipeline that turns a local market's search volume into a positioning board. An outbound engine that drafts personalized email at volume. Research agents that do document work against real deal material. And the ordinary file plumbing underneath all of it. A normal evening touches four or five.
The way I check that work is conventional. Tests run. A build either completes or it does not. A script exits zero or it does not. When something looks wrong I read the output and trace backward. That loop is good, and for the class of defect that announces itself it is close to free.
One evening's session log had a property I did not plan and would not have guessed. Four separate systems produced real defects. Not one of them raised an error. Every failing path exited clean and reported success. This note is about that class specifically, because the larger the model-assisted share of my work gets, the larger a share of my total defects this class takes.
Where the ordinary loop works
Errors that throw are the cheap ones. A stack trace names the file and the line. A failing assertion names the expectation. A non-zero exit stops the pipeline before bad output reaches a person. The entire apparatus of ordinary debugging rests on the assumption that a system will tell you when it is unhappy, and where that assumption holds the apparatus is excellent. Nothing here argues for replacing it.
The argument is about the defects that sit outside its reach, and about the fact that model-assisted pipelines produce more of them than hand-written code does. A model does not throw. Given a degraded input or a truncated instruction it returns something plausible, and plausible output is indistinguishable from correct output to every check that only asks whether output exists.
Where it breaks
Five cases, abstracted from the log.
One: a download job that moved nothing and exited zero. A bulk fetch of 404 documents off a single index page. Transfers were driven from a config file whose output paths were written in native Windows form. Inside a quoted value in that format a backslash is an escape sequence, so every path was quietly malformed. The tool did not object. It reported success on each transfer and wrote no bytes. Exit code zero, 404 times.
A smaller version of the same thing happened one step earlier. The first read of that index page went through a summarized extraction and produced roughly two dozen fewer links than parsing the page directly. The summary was not wrong in any way I could see by reading it. It was short, and nothing about it advertised being short.
Two: a classifier that dropped a third of its input at the buzzer. The bucketing step of the research pipeline sends keywords to a model in chunks. Successful chunks were completing in 20 to 30 seconds. The timeout was set at 30. So a large share of chunks aborted at the boundary, each carrying about a hundred keywords out of the run with it. The pipeline continued. The board rendered. The totals were smaller than they should have been, and nothing in the run said so. It had been shrinking a second, unrelated board the same way for some time before anyone looked closely enough to notice a success that seemed thin.
A timeout set inside the operation's own normal latency band is not a timeout. It is a coin flip that reports heads as completion.
Three: the one that did throw, and why. That product has a hard rule: the subject of a board is never named on the board, because boards live at unlisted URLs and the no-name rule is what makes an unlisted URL safe to send. The rule held for weeks of testing because the first test subject was a business almost nobody searches for. The first subject that actually ranks in its own market broke the rule on contact. Its own name entered the payload as a competitor, because local pack listings are names rather than domains. A payload check refused to ship the build.
That refusal is the only reason this case belongs in a field note instead of in someone's inbox. It is the same class as the other four. The difference was that the invariant had been written down as code that fails rather than as a rule someone intended to follow.
Four: an email that addressed its recipient in the third person. The outbound engine builds each message around one researched line about the recipient. That research arrives as notes, and notes are written about a person, not to them. Nobody converted the voice. So the drafts opened by discussing the recipient to their face, and several carried internal labels and source URLs inside the body text.
Every draft was well formed. Every one passed every check the system had, because the system's checks asked whether a personalization line existed and whether the address verified. The defect was obvious in about one second to a human who opened the folder and read one, which is what happened. That is the only reason nothing went out.
No test I would plausibly have written catches an email that is grammatical, correctly personalized, factually accurate, and pointed at the wrong grammatical person. Read the artifact.
Five: a return value nobody read. In the same engine, the call that appends a draft to a mail folder returns a status, and the code discarded it. It had been working the whole time. That is the point. An unchecked return is not a bug until the day it is, and on that day it is a silent one.
One more, and it is mine. When the drafts folder read empty, I said the appends had been silently failing. That was wrong. Appends were working, verifiable against the identifier the server hands back on every write, and the folder was empty because the drafts had been reviewed and deleted by hand. My diagnosis had the right shape for the evening and no evidence underneath it. It belongs in this list: a confident wrong answer, produced quickly, with nothing in it that announced itself as a guess. Same failure mode, one level up.
What an AI architect would change
Three defenses. Ranked by what they cost, cheapest first.
- Read the return value you already have. Every one of these systems was already being handed the information it needed. The transfer tool reported per-file status. The mail server returned a write identifier. The chunked classifier knew which chunks aborted. The cost of using that information is one conditional. This is not a discipline, it is an oversight with a name, and it accounts for two of the five.
- Write each invariant as a gate that refuses to ship. Not a warning that scrolls past, not a line in a prompt asking the model to behave. The books must reconcile to the source total. The payload must not contain the subject's name. Every populated cell must carry either auditable volume or a written recommendation. A run that violates one of these throws and produces nothing. This is more expensive: gates cost real work to write, and they occasionally refuse a build you wanted. It is still the cheapest thing in this note relative to what it prevents, and it is the only reason case three is a story about a guard instead of a story about a client.
- Read the artifact yourself, in the surface where it lands. Not the logs, not the test summary, not a sample rendered in a terminal. Open the actual folder, read the actual email, look at the actual board. This is the most expensive defense because it does not scale and cannot be automated, and it is the only one that catches a defect the system has no vocabulary to describe. Twenty drafts took two minutes to read and that read was worth more than the test suite that night.
The ordering matters more than the list. Teams reach for the third defense by instinct, apologize for not having more tests, and skip the first entirely.
What it cost
The fixes were small and almost boring. One number changed on the timeout. One name-match filter on the payload. Forward slashes in the config paths. One conditional on a return value. One prompt rule requiring second person and forbidding internal labels in output. Call it an evening of work, most of it spent locating the problems rather than repairing them.
The rebuilds cost more than the fixes did, which is the argument for probing before rerunning. A full board build runs about ten minutes and roughly a dollar. A targeted probe against one endpoint costs cents and seconds and turns a guess into a fact before the expensive path spends anything. Nearly every diagnosis above was cheap to confirm once someone decided to confirm it rather than reason about it.
What would not change
None of the domain logic moved. The job map and awareness rows in the research product, the counting rules, the positioning framework, the outbound sequence structure, the message bodies: all unchanged. What changed was the honesty of the plumbing under them. Localization got more truthful and the totals got smaller, which improved the product, because a number that survives an audit on a call beats a larger number that does not.
Nothing here is an argument against building this way. The pipeline that produced these five defects also produced a finished market board in about ten minutes for roughly a dollar, and twenty personalized drafts in an evening. That throughput is real and it is why the failure mode is worth studying rather than avoiding. Higher throughput moves the bottleneck to verification. It does not remove it.
Closing
This is most of what the job actually is, at least in the unglamorous middle of an engagement. Not selecting a model. Building the gates that catch the model and the data being confidently wrong, then pricing every fix in dollars and minutes so the argument for the gate is an economic one rather than an aesthetic one.
If your firm ships numbers or written output to clients through a model-assisted pipeline, the useful question is not which model you use. It is which of your invariants can currently fail without throwing. Anywhere the honest answer is "I would find out from the client" is the roadmap.