The model had been in production for fourteen months and nobody could say what it was for.
It lived in the refund pipeline of a mid-sized subscription billing company: a fine-tuned classifier that read a cancellation message and decided whether the account qualified for a prorated refund. A cent a call, four hundred milliseconds, ninety-four percent agreement with the written policy. That last number looked excellent in a quarterly review and different in finance, where six percent of a hundred thousand monthly decisions is six thousand wrong answers, and support had quietly begun checking every one by hand.
An engineer eventually did the thing nobody asked for. She read the policy. It fit on two pages. She turned it into a decision table: plan type, days elapsed, prior refunds, region, promotional flag. Ninety-one lines, no dependencies, no vendor. The new path answered in eleven milliseconds, cost nothing per call, and agreed with the refund policy every time, because it was the refund policy. Support stopped double-checking within a month. Disputes fell. The postmortem was two sentences.
That story is unusual only because someone told it. Removing a model is one of the least celebrated decisions in software. Teams that ship AI write launch posts; teams that remove it say nothing, because most org charts read deletion as retreat. That is a failure of product judgment, not engineering nerve. Marty Cagan has spent two decades arguing that teams should be measured on outcomes, not output. An LLM is the most shippable thing there has ever been, and routinely an output wearing an outcome’s clothes.
The question is not whether the technology is capable. It is whether your problem is one where probability is an asset or a liability.
Six conditions for subtraction
Use deterministic logic when the rules are knowable and stable. If an expert can write the logic down and it still holds a year later, encode it.
Use it when the correct output is legally or financially binding. In 2024 a British Columbia tribunal held Air Canada responsible for a refund policy its chatbot had invented, rejecting the claim that the bot was separate from the website it lived on. The damages were trivial; the precedent was not. If an output can bind you, it is your policy, and policy should not be sampled from a distribution.
Use it when failure is silent rather than loud. A crashed service pages someone at 3 a.m. A model wrong four percent of the time in a confident, well-formatted way pages nobody. Zillow ran the large-scale version, shutting down its home-buying arm in 2021 after more than half a billion dollars in write-downs, prices having proved less predictable than its models implied. Nothing broke. The error accumulated until it was structural.
Use it when latency and unit economics matter more than coverage. Per-token pricing turns a fixed engineering cost into a variable one that grows with your success, and attaches a vendor to a code path you cannot inspect. Plenty of "AI-powered" features are lookup tables with a subscription attached, solving what a hash map solved in 2009 for the price of RAM.
Use it when auditability or regulatory explainability is required. "The model assigned high probability to this outcome" is not an explanation that survives an auditor, an adjudicator, or a denied customer. A rules engine produces a trace, and a trace is a defense.
And use it when the error budget is effectively zero: payments, tax, permissions, dosing, compliance, safety interlocks. The target is not 99.9 percent. It is the rule, applied, and an architecture that cannot promise that is the wrong one however well it demos.
The inverse is a short list. Reach for probabilistic systems when the input space is open-ended, when ambiguity is the norm, and when a wrong answer is cheap to recover from. Free-form text, messy documents, ranking, intent detection, first drafts. The test is whether an undo button is sufficient remedy. If it is, sample away.
The deeper objection is architectural. Non-determinism makes testing, regression, and incident response structurally harder, and no tooling has fixed that. You cannot write an assertion against a distribution. You can write an eval, and evals are useful, but an eval suite is not QA. An eval reports average behavior across a sample; QA guarantees that this input produces that output, today and after the next deploy. Confusing them becomes loud during incidents, when there is no stack trace, no diff to revert, and the root cause reads "we adjusted the prompt."
The requirements problem is worse. Models routinely paper over decisions nobody wanted to make. When a team says the rules are too complex to write down, that is occasionally true and usually not; usually the rules were never agreed, and the model absorbs the political cost. Mind the Product and Product School have called this the feature factory for years, and Lenny’s Newsletter keeps surfacing the same confession: features shipped that nobody can tie to a metric. Cagan’s blunter version is that outcomes are hard, and organizations prefer what is easy to count.
The steelman
The strongest case for the model is coverage. Rules handle what you anticipated; models degrade gracefully on what you did not. Real inputs are long-tailed, and a deterministic system meeting an unanticipated case fails absolutely. Rules also rot: ninety-one lines become four thousand, and the table becomes the black box it replaced. Inference costs keep falling. "Deterministic" also flatters systems that fail spectacularly on their own: Knight Capital lost $460 million in forty-five minutes in 2012 to code doing exactly what it was written to do.
Knight is the rebuttal, not the counterexample. That failure was reconstructible in hours because every code path was inspectable; the regulatory finding was about controls, not mystery. Rule rot has mature remedies: versioning, coverage measurement, staged rollout, deletion. Model drift has worse ones. And coverage only helps where partial correctness helps. In a binding domain a plausible wrong answer is worse than a refusal, because a refusal routes to a human and a wrong answer routes to production.
Deterministic core, probabilistic edges
The answer for almost every real product is neither extreme. Put the model at the edges: extraction, ranking, classification, drafting. Keep a deterministic core that owns state, money, and permissions, and a rules-based validator in front of every write action. The model proposes a refund; the rules engine approves it. The model extracts an invoice total; a schema and an arithmetic check confirm it. The model drafts the denial letter; the policy engine decides the denial. Where a product touches money, health, or access, the validator is the product and the model is a very good typist.
That also makes subtraction cheap, which is the point. If the model sits at the edge, removing it is a Tuesday. If it is load-bearing, removing it is a rewrite that never gets prioritized, and the six percent stays in the pipeline another fourteen months.
Nobody gets promoted for the ninety-one lines. But the refund team stopped double-checking, and that was the metric that mattered. Adopting AI was never really a question of ambition. It is a question of how much wrongness a product can absorb, and who pays when it cannot.
