There are two scenarios I keep seeing with AI in production, and they are the same scenarios in different disguises.
The first: a team has seen a demo, the demo was dazzling, and now probabilistic output is headed somewhere that needs exactly one right answer - a total, a status, a legal something. Everyone's excited. The second: a team has a genuinely perfect AI use-case - a summary, a suggestion, a first-pass guess - and they've blocked it, because "we can't have it giving different answers each time". They're holding a fuzzy operation to a bar the operation never required.
Over-trusting and under-trusting. Opposite failures, same root cause: nobody reasonably classified the operation. "Should we use AI?" got treated as one big decision about one big technology, when it's actually hundreds of small decisions about hundreds of small operations - each with its own tolerance for being wrong.
I wrote a section in a previous post about this - "you don't evaluate the final output; you evaluate the space of outputs". So this is that section, given the room it needs, because I've now shipped this three times: a tax platform, a puzzle game, and a mental-health app. Three products, wildly different stakes, same underlying question every time.
"Should we use AI here?" is not a question about AI. It's a question about here.
The three-layer test
When I look at an operation and decide whether the probabilistic machine is allowed near it, I'm running three checks, in order:
Blast radius decides IF. When the output is wrong - not if, when - can it be caught, corrected, or undone cheaply? A mislabelled transaction that a human reviews later: contained. A payment fired at the wrong account: not contained. If the failure is expensive and irreversible, the conversation ends here, no matter how good the demo was.
Answer-space decides WHERE. Is there exactly one right answer, or a space of acceptable ones? A bank balance has one right answer. A merchant name cleaned up for display has dozens of fine answers. A puzzle has thousands of good answers. The moment you realise most of your product's operations live in answer-spaces, not answer-points, the AI conversation gets dramatically less scary.
Verifiability decides HOW. What deterministic thing can check the output afterwards? A schema, an enum, a rule engine, a hash, a human queue. Generation is allowed to be fuzzy when verification is exact. If nothing can verify the output - not code, not a human - then you don't have an AI use-case, you have a liability with good charisma.
Blast radius decides if, answer-space decides where, verifiability decides how. Run it per operation, not per product. Every system I'm about to show you both uses AI and refuses AI - in different rooms of the same house.
WorkMade: the expert who caught the machine
At WorkMade (freelancer tax platform, US, real money, real IRS forms) we used an LLM to classify transactions - is this deposit freelance income, W2 salary, an invoice payment, or not income at all? Thousands of messy bank strings, SQ *KHAO SOI and VENMO CASHOUT and worse, arriving daily. Exactly the kind of brute-force first-pass work AI is brilliant at, and exactly the kind of domain where being confidently wrong has a tax bill attached.
Here's how the system held that tension:
The LLM got a vote, not a veto. The AI's income classification was stored in its own columns, alongside deterministic confidence scores computed from boring old maths - amount variance, cadence regularity, keyword matches. The model's opinion was one signal among several, never the signal.
The database couldn't store an answer that wasn't on the menu. Every LLM response got forced through a clamp:
switch (parsedResult.incomeType.toLowerCase()) {
case 'freelance': incomeType = IncomeType.freelance; break;
case 'w2': incomeType = IncomeType.w2; break;
case 'invoice': incomeType = IncomeType.invoice; break;
case 'not income': incomeType = IncomeType.not_income; break;
default: incomeType = IncomeType.not_valid; // off-menu → rejected
}The model can hallucinate whatever it likes; the schema physically cannot receive it. If you take one implementation detail from this post, take that one - the enum is load-bearing.
The most deterministic source that had an answer, won. Categorisation ran through a fallback chain: exact user rule first, then fuzzy rule match, then Plaid's enrichment, then a plain MCC table lookup, and only then the LLM. The probabilistic machine was the court of last resort, not the first opinion.
Human corrections were sacred. Once a human overrode a category, automation skipped that transaction forever. The machine never, ever re-overwrote the human. This sounds obvious. I promise you it is not standard practice.
And when the model failed, it failed to nothing. No retries, no "let me try rephrasing" - a broken or unparseable response degraded to null, and null meant "route it to the deterministic path or a human". An absent answer beats a forced one. (One spot in the codebase parsed LLM output with no try/catch at all, which I'm telling you about because it proves the rule: the single place without a guardrail is the single place with a latent bug.)
Meanwhile, in the very same codebase: money movement. Locks, idempotency guards, balance checks, and not an LLM within a country mile of it. Same product, same team, same week - opposite answers to the classification test.
The feedback loop: corrections become rules
My favourite part of that system, and the part I'd shout about if I could only shout about one: when a human corrected the AI, the correction didn't just fix that transaction. It became a rule - stored, indexed, and consulted first in the chain from then on. Ahead of Plaid. Ahead of the model.
Think about what that means. Every expert correction permanently shrank the non-deterministic surface. Ask the LLM once, learn the answer, never ask again. Add caching with sensible staleness windows (we re-checked a job's likely tax write-offs every six months, not every request) and the system had a lovely property: the longer it ran, the more deterministic it got. Not because the model improved - because the system metabolised human judgement into rules.
That's what evaluation actually looked like day-to-day, by the way. Not per-item formal verification - confidence-based routing (high confidence flows, low confidence queues for a human), the expert loop tightening things over time, and periodic spot-checks against expert judgement. The threshold was the correctness dial. We turned it like one.
Devdle: fuzzy generation, strict verification
Fast-forward to Devdle, my daily developer puzzle game, where OpenAI generates each day's puzzle. Non-deterministic generation feeding a product that needs valid, fair, non-repeating puzzles - a completely different corner of the classification map, and so, completely different machinery.
The temperatures tell the whole story: generation runs at 0.6, and the QA audit pass runs at 0. Warm when creating, cold when judging. That's this entire post in two config values.
The generation is allowed to be creative because everything around it is strict. Every candidate gets parsed against a Zod schema - parse or die. A check rejects any puzzle that accidentally reveals its own answer in the prompt. A novelty layer fingerprints each puzzle's concept and compares it against the last fortnight's - too similar, rejected, and the rejection reason gets fed back into the next generation attempt as a steering hint. And then, my favourite bit: a second model call, running cold on a cron, re-verifies stored puzzles before they go out - AI auditing AI, with its corrections only accepted after they pass the same deterministic validation. If generation fails entirely, a hand-written seed puzzle ships instead. The product never white-screens because a model had a bad day.
The puzzle answers are never executed. There's no sandbox running the code snippets to prove the claimed answer is true. The truth guarantee is LLM-based, twice over - one model claims, another model checks the claim. For a free daily puzzle, that's fine; the blast radius is somebody tweeting that Tuesday's answer was wrong (and they have). For a tax return? Absolutely not fine. Identical architecture question, different blast radius, different answer. The test does the deciding, not the technology.
Somna: the gate comes before the model
Third product, third corner of the map. Somna is my emotion-pattern app - the LLM suggests emotions, reframes thoughts, writes weekly reflections. And here the stakes invert: a bad output isn't incorrect, it's potentially harmful. Different failure type, different architecture.
The most important guardrail in Somna runs before the model, not after. Deterministic crisis-language detection sits in front of every single LLM call, and if it fires, the model is never consulted - the user gets fixed, hand-written crisis-resource content instead. No temperature, no sampling, no space of outputs. Some inputs should never reach the probabilistic path at all, and no amount of prompt engineering ("be careful, be kind, be safe") is an acceptable substitute for a hard gate that removes the dice from the table entirely.
Below that gate, the usual layering: prompts that constrain hard ("never diagnose", "no advice", length caps), and a full manual mode so the app works with the AI switched off completely. Classify the operations and it maps cleanly: suggesting an emotion word is a space-of-answers problem, AI welcome; responding to someone in crisis has exactly one acceptable behaviour, hardcoded, forever.
What evaluation actually looks like
Pulling the thread out of all three: you don't evaluate an output, you evaluate the output space.
Same input producing different outputs is not a bug - not if every output lands inside the acceptable region. Chasing run-to-run reproducibility on a fuzzy operation is solving for your own comfort, not for the user. The questions that matter are: is the region well-defined? Is the deviation dangerous or merely creative? And who decides where the region's edges are?
That last one has an uncomfortable answer: a domain expert. Not a dashboard - a person. Without someone who can look at an output and say "that's plausible but wrong", you cannot know your output space is acceptable. You can only know it parses. Schemas catch malformed answers; only expertise catches wrong ones. The expert isn't a nice-to-have on top of the system. The expert is part of the system.
The maturity path
So here's the Monday-morning version - four steps, in order, and most teams I meet are stuck before step one:
- Classify. Go through your operations - actual operations, not features - and sort them: one right answer or a space of them? What's the blast radius? What could verify the output? This list is your AI strategy. Everything else is implementation.
- Gate. Put deterministic checks around every model call: schemas, enum clamps, exposure checks, and - where the input itself can be dangerous - gates that run before the prompt. Fail closed to null, never to a guess.
- Route. Confidence thresholds deciding what flows and what queues for a human. Deterministic-first fallback chains. A plan for total model failure that isn't an error page.
- Feed back. Corrections become rules. Answers get cached. Samples get audited. If your non-deterministic surface isn't shrinking month on month, you've built a slot machine with extra steps.
TL;DR
- "Should we use AI?" is hundreds of decisions wearing one trench coat. Classify per operation, not per product.
- Blast radius decides if. Answer-space decides where. Verifiability decides how.
- Make the AI advisory, not decisive - the most deterministic source that has an answer should win.
- Human corrections are sacred: never re-overwrite them, and turn them into rules so you never ask twice.
- Generate warm, verify cold (0.6 to create, 0 to judge).
- Some inputs should never reach the model at all. The best gate runs before the prompt.
- Classify → gate → route → feed back. The longer your system runs, the more deterministic it should get.
The machines that are usually right are extraordinary. Just be precise about which rooms of the house they're allowed in - and make sure the one holding the chequebook stays locked.