Hallucinations Under Control: Catching AI Errors Before the Customer Sees Them

Our support agent once promised a customer a delivery for Tuesday. At that moment the parcel was still with the sender, and that date existed in no system anywhere.

The industry calls those moments hallucinations. The model invents something that sounds plausible and is simply wrong. If you put AI in front of customers, this is your biggest risk. Less because it happens often, more because it happens quietly. A broken workflow stops and reports itself. A hallucinating model keeps replying politely.

Here I write down why models invent, which tasks are most exposed, which countermeasures really work for us, and what still slips through.

What a hallucination actually is

A language model, meaning the AI behind chat and text generation, always predicts the next likely word. It has no database it looks things up in. It has a feel for how a correct sentence sounds.

That is the entire mechanism. Everything else follows from it. The model draws no line between "I know this" and "this sounds right". Both feel identical to the machine.

A hallucination is therefore not a bug someone will patch out eventually. It is the flip side of exactly the capability you want to use. A model that never fills anything in could not draft an email either.

If you want to understand where the language model ends and the agent begins, the distinction is in What is an AI agent.

Why models invent precisely when it gets expensive

Three situations trigger it almost every time.

First: you ask for knowledge instead of processing. "Where is order 4711" is a knowledge question. The model has never seen that order. It fills the gap with something that looks like an order number and a delivery date.

Second: the assignment is too big. Ask a model to read an enquiry, check the rules, weigh up an exception and write an email, and you get four half-good results in one. Errors from step one travel unchecked all the way into the text.

Third: there is no permitted way out. If your instruction only says "answer the question", the model answers the question. Always. Even when the answer exists nowhere. "I do not know" has to be an explicitly allowed exit, otherwise the model never takes it.

Which tasks are most exposed

Not every AI task carries the same risk. Roughly sorted from harmless to delicate:

• Classifying. Sorting an email into one of eight categories. Very robust, because the choice is bounded. How we set that up is in Pre-sorting support emails with AI.
• Summarising. Also stable, as long as the original text is supplied. The risk lies in omission rather than invention.
• Rephrasing. Existing content gets a different tone. Barely any room for invention.
• Free writing with facts. A customer email with a date, a number and an amount. This is where it happens.
• Calculating and deadlines. Models calculate unreliably. Percentages, days until expiry, partial refunds: that belongs in code rather than a language model.
• Explaining your own rules. Return windows, warranties, shipping costs. The model knows thousands of shops from training and blends them into yours.

The last three lines are the ones that cost you money.

Countermeasure 1: Supply the data instead of querying knowledge

This is the biggest lever, and it nearly halves the problem on its own.

Never ask the model what it knows. Give it what it needs and let it only phrase. Concretely: before the text is written, a plain workflow pulls the order from Shopify, the status from the carrier and the applicable return window from a table. Only then does the model see the enquiry, together with those values.

The difference in one sentence: the model used to be the source, now it is the translator. Translating is something it does very well.

That comes with one hard rule in the instruction. Anything that is not in the supplied data must not appear in the answer. If a value is missing, the case goes to a human.

Countermeasure 2: Narrow tasks instead of large assignments

We break every AI task into steps that can be checked individually. One call classifies. A second extracts the required fields. A third writes.

That costs more calls and is still cheaper. Small tasks run on small, fast models. And when something goes wrong, you see immediately at which point. With one big call you only see a wrong result and guess.

That decomposition is also why we do not use agents everywhere. A fixed sequence with three small AI steps is easier to trace than an agent planning freely. Which approach fits when is covered in AI agent, chatbot or workflow.

Countermeasure 3: A check between text and send

Before an answer goes out, it passes two controls.

The first is blunt code, no model. It checks three things. Does the answer contain a number that did not appear in the supplied data? Does it contain a date nobody delivered? Does a word from the blocklist show up, for example a promise like "guaranteed by"? Every hit stops the send.

The second is a second model call with a single question: is every statement in this text covered by the data below it, yes or no? That is classification, the most robust type of task there is. A model checks far more reliably than it invents.

Both controls together cost a few cents per enquiry. One wrong promise to a customer costs more.

Countermeasure 4: Approval on everything that hurts

Some cases get no automatic path with us. Those are refunds, address changes after dispatch, goodwill outside the window, and anything that sounds like trouble.

The AI may prepare these cases. It writes a draft, attaches the data and files it. A human clicks. That takes twenty seconds instead of five minutes, and the error never reaches the customer.

The order matters. Set the limit first, then automate. Never the other way round.

Countermeasure 5: Every answer leaves its evidence

Every automatically sent reply lands in our log with its full context. Enquiry, data pulled, intermediate results, final text.

That sounds like bookkeeping and is in truth your early warning system. Only when you lay twenty cases side by side do you see patterns. Always the same product group, always the same phrasing, always the same invented detail. Without a log you are searching in fog.

What this does day to day

An example from operations. At mate around 40 percent of all enquiries fall under "where is my parcel". The model used to phrase the answer freely, including an estimated delivery date. That is exactly where the Tuesday from the introduction came from.

Today the flow is different. The workflow pulls the real shipment status. If the carrier gives no delivery date, no date may appear in the email. The answer then says where the parcel was last scanned and names no date.

Less satisfying to read. Wrong far less often. Overall AI answers around 65 percent of our support enquiries automatically. That only works because the remaining enquiries are sorted out cleanly. The complete setup is in the guide to customer service automation.

What still slips through

Now the uncomfortable part. We have not solved the problem, we have made it small.

• Plausible half-truths. The model quotes a rule correctly but applies it to the wrong case. The check finds no invented value, because there is none. We only catch this through samples.
• Tone instead of fact. An answer is factually right and still wrong, because it meets an upset customer too smoothly. No control in the world measures that.
• Gaps in your own data. If the return window is maintained differently in three systems, no model helps you. Then you have a data problem, and AI is not the issue.
• Rare phrasings. An enquiry in dialect, with irony, or with two issues in one sentence. The sorting misses, and from there the wrong branch runs.
• Quiet degradation. Models and vendors change. What ran cleanly last month can answer differently this month. Without a regular sample you notice it at the customer.

That is why we read twenty random automated replies every week. Twenty minutes, every week, for two years. It is the least spectacular part of the system and the part I would be least willing to cut. Where automation generally hits its limit is covered in When automation makes no sense.

Conclusion

Hallucinations do not go away. They do become manageable if you do three things. Supply the data instead of querying knowledge, cut tasks small, check before sending. Everything else is polish.

The mistake in thinking I see most often: people look for the better model when what they need is the better process. An average model with clean data and a check step beats a strong model without either. Every time.

If you are working out how much AI you can put in front of your customers, talk to us at Flowhouse. We will also tell you when a plain workflow is the better answer.