Human in the Loop: Why Our Automations Have an Approval Step

The nicest automation we ever built ran flawlessly for two weeks and then nearly cost us 400 euros. It was about to trigger a refund for a return that never arrived at the warehouse.

It didn't, because a human had to click once first. The click took four seconds. That's what this article is about.

"Human in the loop" means a person stays in the flow. They're a deliberately placed point where someone agrees before something irreversible happens, and they slow nothing down. I'm writing down where that point belongs and how to build it without creating a bottleneck. And when you're allowed to remove it again.

The question isn't "how much automation"

At some point in almost every conversation comes the line: "We want this fully automatic." I understand the reflex. Semi-automatic sounds like half finished.

It isn't. The right question is what happens in the worst case, and who carries it. The percentage running automatically comes second.

In our setup, the AI answers around 65 percent of support requests automatically. What such a system can actually do is in What Is an AI Agent. That number isn't there because we stopped building at 65 percent. It's there because the rest are cases where a mistake costs more than the minute you save.

The one question that decides: reversible or not

We run every step of an automation past three questions. Only when all three come back "uncritical" does it run without approval.

That gives us our split. Reversible and internal always runs automatically. Irreversible or connected to money gets an approval. Everything in between we decide by frequency.

Concretely, without approval on our side: sorting, routing, tracking information, stock reconciliation, building reports. With approval: refunds, goodwill outside the deadline, address changes after shipping, any reorder above a threshold, any reply to a visibly angry customer.

Where the approval sits decides how useful it is

A common mistake: putting the approval right at the end, just before sending. That feels safe and is the most expensive version.

By that point everything has already happened. The human reads a finished text and is supposed to judge whether the decision behind it was right. To do that, they have to reconstruct the whole thing.

Better is to put the approval where the decision falls. The question is "is this case goodwill, yes or no", instead of "does this email go out". The first question costs two minutes of reading. The second costs four seconds.

The difference sounds small and it decides everything. A four-second approval works permanently. A two-minute one gets clicked through blindly after three weeks.

How we build an approval that holds nobody up

Four rules we've worked out over the years.

First: the automation prepares everything. When the approval arrives, the work is done. Data pulled, rules checked, text drafted, recommendation stated. The human decides, they don't research.

Second: one decision, one click. The Slack message contains the case in three lines, the recommendation and two buttons. No switching to the backend, no hunting for the order number. Anyone who has to open three systems for an approval will postpone instead of approving.

Third: a pre-selection is allowed. The automation may say what it thinks is right and mark it as such. Agreeing is faster than deciding. Disagreeing always stays possible.

Fourth: there's a defined path. If nobody reacts within a set time, something defined happens. For uncritical cases it then continues automatically. For critical ones someone gets a reminder, and the case stays open. What must never happen: a case disappearing quietly.

These four rules make the difference between 20 seconds a day and 40 minutes. For us it's 20 seconds.

An example from daily operations

At mate we got order processing from 4 hours a day down to 15 minutes. Those 15 minutes are the approvals. They're the part we deliberately kept, rather than leftover work we haven't automated away yet.

Here's what it looks like. Orders run through overnight. In the morning there are three to five cases in Slack where the automation didn't proceed on its own. Usually an implausible delivery address, an unusually large order quantity, or a return whose goods never arrived according to the warehouse.

Each case sits there as a short message. What happened, what the system recommends, why. Two buttons underneath. Five cases, two minutes, done.

That's exactly the loop the return from the intro was hanging in. The system correctly recognized that the customer had registered a return. It didn't see that the parcel got lost on the way back, because the goods receipt at the warehouse was missing. A human saw it in four seconds.

Without that loop we couldn't have kept three brands running with 2 people in day-to-day operations. That's because of the approvals, not despite them. More on the overall setup is in the Guide to Customer Service Automation.

When you're allowed to remove the approval

Approvals aren't a permanent state. They're a learning mode, and it's allowed to end.

We remove an approval when three conditions are met.

The agreement rate is the most important number, which is why we log every approval with its outcome. Without that log, the question "can we let this run automatically now" stays a gut feeling forever.

Over two years, roughly half of our original approvals have disappeared this way. The other half is still standing, and some of them never will disappear. Refunds, for example. The rate there is good, the risk is unchanged.

The limits

Human in the loop is no magic word. Four places where it doesn't help or even does harm.

And one caveat about the model behind it: an approval only catches what a human can see in a few seconds. An AI answer that is factually wrong but plausibly worded slips through. Why that happens comes down to how these models work. The distinction between agent, chatbot and workflow is in AI Agent, Chatbot or Workflow.

Conclusion

An automation without approval isn't braver, just less checked. The craft lies in placing the human exactly where four seconds of attention prevent the biggest damage.

My advice if you're just starting: build with approval, even if you're sure. Measure the agreement rate. Take the approval out when the numbers allow it, not when impatience demands it. That way you get both, speed and a good night's sleep.

If you're unsure which steps in your shop need an approval: talk to us at Flowhouse. We'll go through your processes with you and also tell you where you can safely let things run automatically.