Our Biggest Automation Fails and What They Taught Us

One Sunday morning, one of our automations sent 1,400 emails. To 340 customers, four each, within eleven minutes.

I am writing this down because I have had a rule ever since that I did not have before. In conversations I often hear that automation is either finished or broken. Most of the time it is somewhere in between.

Here are five failures from our own brands. Each one with what happened, why it happened, and what we built out of it. No anonymous client stories, no numbers massaged into shape. Ours.

Fail 1: A loop with no stop condition

We had a workflow that hands orders over to the fulfilment provider. If the provider does not respond, it tries again. Sounds sensible.

The problem: there was no upper limit. The provider had a maintenance window and returned errors for 40 seconds. The workflow kept trying. After an hour there were 6,000 calls in the log, and our API quota for the day was used up. After that, real orders stopped getting through.

The cause: I had thought about the error case, but not about the error case that lasts. A single hiccup and a twenty-minute outage look identical to the workflow.

The lesson: every retry needs three limits. A maximum number, a growing gap between attempts, and a defined end state. For us the end state is: the case lands in a list, a human gets a message, the workflow stops. Standing still with a notification always beats moving with no destination.

Fail 2: 1,400 emails on a Sunday

Back to the intro. The workflow sent a shipping confirmation as soon as a tracking number appeared in the system. On Sunday night the server restarted. On the way back up, the system worked through the queue of the previous hours a second time. And because the restart happened three times, three extra times on top.

Nowhere was the question asked: have I already sent this email for this order?

The cause: a workflow that only knows its trigger and not its own result. The technical term for what was missing is idempotence. It means: if you run the same step twice, nothing should happen the second time.

The lesson: everything that leaves the house gets a key and a record at our end. Before sending, the workflow checks whether that key is already recorded. That is two extra steps per workflow and roughly ten minutes of build time. The Sunday cost us around 300 euros in vouchers and half a day of apologies.

Fail 3: Stock numbers nobody believed any more

We sync stock levels between the shop, the marketplaces and the fulfilment provider. For a while that ran as a sync every 15 minutes. Every channel wrote its own level, the last one won.

Over weeks the numbers drifted. Sometimes 3 units too many, sometimes 8 too few. We sold goods twice that no longer existed. And for weeks we listed a product as sold out while 60 units sat on the shelf.

The cause: two systems changed the same value at the same time. If both start from an old level, the slower one overwrites the faster one. On top of that came a time zone: one channel delivered timestamps in UTC, another in local time. A two-hour offset is enough to declare the wrong change the newer one.

The lesson: for every value there is exactly one system that owns it. Everyone else reads it and never writes it. For us that is the fulfilment provider, because that is where the goods physically sit. And every timestamp in the house is UTC, without exception. That decision would have cost two hours of thinking at the start. Retrofitted, it cost a week of rebuilding.

Fail 4: We automated a process that no longer existed two weeks later

At one of our brands we automated the returns process. Form, label, inspection, refund, stock booking. Six days of work, cleanly built, well tested.

Twelve days after go-live we switched fulfilment providers. The new one worked with a different returns procedure. About one day of the six days of work survived.

The cause: the switch had been on the table for months. I knew that. I built it anyway, because the process hurt right then and because I felt like doing the work. It was impatience rather than a misjudgement.

The lesson: before we build, we now ask: will this process still look like this in six months? If the answer is uncertain, we build the smallest version that takes the pain away and wait. We wrote down when automation does not pay off at all in when automation makes no sense. The opposite mistake, automating too late, has a price too, described in the 10,000 euro mistake.

Fail 5: A workflow was dead for eleven days and nobody noticed

Our weekly report for one brand had been running for a year. At some point it stopped arriving. We only noticed because somebody went looking for a number.

Eleven days earlier a vendor had rotated its credentials. The workflow broke at the first step. It dutifully reported the error into a log that nobody reads.

The cause: we monitored whether errors occurred. We did not monitor whether something failed to appear. A workflow that dies makes no noise. It simply goes quiet.

The lesson: every recurring workflow needs a watcher that checks whether it ran. At our end a second, small routine speaks up when the first one leaves no sign of life inside its time window. It is unspectacular and has since found three dead workflows before anyone missed them.

What the five have in common

For a long time I thought these were five different topics. They are the same one. Four of the five cases share one core: the workflow did not know its own state.

It did not know how often it had already tried. It did not know whether it had already sent this email. It did not know whose number was currently the right one. It did not know that it had run last week.

A workflow that only reacts to triggers and knows nothing about itself works perfectly as long as everything runs smoothly. That is the dangerous kind of bug, because it only shows up on the first bad day.

What would not have saved us

Two things that often come up as the answer to stories like these and were not the solution for us.

More tests. All five workflows had tests. What got tested was the path I had imagined. The Sunday with the restart was in no test, because it never occurred to me. Tests check your assumptions, they do not replace them.

A human approval at every step. Approvals help enormously, but only for decisions a human can judge in seconds. For duplicate emails or a silent outage, no click helps. Where approvals genuinely carry weight is in human in the loop.

The limits of these lessons

I do not want to pretend we have solved this now. Three caveats.

First: the rules above cost time. A workflow with a watcher, keys and clean limits takes roughly 30 percent longer than the quick version. For an automation that runs twice a month, that is simply too much. We deliberately build more sloppily there, and we know it.

Second: we keep making mistakes. This year we had two new varieties that fit into none of the five categories. The list is not a closing list.

Third: none of these rules apply to AI steps. A model that writes a wrong but plausible-sounding answer violates no technical condition. Why that is and what helps is in avoiding AI hallucinations.

Conclusion

The most expensive part of these five cases was the hours we spent distrusting our own numbers and recalculating everything by hand. The money was the smaller line item. An automation you do not believe is worse than none at all.

If you are building right now: take the table above and walk your most important workflow through it. Four questions, twenty minutes. For most people something jumps out straight away.

And if you want to know which processes are worth it for you in the first place, automating e-commerce processes is a better starting point than this article.

If you want your existing automations reviewed from the outside once: talk to us at Flowhouse. We will also tell you when everything is fine.