A Day with AI Agents: What Gets Finished Overnight and What Stalls
At 7:40 this morning, six finished change proposals were waiting for review. I approved four, sent one back and deleted one.
At Flowhouse we run our own system in which AI agents work through tasks overnight. An agent here is a program that breaks a task into steps by itself and works through them. What such a system even is, is covered in what is an AI agent. This article describes a day with one, and it is no build guide.
I am writing it because expectations about systems like this usually sit at one of two extremes. Either it builds the company overnight, or it is a toy. Day to day life is somewhere else.
5 pm: preparing the tasks
The most important part of the day happens before any agent starts. I write the tasks.
A good task for an agent looks like this here: it has a clear goal in two sentences. It names the place where the work happens. It says how you can tell it is done. And it is small enough that a human would finish it in under two hours.
Tasks that miss those four points come back as a half-finished proposal. That is the most reliable rule I have on this topic. The quality in the morning depends almost entirely on the wording the evening before.
Preparation costs me 20 to 40 minutes per evening for four to eight tasks. That time is the real price of the system.
10 pm: the agents start
From then on nothing happens that I watch. Each agent gets one task, its own working copy and a time limit. It may read, write and run tests.
At the end it puts up a change proposal for approval. It may not merge anything itself. That boundary is the most important safeguard in the whole system, and it is deliberately non-negotiable. Why we set approval points like that as a matter of principle is in human in the loop.
Two further rules apply. First: if the tests are red, there is no proposal. Second: if the agent is not done after a set time, it stops and writes down where it got stuck. An agent running in circles for two hours costs money and delivers nothing.
One night produces four to eight proposals here. More would be technically possible. The ceiling is my review time in the morning, the agents could do more.
7:40 am: reviewing and approving
Here is the part people rarely talk about. Reviewing costs real time.
A medium-sized proposal takes 5 to 15 minutes. At six proposals that is around an hour in the morning. That hour does not disappear, however good the system gets. I am shifting work from writing to judging.
What I look at when reviewing, in this order:
- Does it solve the task I set? Surprisingly often the proposal solves a similar but different task.
- What else did it touch? Changes in places the task never mentioned are the most common warning sign.
- Are the tests real? A test that only checks that the function does not crash looks green and measures nothing.
- Do I understand it within a minute? If no, it goes back. Code nobody understands is a debt against tomorrow.
Out of six proposals, on average three to four go through here, one goes back with a correction note, one gets discarded.
What works well regularly
After a few months I see a clear pattern in which tasks run through cleanly overnight.
| Type of task | Hit rate | Why it works |
|---|---|---|
| Writing tests for existing code | high | The goal sits in the code itself |
| Applying one pattern across many files | high | Repetition without judgement |
| Updating dependencies, fixing errors | high | The feedback is unambiguous |
| Fixing small, clearly bounded bugs | medium to high | The bug report is the task |
| Making interface texts consistent | medium | Lots of diligence, few decisions |
The common denominator: there is unambiguous feedback on whether the work was right. Where a test or an error message supplies the answer, an agent works reliably and without tiring. That is exactly the reason we build our own tools in the first place.
What fails regularly
And now the opposite. Four kinds of task that almost always come back here.
Anything that contains a decision. When the task says "find a good way to solve X", the agent delivers a way. Usually the first one it finds. It does not weigh up alternatives it does not know about.
Anything touching several systems at once. As soon as one change has to line up in three places, the hit rate collapses. The agent builds two places cleanly and forgets the third.
Design. An interface that works comes out of it. One that feels right does not. Spacing, hierarchy and calm are almost always missing.
Anything with unspoken context. "The way we usually do it" is written down nowhere. What is not written down does not exist for the agent. That is at the same time the strongest side effect of the whole system: it forces us to write down rules that previously only lived in people's heads.
The numbers after half a year
What the system delivers is harder to measure than a support automation. Still, a few solid figures from our operation.
- Most proposed changes get approved, some after a correction.
- The time gain is much higher on routine work than on new features.
- My work has shifted: less typing, more formulating and judging tasks.
- Preparation and review together take one to two hours a day.
One caveat: these numbers apply to our own system, on our own code, with rules we wrote over months. They are no promise for your first month.
The limits and the mistakes we made
Three things went wrong that I did not see coming.
Too many proposals at once. For a while twelve tasks ran per night. In the morning ten proposals were waiting, and from the sixth onwards I reviewed superficially. That is exactly when a mistake slipped through. Since then there is an upper limit, oriented on my review time and not on the capacity of the machine.
Green checks with no meaning. We had an automated test that was supposed to check links. It ran green for months and checked exactly zero links, because it loaded the page wrongly. A test that finds nothing looks the same as a test that has nothing to find. Both are green.
Plausible-sounding nonsense. An agent can write a justification that sounds coherent and is factually wrong. With code that shows up through tests. With texts and summaries it only shows up if somebody checks. Why this happens is in avoiding AI hallucinations.
Plus one restriction that stays: the system makes good developers faster. It makes nobody a developer. If you cannot judge the proposal, you either approve it blindly or not at all. Both are bad.
Conclusion
A day with AI agents looks unspectacular. 30 minutes writing tasks in the evening, an hour reviewing in the morning, and in between work happens that would otherwise sit around during the day.
The gain sits in the tasks that always used to get postponed. Tests, cleanup, updates. None of it is exciting, and that is why it stayed undone.
If you want to try something like this, start with the tasks and not with the tool. For one week, write down three tasks every evening precisely enough that a stranger could work on them without asking a question. If you find that hard, that is the actual construction site. How a project like this runs with us is in an automation project: the process.
If you are wondering where AI agents really carry weight in your operation: talk to us at Flowhouse. We will also tell you where a simple workflow is the better answer.