Every weekday morning at 6am, a system we built writes, voices, and renders a short video with no human touching it. Most days it works. One day it didn't, and the way it failed — and what we changed afterward — is a better explanation of what "production AI system" actually means than anything we could just tell you.
The setup
The pipeline researches a topic, drafts a script with Claude, sends that script to an AI avatar generator to render into video, and packages it for upload. The script has to land inside a duration window — an AI avatar speaks at a roughly fixed pace, so a script that's too long produces a video that runs over, and a script that's too short feels clipped and rushed.
So there's a rule: the script has to fit under a word-count ceiling. Simple enough to write. Simple enough to be wrong.
Where it broke
One morning the script came back at 111 words against a 90-word cap. The system did exactly what it was told to do: it rejected the script and stopped. No video, no email, no output — just a failure sitting silently in a log until someone noticed hours later.
The instinct here is to just raise the ceiling. That's the wrong fix, and it's worth saying why: a hard ceiling with no recovery path isn't a rule, it's a landmine. Somewhere down the line a script comes in at whatever the new number is, and the exact same silent failure happens again — just later and more expensively.
The actual fix
The system needed a way to fail productively — to notice the problem and try to solve it before giving up. So instead of a hard reject, the pipeline now does this: if the script comes back too long, it goes back to Claude with the actual word count and a direct instruction to compress it to fit, preserving the core point. It gets two attempts before it finally throws an error and tells us. Two chances to self-correct, then an honest failure — not a silent one, and not an infinite retry loop either.
if (word_count(script) > 105) {
if (attempts < 2) {
script = claude.compress(script)
attempts++; recheck()
} else {
fail_loudly("still over length")
}
} else { proceed() }
That's a small change in the code. It's a much bigger change in what the system is. A script that fits on the first try and a script that gets compressed twice before fitting produce the same output from the outside. But only one of those systems survives contact with a Tuesday where the topic happened to need a few more words to make sense.
The general lesson
Most "AI automation" we see demoed is the happy path — the one run where everything lines up and nothing needs correcting. The actual engineering work is everything that happens when it doesn't: what does the system do when the AI output is too long, too short, malformed, or just wrong? Does it fail loudly and safely, quietly and dangerously, or does it try to fix itself first and only fail if that doesn't work?
That question — retry, or fail, or fix — has to be answered for every step in a system before it's something you can actually leave running unattended. It's the least visible part of building these things, and it's almost the entire job.
Systems like this one, built for your business.
We build production AI systems for service businesses — and write up how they actually work.
Book a free strategy call →