How to monitor an AI agent in production

Nobody writes about this because nobody selling AI agents wants to talk about them failing. But agents in production do fail, and the failure mode that actually hurts is not the dramatic one — it is the quiet one.

The failure that costs you money

An agent rarely breaks on day one. Day one is when everyone is watching.

It breaks in month three, when a third-party API changes a field name, or a customer phrases something in a way nobody anticipated, or a messaging window closes and a follow-up silently never sends. The system keeps running. The dashboard, if there is one, stays green. And you find out six weeks later when someone asks why bookings dropped.

The problem was never that it failed. The problem is that nothing told you.

What to instrument

Four layers, in order of how often they save you:

Execution log. Every process run, with a timestamp, a result and enough context to understand it. Not “workflow completed” — which workflow, for which contact, with what outcome. This is the layer you will actually read.

Failure states, not just errors. An error is when something throws. A failure state is when nothing throws but the outcome was wrong: the agent answered but did not book, the follow-up was skipped because it fell outside the messaging window, the CRM write succeeded but wrote to the wrong record. These need explicit checks. They are invisible otherwise.

Escalation queue. Every conversation the agent handed to a human, with a reason. If this queue is growing and nobody is watching it, you have automated the intake and manually created a backlog.

Business metrics. Conversations, leads, bookings, conversion. Not for a report — as a canary. A 40% drop in bookings tells you something broke long before anyone traces which integration it was.

Alerts: fewer than you think

The instinct is to alert on everything. Do that and within two weeks everyone mutes the channel, which is worse than having no alerts at all.

Alert on three things:

• A process that has not run when it should have. Silence is the most under-monitored failure mode in automation.
• An escalation that nobody has picked up within your response window.
• An error rate above baseline, not a single error. One failed message is noise. Twenty in an hour is a signal.

Everything else goes in a dashboard you look at, not a notification that interrupts you.

Idempotency is a monitoring feature

This sounds like an implementation detail and it is not. If your system can process the same webhook twice and create two records, then every retry — and there will be retries — corrupts your data. Which means your metrics lie. Which means your canary stops working.

Deduplicate on a stable ID from the source system. It is twenty minutes of work and it is the difference between a dashboard you trust and one you learn to ignore.

The uncomfortable design decision

At some point you will have to decide what happens when a send fails: retry, or mark it done and move on?

Retrying forever means one invalid phone number gets attempted every thirty minutes for the rest of time. Marking it done means some messages silently never arrive.

There is no clean answer. Our default is to mark it as attempted and record the failure in the execution log, so a bad number does not generate infinite noise but the failure is still visible to anyone who looks. What matters is that this is a decision someone made deliberately, not something that happened by accident.

That is really the whole argument for monitoring: not that it prevents failures, but that it converts them from surprises into decisions.