To monitor AI agents in production is to watch what they actually did, not what they said they did. One in five financial firms runs agents with no monitoring at all, while the fleets keep growing. This post sets out the three levels that catch the failures a dashboard misses.
TL;DR
• New Relic found 22% of financial firms run AI agents in production with no monitoring.
• Komodor found nearly 84% of engineering organisations run fleets of 10 or more agents.
• Watch three levels: the service, the run, and the outcome in the system of record.
• Trace the reasoning, check the outcome, and track the cost per agent.
One in five financial firms runs agents with no monitoring
New Relic released its 2026 Observability Forecast for Financial Services (https://newrelic.com/press-release/20261020-0?has_gdpr=true) on October 20, 2026. The report, produced with Enterprise Technology Research, draws on 278 financial services IT and engineering leaders inside a global study of 2,575 respondents. Its headline finding is blunt: 22% of financial services companies run AI agents in production without monitoring them.
That is the finding to sit with before anything else in this post, because it describes firms that move money under rules like the EU Digital Operational Resilience Act, which took full effect earlier in 2026. That is why our AI custom automations work (https://buzzintechhub.com/services/ai-custom-automations.php) starts from one premise: an agent that touches a live system needs a record of what it did, kept somewhere it cannot edit.
The same report gives the other side of the picture. Today, 42% of financial firms monitor AI applications and 38% monitor agentic systems, against 47% and 44% globally.
Over the next 12 months, those rates are projected to reach 83% and 75%. The gap is closing, but the firms in the 22% are running live agents now, and the report calls unmonitored agents its single highest-risk finding.
New Relic gives the gap a name: phantom velocity. The productivity an agent adds gets cancelled by the downtime and emergency fixes the same system creates, and nobody sees the trade because nobody watches the runs.
A team that ships agents without monitoring is not moving faster. It is moving blind, and the bill arrives as an incident, a failed audit, or a customer complaint.
Fleets are already the norm
Komodor released its 2026 Agentic Operations Survey (https://www.globenewswire.com/news-release/2026/11/04/3395406/0/en/komodor-survey-finds-more-than-50-of-engineering-organizations-face-control-and-governance-issues-as-they-scale-agentic-operations.html) on November 4, 2026, based on more than 1,000 engineering and technology leaders, over 45% at director level or above, and about 85% in organisations with 100 or more engineers. Nearly 84% of them run fleets of 10 or more operational agents.
The single agent with one job is no longer the picture. The picture is a dozen agents, each with its own tools, each writing to systems that matter.
The survey names the strain that follows. Building safe guardrails is the most-cited challenge at 53.6%, then governance and security at 50.4%, and standardisation and accuracy at 45.9%. Another 42.7% say they lack visibility into agent permissions.
Three quarters, 75.7%, keep some restriction on autonomous work, such as read-only operation, human approval, or non-critical workloads only. The work itself is ordinary: routine task automation leads at 49.1%, then support ticket triage at 47.1%, self-remediation at 44.2%, and incident investigation at 43.2%.
One limit stays attached to every Komodor figure in this post. Komodor sells an agentic-operations platform, so read these numbers as survey findings from a vendor in the space. They are still worth reading, because the direction matches what every other source in this post shows: more agents, more spend, and visibility that lags behind both.
The three levels: service, run, outcome
Monitoring an agent has three levels, and most teams stop at the first two. Level one is the service: is the agent up, responding, and inside its error budget?
Level two is the run: what did each step read, call, return, and cost? Level three is the outcome: did the record, the summary, or the update actually reach the system and the person who depend on it?
Traditional monitoring covers levels one and two. Outcome checks are what catch the silent failures, the runs that report success and change nothing.
The Measuring Agents in Production study (https://arxiv.org/html/2512.04123v4), published as an ICML 2026 paper, gives a useful frame for level two. It studied 20 case studies and surveyed 306 practitioners, 86 of them working on deployed systems, across 26 domains. It found that 68% of production agents run 10 steps or fewer before a person steps in, and 47% run fewer than 5 steps.
Reliability was the top reported development challenge. Humans are already in the loop most of the time. Monitoring should show them what to check.
Trace the reasoning, not just the errors
Sherlocks analysed 73 production agent incidents (https://www.sherlocks.ai/blog/why-ai-agents-fail-in-production) between January and May 2026. Tool-call failures and retrieval errors together account for over 57% of them.
The figure that matters for this post is the resolution time: incidents without reasoning traces averaged 4.2 hours to resolve, more than four times longer than incidents with full decision trace logging. A trace is the difference between an afternoon lost and a fix shipped before lunch.
A useful trace records the prompt, the tool calls with their inputs, and the decision the agent made from what came back. A log that only records failures cannot distinguish a legitimate instruction from an injected one, and it cannot show which step caused a gap.
The permission side of this problem has its own post: our AI agent guardrails piece (https://buzzintechhub.com/blog-detail.php?slug=ai-agent-guardrails-outside-the-model) covers approval gates and enforcement outside the model. This post covers the other half: once the agent is allowed to act, can you see what it did?
Datadog's State of AI Engineering 2026 report (https://www.datadoghq.com/about/latest-news/press-releases/datadog-state-of-ai-engineering-report-2026/), built on telemetry from thousands of customers, adds the error baseline. Around 5% of AI model requests failed in production in February 2026, and nearly 60% of those failures came from capacity limits. In March the error rate fell to 2%, with about a third of failures from rate limits and 8.4 million rate-limit errors in the month.
Errors are normal at this scale. Unlogged errors are the problem, because a rate limit the agent retried quietly looks exactly like a run that worked.
Three silent failures to watch for
Silent failures share one shape: the run reports success and the system of record disagrees. Three patterns cover most of them.
First, partial or stale data treated as complete. A tool returns 10 rows of 400, or a cached page from last week, and the agent writes its summary as if it saw everything. The output reads well.
The numbers are wrong. The fix is an outcome check that compares the count the tool returned against the count the system holds, before the agent writes anything downstream.
Second, outdated context carried into the next step. An old price, a superseded policy, or a ticket that already closed rides along in the conversation and shapes the next action.
The agent worked from a record that had already expired, and the next action inherited the error. The fix is to re-read the source of truth at the step that writes, not at the step that started the run.
Third, a run that succeeds while writing nothing. A CRM update that never landed, a batch of inputs left unprocessed, a notification nobody received. The log shows a clean run.
The customer shows you the gap a week later. The fix is the same in each case: check the destination. Ask the system of record whether the record exists, and alert a human when it does not.
Cost is an observable, not a surprise bill
Komodor's survey puts numbers on the spend. Over two thirds of respondents, 68.4%, spend at least $10,000 a month on AI agents. Another 38.4% spend $50,000 or more, and 12% spend $250,000 or more.
Yet 5.8% cannot say what they spend at all, and 25.8% rank inadequate cost visibility among their top three challenges. A fleet that nobody prices is a fleet nobody controls.
Treat spend as an observable, like latency or error rate. Track it per agent and per workflow, with caps and alerts on both.
A loop that retries a failing call for an hour is a billing event before it is an incident, and a cap in the orchestrator stops it the same way a step cap stops a runaway task. Our Zapier vs n8n vs Make comparison (https://buzzintechhub.com/blog-detail.php?slug=zapier-vs-n8n-vs-make-billing-unit-cost) priced the billing unit; this post is about seeing the bill while it grows.
The small-team minimum
A small team does not need a platform to start. It needs four habits. Traces on every run, kept where the agent cannot edit them.
One outcome check per workflow: did the CRM record, the invoice, or the ticket actually land? A human-approval gate on irreversible actions, the pattern our guardrails post (https://buzzintechhub.com/blog-detail.php?slug=ai-agent-guardrails-outside-the-model) sets out, used only where an action cannot be undone. And a weekly review of the traces that needed a human step-in, so the same failure gets fixed once instead of watched every week.
When an outcome check fails, recovery needs care. Restore the missing results for the affected period first, then re-run the dependent workflows with idempotency keys, so no record is written twice.
A retry that duplicates invoices or tickets trades one silent failure for a loud one. The check, the fix, and the re-run all belong in the same log the team reads each week.
The same discipline shows up in the client work on our projects page (https://buzzintechhub.com/projects.php). The Ecovers rebuild paired a custom website with GoHighLevel automation and a blog admin panel, so every quote request was captured, answered, and followed up through one governed system, with a human picking up where automation stops. Monitoring is the same idea applied to agents: the system does the routine work, and a person sees everything it did.
Do your agents leave a trail you can read?
If your agents run in production and you cannot say what each one did last week, that is the gap to close first. We build AI automations with traces, outcome checks, and human gates from the start, through our four-step method: Discovery and Architecture, Design and Prototyping, Engineering and Build, then Deployment and Scaling. See how we build AI automations with traces and checks (https://buzzintechhub.com/services/ai-custom-automations.php)
Frequently asked questions
What does it mean to monitor AI agents in production?
It means keeping a record of what each agent did in a live system: which tools it called, what came back, what it wrote, and what it cost. The record lives where the agent cannot edit it. Monitoring also covers outcome checks, which confirm that the record, invoice, or ticket the agent was meant to create actually exists in the system of record.
How is agent monitoring different from regular application monitoring?
Regular monitoring watches uptime, latency, and error rates, which is the service level. Agent monitoring adds the run level, every step and tool call, and the outcome level, whether the intended change landed. An agent can be up, fast, and error-free while writing nothing, and only the outcome level catches that.
What should an agent trace contain?
The prompt, every tool call with its inputs, the results that came back, the decision the agent made, and the cost in steps, tokens, and dollars. Store it where the agent cannot edit it, and read it on a schedule. If a dispute arises later, the trace should answer who approved what and when, without asking the agent.
How do I catch a silent failure?
Check the destination, not the log. After each workflow, ask the system of record whether the expected record exists: the CRM entry, the invoice, the processed input. Alert a human when it does not. The three patterns to watch are partial data treated as complete, outdated context carried forward, and runs that report success while writing nothing.
How much does agent monitoring cost a small team?
The tooling is the small part: trace storage and a few alerts. The real cost is one weekly review hour, reading the traces that needed a human step-in. Compare that with the spend it watches: Komodor found over two thirds of engineering organisations already spend at least $10,000 a month on agents, and a quarter cannot see what they spend.
How often should a team review agent traces?
Weekly at first, then monthly once the pattern holds. The review looks for runs that needed a human step-in, outcome checks that failed, and spend that moved without explanation. Each finding should end in a fix to the workflow, the prompt, or the check itself, so the same failure is not watched twice.
More from Buzzin Tech Hub
AI agent guardrails: enforcement outside the model (https://buzzintechhub.com/blog-detail.php?slug=ai-agent-guardrails-outside-the-model)
MCP governance: give your agents a permissions desk (https://buzzintechhub.com/blog-detail.php?slug=mcp-governance-agent-permissions-desk)
Browse the full Buzzin Tech Hub blog (https://buzzintechhub.com/blog.php)