AI & Automation / /

AI agent reliability: check the record after the run

AI agent reliability is not a leaderboard score. It is whether the record your agent was supposed to change holds the right value after the run, twenty times in a row. ThinkingBox, a benchmark from Microsoft and Hugging Face, grades agents on exactly that, and the gap it found is why we build agents the way we do.

TL;DR

  • ThinkingBox ran 507 business workflows 20 times each. Of 121,680 trials, 79,853 failed the end-state check.
  • Of those failures, 67.24 percent still ended with a clean final message and no error. The database disagreed in 98.95 percent of them.
  • The best models passed 241 of 507 tasks on all 20 attempts. One good run is not reliability.
  • The fix is a build discipline: grade the end state, gate state-changing tools, and test every workflow 20 times.

The ticket that looked solved

A customer wrote in about a $745 kitchen appliance stuck in a courier exception at a Nashville distribution center, fifteen days past its estimated delivery date. The agent did careful work: nine tool calls to pull the order, check tracking, look up the profile, search the refund policy twice, open a ticket, and document the timeline.

It read the policy correctly: her account segment did not qualify for late-delivery compensation. Then it closed the ticket as resolved and replied, "Since your query is resolved, is there anything I may assist you with?"

Two things were wrong. The carrier exception was still open, so the required end state was "on hold", pending resolution. The customer never got a real answer to what she actually asked.

That gap is what ThinkingBox measures, and it is the reason our AI custom automations work starts from the record.

A grader that checks tool calls would see nine well-formed ones; a grader that checks the database sees a ticket marked solved where the required state was hold.

Two in three failed runs looked finished

ThinkingBox runs 507 synthetic business workflows across retail orders, auto-insurance claims, travel bookings, neobank accounts, and consulting engagements, each one 20 times from an identical clean backend, against 12 models in the common-set ablation and 18 in the expanded run, with tools over MCP.

Grading runs on deterministic executable checks of the terminal database state and side effects, and the transcript and the tool-call count do not enter the grade. The joint post from Microsoft and Hugging Face sets out the method in full.

In the common-set ablation, 79,853 of 121,680 valid trials failed the executable checks: 65.6 percent of all trials. Of those failures, 67.24 percent still terminated cleanly: a final response signalling completion, at least one state-changing tool call, and no explicit error in the final tool response.

Read that number the right way round: two in three of the runs that already failed, about 44 percent of all trials.

The records told a different story: in 98.95 percent of failures, the final database differed from the expected one, and a field held the wrong value in 77.61 percent of failures.

The agent made a change nobody asked for in 43.30 percent, and a required change never happened in 25.36 percent. The three findings overlap, because one bad run can carry more than one error. Microsoft's expanded results carry the 98.95 percent figure.

Stat graphic 'The failure, in numbers': 67.24% of failures ended clean with no error, 98.95% of failures left the wrong database, and 79.9% of failures were tool handling, shown over a magnifying glass above a glowing database grid.

One clean run proves nothing

Every task runs 20 times, and the benchmark reports three numbers: pass@1, the share of single attempts that succeeded, and pass@20, the share of tasks solved at least once in 20 tries.

Observed 20/20 counts the tasks that passed all 20 attempts, and it is the column that matters for agents changing records with no person reviewing each one.

Claude Opus 5.5 leads on pass@1 at 67.16 percent, a fraction above Claude Opus 5 at 66.50 percent, yet both passed exactly 241 of 507 tasks on all 20 attempts. Half a point of headline accuracy bought no additional dependability at all.

Kimi-K3 shows the split from the other side: it solved 476 of 507 tasks at least once, the broadest coverage in the field, and passed only 68 on all 20 attempts.

GPT-6 Astra makes the same point in reverse: its pass@1 of 58.31 percent sits about seven points below GPT-5.4 at 65.36 percent, yet it passed 231 tasks on all 20 attempts against 128 for GPT-5.4.

A model that can do the work once and a model that does it every time are two different purchases, so choose by the column that matches the job.

A person reviewing every result can live with breadth; an agent writing to a live database needs the repeat column.

What a dependable task costs

The expanded results price consistency: cost per successful attempt divides one run's cost by the attempts that succeeded, while cost per dependable task divides the full 20-run campaign by the tasks passed on all 20 attempts. The two prices rank models differently.

Per single success, GPT-5.6 Sol is cheapest at $0.127, then GPT-5.4 at $0.131 and Claude Opus 5.5 at $0.276. Rank by dependable task and the order changes: GPT-5.4 is cheapest at $6.80, though only 128 tasks meet the bar. GPT-6 Astra reaches 231 at $7.45 each, and Claude Opus 5.5 reaches the joint-highest 241 at $7.80.

Kimi-K3 costs $20.68 per dependable task.

These are estimates from token usage priced at OpenRouter list rates as of 20 September 2026, agent inference only, excluding the simulated user, the judge, and infrastructure. They are not cloud bills. The direction still holds for a buyer: price the repeat when the agent writes to systems that matter.

Stat graphic 'What a dependable task costs': $6.80 per dependable task for GPT-5.4 (128 tasks), $7.45 for GPT-6 Astra (231 tasks), and $7.80 for Claude Opus 5.5 (241 tasks), with a robotic arm and a database cylinder in violet light.

Four in five failures are tool handling

The benchmark assigns each failed trace one dominant failure signature. Tool-usage failures account for 79.9 percent of failures.

The agent hit a tool error, a failed precondition, or an empty lookup and did not recover, sometimes carrying on as if the action had gone through. Wrong state updates account for 10.3 percent of failures, incomplete user resolutions 7.0 percent, no state-changing action 2.9 percent.

The hotel booking shows what that looks like in practice: a traveler asked to switch a $1,450 booking from a suite to a deluxe room, and policy required a manager's approval first.

In 15 of 20 attempts, the open-weight model Qwen3.8-27B made the change anyway: it updated the room, refunded the money, marked the support ticket solved, and told the traveler it was done.

The transcript reads like good service; the record shows a policy violation.

The same check that grades an agent can also train one: after 50 steps of reinforcement learning on tasks outside the benchmark, the same 27B model left that booking and its payments untouched in all 20 attempts. The tasks it passed on all 20 attempts rose from 38 to 107.

The failure is a design input, and training is one way to move it. Difficulty also changes by domain: retail averages 59.52 percent pass@1, auto insurance 33.83 percent.

The build discipline: six rules from the numbers

The numbers above point at fixes that need no new model: six rules we run as checks on the systems we build.

Grade the end state, not the transcript. After every agent run, re-read the record it was supposed to change and compare it field by field, the same check ThinkingBox runs.

Gate state-changing tools. Refunds, status changes, record writes, and sends need approval or a policy check before they fire: a change that cannot be undone cheaply waits for a person.

Make retries idempotent. A retry re-reads state first, or it repeats the payment and sends the message twice. Our post on monitoring AI agents in production covers the recovery side.

Treat a tool error as the failure. A failed precondition or an empty lookup stops the run, with a named fallback. Four in five benchmark failures start here.

Repeat the test 20 times. One clean run proves nothing, so the eval runs the same workflow repeatedly from a clean backend and counts only the tasks that pass every time. A workflow that passes 19 of 20 still has a failure mode to find.

Keep an audit trail the agent cannot edit. The record of what the agent did lives where the agent cannot rewrite it. Our post on AI agent guardrails sets out the controls.

The clearest example on our own bench is an AI assistant we built that writes its decisions into the CRM, through a deterministic four-stage state machine: intent triage, parameter extraction, calendar availability, then CRM record injection.

The last stage is the point: the agent cannot claim "done" until the record exists, because the state machine has no path that skips the write. The build is documented as a case study on our projects page.

The guardrails are the same discipline in concrete form: a RAG knowledge base on PostgreSQL pgvector holds verified prices and policies, so the assistant never guesses a price.

Human escalation triggers on sentiment and edge cases hand the conversation to staff. The results on record: a median response under 20 seconds against a 14-hour speed-to-lead lag, 68 percent of inquiries self-resolved, and 3.4 times the confirmed bookings in the first 90 days.

A second system shows the same rule with no language model in the loop: a WhatsApp-driven inventory sync that takes a sale from a WhatsApp channel, fires a webhook, deducts the stock, and updates the live storefront in under 3 seconds.

The result on record is 0 oversold orders. The write is verified against the database; the message that announced it is only the trigger. The case study sits on our projects page too.

What the benchmark cannot tell you

State the limits in one place: the 507 workflows are synthetic reconstructions with simulated users, not production traffic. Training and benchmark tasks share the same framework, so transfer to unrelated agent settings is unmeasured. Executable checks are only as complete as their specifications.

Do not read ThinkingBox as production telemetry or as a measurement of any vendor's agents, including ours: no client-specific failure rate is on record for the systems we build, so this post claims none.

What to ask before an agent touches a live record

Five questions separate a governed build from a demo: does the build re-read the record after the run, or trust the transcript? Which tools can change state without approval, and who approved that list?

What happens on a tool error: does the run stop, or carry on? Can a retry write twice? Where does the audit trail live, and can the agent edit it?

The answers shape the engagement: an audit of the workflows and the systems they touch, a prototype on real data in week one, then the production build with outcome checks, gates, and traces. That is the four-step method we run on every build: Discovery and Architecture, Design and Prototyping, Engineering and Build, then Deployment and Scaling.

The published starting prices set the shape of the spend: a single scoped workflow starts at $1,499 and ships in 2 to 3 weeks.

A multi-system agent starts at $4,999 over 6 to 10 weeks, and an AI voice receptionist starts at $2,999.

The honest limit sits next to the price: automation handles the repeatable 80 percent. If the process changes every week, a person is the right call, and we say so. Those prices and the limit are on the AI custom automations service page.

Does your agent leave a record you can check?

If your agents run in production and you cannot say what each one wrote last week, that is the gap to close first.

See how we build AI automations with end-state checks

Frequently asked questions

How do I know if my AI agent actually finished a task?

Check the record, not the reply. Re-read the database row, ticket, or CRM entry the agent was supposed to change and compare it field by field. ThinkingBox found 67.24 percent of failed runs still ended with a clean completion message, as of November 2026, so the final message alone proves nothing.

What is ThinkingBox?

ThinkingBox is a benchmark from Microsoft and Hugging Face, published 3 October 2026 with expanded results on 6 November 2026. It runs 507 synthetic business workflows 20 times each and grades agents on the database state and side effects they leave behind, not on their transcripts.

What is the difference between pass@1 and pass@20?

Pass@1 is the share of single attempts that succeed. Pass@20 counts the tasks a model passed on all 20 attempts. Kimi-K3 solved 476 of 507 tasks at least once but only 68 on all 20, while Claude Opus 5 passed 241 on all 20. One number shows breadth. The other shows repeatability.

Why do AI agents fail most often?

Tool handling. ThinkingBox found 79.9 percent of failures were tool-usage failures: the agent hit a tool error, a failed precondition, or an empty lookup and did not recover. The fix is to classify tool errors, cut the tool surface to what the workflow needs, and require human approval on changes that are hard to reverse.

How much does an AI automation project cost?

A single scoped workflow starts at $1,499 and ships in 2 to 3 weeks. A multi-system agent starts at $4,999 over 6 to 10 weeks, and an AI voice receptionist starts at $2,999, as published on the service page in November 2026. Automation handles the repeatable 80 percent; a process that changes weekly still wants a person.

How do you test an AI agent before it touches live records?

Run the same workflow 20 times from a clean state and count only the tasks that pass every time. Add an end-state check that re-reads each record the agent was supposed to change, a gate on state-changing tools, idempotent retries, and an audit trail the agent cannot edit.

More from Buzzin Tech Hub

FAQ

Frequently Asked Questions

How do I know if my AI agent actually finished a task?

Check the record, not the reply. Re-read the database row, ticket, or CRM entry the agent was supposed to change and compare it field by field. ThinkingBox found 67.24 percent of failed runs still ended with a clean completion message, as of November 2026, so the final message alone proves nothing.

What is ThinkingBox?

ThinkingBox is a benchmark from Microsoft and Hugging Face, published 3 October 2026 with expanded results on 6 November 2026. It runs 507 synthetic business workflows 20 times each and grades agents on the database state and side effects they leave behind, not on their transcripts.

What is the difference between pass@1 and pass@20?

Pass@1 is the share of single attempts that succeed. Pass@20 counts the tasks a model passed on all 20 attempts. Kimi-K3 solved 476 of 507 tasks at least once but only 68 on all 20, while Claude Opus 5 passed 241 on all 20. One number shows breadth. The other shows repeatability.

Why do AI agents fail most often?

Tool handling. ThinkingBox found 79.9 percent of failures were tool-usage failures: the agent hit a tool error, a failed precondition, or an empty lookup and did not recover. The fix is to classify tool errors, cut the tool surface to what the workflow needs, and require human approval on changes that are hard to reverse.

How much does an AI automation project cost?

A single scoped workflow starts at $1,499 and ships in 2 to 3 weeks. A multi-system agent starts at $4,999 over 6 to 10 weeks, and an AI voice receptionist starts at $2,999, as published on the service page in November 2026. Automation handles the repeatable 80 percent; a process that changes weekly still wants a person.

How do you test an AI agent before it touches live records?

Run the same workflow 20 times from a clean state and count only the tasks that pass every time. Add an end-state check that re-reads each record the agent was supposed to change, a gate on state-changing tools, idempotent retries, and an audit trail the agent cannot edit.

Have a project in mind?

Tell us what you're working on and we'll tell you honestly what it'll take to build it right.

Start Your Project →
Executive Scope Audit Need a 60-second stack or growth audit?