What Zespan Does: Catching the Silent Failures Your AI Agents Make in Production

Your AI agent didn't crash. It failed silently.

A customer asks for a refund on order 4821. The agent replies: "Your refund has been issued." The run completes in 1.4 seconds. The trace is green, the error rate is flat, and every dashboard you own agrees that everything is fine.

The refund tool was never called.

You find out three days later, from the customer, in a support ticket that starts with "your bot lied to me". That gap, between a run that succeeded and a run that did the right thing, is the problem Zespan exists to close.

This post is the whole picture: what goes wrong, why your current tools don't see it, and how Zespan works from the first trace to a failure that stays fixed.

The problem: agents fail without throwing

Traditional software fails loudly. An exception is raised, a status code turns red, someone gets paged. Agents mostly don't fail like that. They fail by producing a well-formed, confident, wrong outcome, and every system around them reports success.

We sort these failures into two kinds, because they are caught in different ways.

Technical failures that never throw. How the agent ran:

• Loops and hidden retries. search.query called 7 times in one turn, $0.42 spent, and the run still completes.
• Swallowed errors. inventory.check fails with a 503, the tool wrapper returns null, and the agent reads that as "yes, it's in stock".
• Truncated, refused or empty responses. The system prompt grew by 1,900 tokens, max_tokens is still 512, and 6.1% of replies now stop mid-sentence.
• Creep after a change. p95 latency goes from 1.1s to 8.4s after a deploy because the new retriever fetches 40 chunks instead of 8.

Functional failures that look normal. What the agent said and did:

• An action claimed with no matching call. "I've issued the refund", and refund.issue never ran.
• An answer the sources don't support. Retrieval returned last year's policy; the reply quotes the wrong return window.
• A rule from your own system prompt broken. The prompt says "never discount beyond listed plans"; the reply offers 40% off.
• A figure that doesn't match the record. The answer says Q3 revenue was $4.2M; the query returned $3.8M.

And then the most expensive one: the same failure, again. You patched the prompt last week. It's back in a slightly different shape, and nothing stopped it.

If you want the longer version of this taxonomy, we wrote it up in silent agent failures.

Why your current tools miss it

Each category of tool you might already run sees part of the picture, and none of them sees this:

• APM tools tell you a request took 2.3 seconds. They don't know the agent called the wrong tool twice getting there.
• Agent tracing tools show you every span of the run. The span for "agent replied: refund issued" looks exactly like a correct one.
• LLM observability tools record tokens, cost and latency for each model call. A grounded answer and an ungrounded one cost the same.
• Guardrail tools block content that matches known-bad patterns: prompt injection, PII, toxicity. A wrong refund isn't unsafe content. It's a valid response that is wrong for your business. (More on that distinction in guardrails beyond security.)

The data to catch these failures is usually already in the trace. What's missing is something that reads the trace the way an engineer would, checks it against what should have happened, and does something about it.

What Zespan is

Zespan is a reliability platform for AI agents. It does three things in a loop:

1. Catch. Check every trace, not just the ones that error, for technical and functional silent failures.
2. Explain. Group the repeats, name the cause and the layer that failed, and propose a fix.
3. Prevent. Turn the failure into something that stops it shipping twice: an alert rule, a regression test, an eval or a guardrail, tested on your own past traffic before it enforces anything.

The difference from observability is that last step. Seeing a failure is where Zespan starts, not where it stops.

Step 1: one trace across the whole stack

An agent run isn't one call. It's a stack, and every stage has to work for the run to land. Zespan is wired into each one, so when a stage fails silently you know which one:

Stage	What Zespan captures	What it catches there
Prompt	The rendered prompt, its version and variables, token count	A prompt rule broken by the reply; prompt growth that truncates answers; a version change that raised failures
Agent	The plan, every step, delegations between agents	Loops and hidden retries; dropped hand-offs; required steps skipped
LLM	Model, tokens, cost, latency and finish reason on every call	Truncated, refused or empty responses; cost and latency rises; drift in length, language or format
RAG	Query, chunks and scores, the version of each source	Answers not backed by the sources; old or wrong-customer context
MCP	Server, tool and arguments; which tools each agent can see	A tool missing from an agent; errors swallowed at the server
Tools	Arguments, results and errors; actions with real consequences	"I did X" with no matching call; destructive actions with no approval; tool errors the agent ignored

Setup is two lines:

pip install zespan
# or
npm install @zespan/sdk

import zespan
zespan.init(api_key="YOUR_API_KEY")

import { zespan } from "@zespan/sdk";
zespan.init({ apiKey: process.env.ZESPAN_API_KEY });

On init, Zespan patches the OpenAI, Anthropic, Gemini, Bedrock and Mistral clients, so model calls are captured with no other code changes. Agent identity and delegation take one framework line, for example callbacks=[ZespanCallbackHandler()] for LangChain or ZespanCrewAIListener() for CrewAI. It's OpenTelemetry native, so you can send the same data to Zespan, to your existing tools, or both. Every span carries agent identity, and delegations between agents are first-class spans: who handed off to whom, what context was passed, what came back, and what it cost.

Step 2: findings, not just traces

Every trace is checked. When something is wrong, you don't get a red span to go and interpret. You get a finding, written the way a senior engineer would write it in an incident channel. Here's one:

Action claimed, no call · Functional

What happened: The reply said "Your refund has been issued." refund.issue was never called.

Cause: The plan lets the model confirm before the tool step runs.

Layer: Prompt

Fix: Make the refund.issue call required before any confirmation.

Prevent: Block confirmations that have no matching tool call. (guardrail)

Every finding has the same five parts: what happened, the cause, the layer it came from, the fix, and the prevention. That last part is a concrete, typed artefact (an alert rule, a regression test, an eval or a guardrail), not advice.

Findings are also grouped by what actually went wrong. Forty conversations that promised a refund the company doesn't offer are one issue with one root cause, not forty alerts. In our pilot, that grouping is what turned 104 raw failures into 10 issues a team could actually work through. Alerts go to Slack and PagerDuty with the trace and the root cause attached, so the first person to see the alert can act on it.

Quality is scored on every trace too. LLM-as-judge evaluations check things like faithfulness, groundedness and instruction following, so a prompt regression shows up the moment it ships rather than when users complain.

Step 3: prevent it, tested on your own traffic

Most teams write very few guardrails, for a good reason: shipping a rule blind can block real customers. Zespan removes the guesswork by backtesting.

Before a guardrail enforces anything, Zespan replays it against conversations you have already served and shows you two numbers: how many real violations it would have caught, and how many legitimate requests it would have wrongly blocked. You see the real conversation behind every result.

$ zespan policy test --against last:7d

Guardrails live as code in your own repository and are reviewed in pull requests like everything else. They move through three stages, from dry run (log what it would have done) to warn to block, and a guardrail can only start blocking after it passes its test. More detail in how to backtest guardrails.

When a guardrail does enforce, it runs inside your own service. Rules compile into a bundle the SDK evaluates in-process, with no network call and no proxy in the request path. That's why it works on real-time voice calls, where an extra round trip turns into audible dead air (why that matters).

Step 4: risky actions wait for a person

Some actions shouldn't be up to the agent alone: a refund above a ceiling, a destructive database query, a discount outside policy. Zespan can hold those tool calls until a named reviewer approves or rejects them.

refund.issue   $4,000, over the $2,000 ceiling   held for finance-oncall

The decision is recorded against the person who made it, not against the agent that asked. Each audit entry shows whether a person, an API key or the system acted, and exports as SOC 2 control evidence. When someone asks "what was this agent allowed to do, and who said yes?", you have the answer. We cover this in proving what an agent was allowed to do.

Step 5: fixes you approve, and failures that stay fixed

When an incident closes, Zespan proposes a fix, for example a prompt change from v9 to v10 that states the 30-day refund rule the old prompt never mentioned, and replays it against every failure that incident captured. You see the pass rate before you decide anything: 39 of 41 captured failures now pass, 2 regressions flagged, with the conversations behind both.

Nothing ships automatically. You approve it, or you don't. Either way, the incident becomes a permanent regression test built from a real production failure, so the same mistake is caught before it ships again. That's how "we patched it last week" stops being followed by "it's back". The longer version is in how to stop an agent repeating mistakes.

Around the loop

The loop runs on a toolkit you'd expect from a production platform:

• Tracing and agent monitoring: every LLM call, tool call and hand-off, with cost and latency, plus health and delegation graphs per agent.
• Cost management: spend by agent, tool and model, with forecasts and recommendations for cheaper model swaps that keep quality.
• Simulations: replay real failures against a new prompt or model before you ship it.
• Prompt management: versions, labels and regression checks.
• ZespanPilot: ask your production data anything in plain English ("why did costs spike on Tuesday?") and act on the answer from the same conversation.
• An MCP server with 35 tools, so Claude Desktop, Cursor or any MCP client can find what's failing most, see the root cause and draft a fix. Every tool runs under your own role.

Who it's for

If an agent of yours talks to customers or touches money, data or systems, a silent mistake is expensive. The failures look different in each domain:

• Support bots: promises with no action, customers who repeat themselves and then ask for a human, conversations that get reopened. (Customer support AI)
• Banking and lending: a rate, fee or EMI that doesn't match the record, a skipped KYC or eligibility step, a missing disclosure. (Fintech)
• Insurance: cover or exclusions stated wrong, payouts that don't match the deductible and limit, denials with no reason or no human review. (Insurance)
• Finance and investing: returns stated as promised, advice with no risk disclosure, a suitability step skipped.
• Data and BI: numbers that don't match the query behind them, the wrong period, mixed currencies.
• Marketing: a campaign budget set to ten times the request, duplicate creates, updates that changed nothing.
• Voice agents: wrong prices or commitments on live calls, where you can't afford latency. (Voice agents)
• Any AI product: retry storms, ignored tool errors, secrets or personal data in answers, untrusted text driving a write, destructive actions with no approval.

Built for teams with an auditor

For security, risk and compliance teams, Zespan can run in your own environment and keep data there. It supports SSO over SAML or OIDC, role-based access on every project, approval flows for risky tool calls, an audit trail of every agent action and tool call with who approved it, SOC 2 evidence packs ready to export, bring-your-own-keys, and redaction of personal data before it reaches any model. See enterprise for the full picture.

Does it work on real traffic?

We ran a pilot with 10 agents in production through Zespan. It caught 104 silent failures, none of which raised an error. Grouped by cause, they were 10 real issues, and 2 of them were turned into permanent tests so they can't come back. The full write-up is in the pilot case study.

Getting started

The free plan has no time limit and needs no credit card: 50,000 traces a month, 14-day retention, silent-failure detection, issue clustering and pre-built guardrails. Paid plans start at $29 a month and keep ingesting past their included traces instead of stopping, billing overage at $2.50 per 50,000 traces.

Install the SDK, add two lines, and point it at an agent you already run in production. Within minutes you'll see every trace, and within a day you'll probably see the first thing your agent has been getting wrong that no dashboard told you about.

Start free, or book a 20-minute walkthrough and we'll show it on your own agents.