Glossary
Guardrail backtesting
Replaying a proposed guardrail against conversations you've already served, to see what it would have caught and what it would have wrongly blocked before it enforces anything.
Guardrail backtesting is testing a new guardrail against real traffic you have already served, before the rule is allowed to block anything.
A backtest answers two questions with real numbers:
- What would it have caught? The past conversations where the rule would have fired on a genuine violation.
- What would it have wrongly blocked? The legitimate requests it would have stopped, which is the cost of shipping it.
Seeing the actual conversation behind each result matters as much as the counts. A rule that "caught 47" is only useful if those 47 are the failures you meant.
Why it matters
Most teams write very few guardrails, because a rule shipped blind can block real customers. Synthetic test cases don't help much: they only contain the cases someone thought of. Production traffic contains the ones nobody did.
In Zespan a guardrail moves through three stages, from dry run (log what it would have done) to warn to block, and it can only start blocking after it passes its backtest:
$ zespan policy test --against last:7d
For a longer walkthrough, see how to backtest guardrails.