Paper
AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows
Authors
Varun Pratap Bhardwaj
Abstract
Autonomous AI agents are deployed at unprecedented scale, yet no principled methodology exists for verifying that an agent has not regressed after changes to its prompts, tools, models, or orchestration logic. We present AgentAssay, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees. Our contributions include: (1) stochastic three-valued verdicts (PASS/FAIL/INCONCLUSIVE) grounded in hypothesis testing; (2) five-dimensional agent coverage metrics; (3) agent-specific mutation testing operators; (4) metamorphic relations for agent workflows; (5) CI/CD deployment gates as statistical decision procedures; (6) behavioral fingerprinting that maps execution traces to compact vectors, enabling multivariate regression detection; (7) adaptive budget optimization calibrating trial counts to behavioral variance; and (8) trace-first offline analysis enabling zero-cost testing on production traces. Experiments across 5 models (GPT-5.2, Claude Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4), 3 scenarios, and 7,605 trials demonstrate that behavioral fingerprinting achieves 86% detection power where binary testing has 0%, SPRT reduces trials by 78%, and the full pipeline achieves 100% cost savings through trace-first analysis. Implementation: 20,000+ lines of Python, 751 tests, 10 framework adapters.
Metadata
Related papers
Fractal universe and quantum gravity made simple
Fabio Briscese, Gianluca Calcagni • 2026-03-25
POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan
Marta Moscati, Muhammad Saad Saeed, Marina Zanoni, Mubashir Noman, Rohan Kuma... • 2026-03-25
LensWalk: Agentic Video Understanding by Planning How You See in Videos
Keliang Li, Yansong Li, Hongze Shen, Mengdi Liu, Hong Chang, Shiguang Shan • 2026-03-25
Orientation Reconstruction of Proteins using Coulomb Explosions
Tomas André, Alfredo Bellisario, Nicusor Timneanu, Carl Caleman • 2026-03-25
The role of spatial context and multitask learning in the detection of organic and conventional farming systems based on Sentinel-2 time series
Jan Hemmerling, Marcel Schwieder, Philippe Rufin, Leon-Friedrich Thomas, Mire... • 2026-03-25
Raw Data (Debug)
{
"raw_xml": "<entry>\n <id>http://arxiv.org/abs/2603.02601v1</id>\n <title>AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows</title>\n <updated>2026-03-03T04:59:25Z</updated>\n <link href='https://arxiv.org/abs/2603.02601v1' rel='alternate' type='text/html'/>\n <link href='https://arxiv.org/pdf/2603.02601v1' rel='related' title='pdf' type='application/pdf'/>\n <summary>Autonomous AI agents are deployed at unprecedented scale, yet no principled methodology exists for\n verifying that an agent has not regressed after changes to its prompts, tools, models, or\n orchestration logic. We present AgentAssay, the first token-efficient framework for regression\n testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining\n rigorous statistical guarantees. Our contributions include: (1) stochastic three-valued verdicts\n (PASS/FAIL/INCONCLUSIVE) grounded in hypothesis testing; (2) five-dimensional agent coverage metrics;\n (3) agent-specific mutation testing operators; (4) metamorphic relations for agent workflows; (5)\n CI/CD deployment gates as statistical decision procedures; (6) behavioral fingerprinting that maps\n execution traces to compact vectors, enabling multivariate regression detection; (7) adaptive budget\n optimization calibrating trial counts to behavioral variance; and (8) trace-first offline analysis\n enabling zero-cost testing on production traces. Experiments across 5 models (GPT-5.2, Claude Sonnet\n 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4), 3 scenarios, and 7,605 trials demonstrate that\n behavioral fingerprinting achieves 86% detection power where binary testing has 0%, SPRT reduces\n trials by 78%, and the full pipeline achieves 100% cost savings through trace-first analysis.\n Implementation: 20,000+ lines of Python, 751 tests, 10 framework adapters.</summary>\n <category scheme='http://arxiv.org/schemas/atom' term='cs.AI'/>\n <category scheme='http://arxiv.org/schemas/atom' term='cs.SE'/>\n <published>2026-03-03T04:59:25Z</published>\n <arxiv:comment>Technical Report. 52 pages, 5 figures, 9 theorems, 42 formal definitions. Zenodo DOI: 10.5281/zenodo.18842011</arxiv:comment>\n <arxiv:primary_category term='cs.AI'/>\n <author>\n <name>Varun Pratap Bhardwaj</name>\n </author>\n <arxiv:doi>10.5281/zenodo.18842011</arxiv:doi>\n <link href='https://doi.org/10.5281/zenodo.18842011' rel='related' title='doi'/>\n </entry>"
}