Failure isolation for tool-calling agents.
Proven with counterfactual verification.
When an AI agent breaks under complex prompts, AgentXRay isolates the exact directive responsible using black-box Delta Debugging. It partitions candidates hierarchically and proves causality by verifying that removing the isolated element restores clean success—offline, in milliseconds, at $0.00 spend.
Deterministic Execution Tape
Inspect genuine Delta Debugging trajectories across five curated benchmark testbeds. Every step and verification check reflects real offline testbed execution.
Why Agent Debugging Breaks Down
Observability frameworks log what broke. AgentXRay proves which input condition caused it.
Linear Trial-and-Error Scan
When a tool-calling workflow fails, prompts often carry dozens of instructions, schema specifications, and few-shot examples. Standard log analysis reveals that an AST error or tool crash occurred, but cannot identify which directive provoked it.
Developers manually eliminate rules one by one—an $O(N)$ linear pass that wastes hours of engineering time, burns API tokens, and leaves uncertainty over whether removing an instruction actually resolved the underlying conflict.
Counterfactual Delta Debugging
AgentXRay treats the agent and tools strictly as a black-box system. It partitions the candidate directives into hierarchical subsets and complements, evaluating outcomes with logarithmic efficiency ($O(\log N)$).
When a minimal candidate subset is isolated, AgentXRay performs an explicit counterfactual ablation: proving that removing the candidate restores clean execution (1-minimality).
The 3-Phase Minimization Cycle
Candidate Pool & Evaluator
Pass candidate directives (prompts, tool parameters, or flags) and a callable that returns binary pass/fail based on observable tool execution.
Binary Partition Search
Delta Debugging partitions the active candidate pool into subsets and complements, rapidly zooming in on the active failure trigger.
Counterfactual Minimality
AgentXRay evaluates removing the isolated candidate. If removal eliminates the failure, 1-minimality is empirically confirmed.
Engineered for Code & Terminals
Import the typed Python SDK or invoke the CLI directly in test harnesses and CI pipelines.
Python SDK Facade
Pass candidates and a pass/fail evaluator callable. Unpack structured diagnosis telemetry.
from agentxray import diagnose
result = diagnose(
candidates=my_candidates,
evaluator=my_evaluator_fn,
max_evaluations=20,
)
print(result.summary)
print(result.is_verified) # True (1-minimal)
print(result.isolated_candidates) # ('strict_currency: ...',)Developer CLI
Run deterministic offline demonstrations or evaluate declarative scenario JSON specifications.
# 1. 10-second deterministic demo
$ agentxray demo
# 2. Diagnose declarative scenario JSON
$ agentxray diagnose examples/scenario_example.json
# 3. Diagnose built-in outage testbed
$ agentxray diagnose --scenario trace_env_01Truthful Boundaries & Disclosures
Strict Black-Box Boundary
AgentXRay evaluates observable tool-calling inputs and execution outputs only. It has zero access to proprietary model weights, hidden chain-of-thought tokens, or internal activations.
Deterministic vs Stochastic Models
The primary SDK and CLI engine is optimized for reproducible testbeds. Minimizing failures in live non-deterministic LLMs requires multi-trial statistical sampling with Wilson intervals, available via AgentXRay's research benchmark tier.
Integration Contract
AgentXRay does not magically intercept running third-party agents without an evaluator function. Developers provide an evaluator callable or pass declarative scenario specifications.
Zero-Cost Offline Guarantee
All benchmark testbeds and offline CLI demonstrations execute locally with $0.00 commercial API spend, zero network calls, and zero external daemon dependencies.