agentxray / 0.1.0
GitHub HF Space
Overview Quickstart Guide Documentation & API Interactive Playground Research & Methodology
Technical Reference / API & CLI Specifications / Process Contracts

Complete Technical Reference

Strict process exit codes, typed Python SDK contracts, evaluator function requirements, and declarative scenario schemas for automated diagnosis pipelines.

CLI Reference

CLI Commands & Flags

The AgentXRay CLI provides command-line subcommands designed for interactive debugging and automated shell pipelines.

agentxray demo

Runs an offline demonstration scenario and displays an ANSI terminal card.

Syntax 
$ agentxray demo [--scenario <id>] [--seed <int>] [--no-color] [--json] [--save <path>]
Flag Type Default Description
--scenario <id> string currency_ast Target scenario identifier: currency_ast, trace_prompt_01, trace_schema_01, trace_env_01, trace_null_01.
--seed <int> integer 42 Deterministic random seed for partitioning consistency.
--no-color flag false Disable ANSI styling. Automatically active when NO_COLOR env var is set or output is piped.
--json flag false Output raw machine-readable JSON representation of DiagnosticResult.
--save <path> filepath None Write the complete JSON diagnosis result to disk at the specified path.

agentxray diagnose

Loads a declarative scenario JSON file and isolates the failure-inducing directive.

Syntax 
$ agentxray diagnose <scenario.json> [--seed <int>] [--max-evals <int>] [--timeout <float>] [--no-color] [--json] [--save <path>]

agentxray version

Prints package version string, Python version, and minimizer engine status.

Process Contract

Process Exit Codes

AgentXRay adheres to strict POSIX process exit codes for integration into CI/CD pipelines, test runners, and bash automation scripts.

Code Status Trigger Condition Recommended CI Action
0 CLEAN / VERIFIED A minimal failure-inducing culprit was isolated and verified (VERIFIED_CULPRIT), or baseline input passed cleanly without faults (NO_FAILURE). Pass build step. Parse isolated directive from JSON or terminal card.
1 INCONCLUSIVE Search halted without confirming 1-minimality: budget exhausted (BUDGET_EXHAUSTED), timeout exceeded (TIMEOUT), or ambiguous evaluator outcomes. Warn or fail build step depending on strictness. Inspect telemetry in saved output.
2 USAGE_ERROR Invalid arguments, missing scenario file, duplicate candidate strings, or malformed JSON schema. Handled before search commences. Fail build step immediately. Fix command flags or scenario syntax.
3 RUNTIME_ERROR The provided evaluator callable or agent harness raised an unhandled Python exception (ERROR). Fail build step. Check stack trace printed to stderr.
Python SDK

Public Python SDK Facade

The public SDK facade exposes a clean entry point agentxray.diagnose without requiring knowledge of internal research data structures.

Python Signature 
def diagnose(
    candidates: Sequence[str] | None = None,
    evaluator: Callable[[tuple[str, ...]], EvaluatorReturn] | None = None,
    *,
    elements: Sequence[str] | None = None,
    max_evaluations: int = 50,
    timeout_seconds: float = 60.0,
    seed: int = 42,
    ground_truth: Sequence[str] | None = None,
    algorithm: str = "ddmin",
) -> DiagnosticResult

DiagnosticResult Properties

Property Type Description
status DiagnosticStatus Enum status indicating the search result.
summary str Human-readable plain English summary of the diagnostic finding.
is_verified bool True if the isolated condition was counterfactually verified (removing it restored success).
is_one_minimal bool True if removing any single element from the isolated set restores clean success.
isolated_candidates tuple[str, ...] The minimal failure-inducing directive strings.
evaluations_used int Total number of agent evaluations conducted.
execution_time_seconds float Wall-clock duration in seconds.
candidate_map dict[str, str] Mapping of internal element IDs (e.g. elem_0) to raw candidate strings.
Evaluator Contract

Evaluator Callable Contract

The evaluator callable represents your test harness. It receives an active subset of candidate strings and returns whether that subset triggers a failure.

Return Type Meaning Example Usage
bool True = Failure reproduced.
False = Success / clean execution.		 return any("bad_rule" in x for x in subset)
tuple[bool, str] (is_failure, diagnostic_message). Allows passing failure output for logs. return (True, "SyntaxError in tool call")
TargetOutcome Explicit enum: TargetOutcome.FAILURE, TargetOutcome.SUCCESS, TargetOutcome.INCONCLUSIVE. return TargetOutcome.FAILURE
TargetResult Full structured outcome object with outcome, output_message, and error_code. return TargetResult(...)
Outcome Semantics

DiagnosticStatus Enum

Machine-readable outcome statuses returned in DiagnosticResult.status.

Status Value Semantics
VERIFIED_CULPRIT A minimal failure-inducing subset was isolated and proven via counterfactual verification (removing it restores clean execution).
NO_FAILURE The full candidate set passed without failure; no error condition was present to minimize.
INCONCLUSIVE The evaluator returned inconclusive outcomes or candidate search halted without verifying 1-minimality.
BUDGET_EXHAUSTED The maximum evaluation budget (max_evaluations) was reached before search could verify minimality.
TIMEOUT Search exceeded the configured wall-clock timeout (timeout_seconds).
ERROR The evaluator callable or harness threw an unhandled Python exception during execution.
Declarative Schema

Scenario JSON File Specification

Standard JSON schema for declarative scenarios loaded by agentxray diagnose.

Scenario File Schema 
{
  "$schema": "https://agentxray.dev/schemas/scenario.v1.json",
  "name": "string (required, unique scenario identifier)",
  "description": "string (optional summary)",
  "candidates": [
    "string (unique candidate directive 1)",
    "string (unique candidate directive 2)"
  ],
  "failure_condition": {
    "type": "contains_any | contains_all | exact_match | always_fail | never_fail",
    "patterns": ["substring pattern to match against active candidates"]
  },
  "ground_truth": ["optional expected minimal failure subset"]
}
Testbed Catalog

Built-in Benchmark Catalog

Self-contained offline scenarios included in the package for testing, calibration, and demonstrations.

Scenario ID Category Fault Description
currency_ast Prompt Formatting / Syntax Directive injects currency symbols into tool arguments, violating AST arithmetic parser.
trace_prompt_01 Prompt Steering / Policy Directive instructs agent to calculate mentally, breaching required tool-use policy.
trace_schema_01 Tool Interface Contract Directive changes parameter key from expression to expr, causing API rejection.
trace_env_01 Environment / Infrastructure Mock service returns HTTP 500 error, proving failure is external downtime rather than agent prompt flaw.
trace_null_01 Control / Null Baseline Clean candidate pool with zero planted faults; verifies NO_FAILURE control behavior.
Boundaries

Operational Boundaries & Disclosures

AgentXRay is a causal diagnostic instrument designed for deterministic and black-box verification:

  • Black-Box Guarantee: AgentXRay does not inspect model weights, internal activation vectors, or proprietary attention heads.
  • Zero Cloud Spend in Testbeds: All built-in benchmark tasks run completely in memory without making external network requests.
  • Stochastic Agents: When evaluating live non-deterministic LLMs, repeated trials are required to establish statistical confidence (see Research Methodology).