Same agent. Different model. Different behavior. Agent fleets swap models constantly — cost, availability, routing. Agent Verify certifies the swap before your users find out the hard way.
Multi-model agent platforms re-route the model beneath an agent every day. The prompt is identical — the behavior is not: different tools called, malformed arguments, broken JSON, silent refusals. And the research is clear on why single-judge spot-checks don't catch it:
One task suite. One baseline. Three swap candidates racing in parallel. Watch every tool call, argument, and schema check — then the panel verdict. This replay runs on labeled sample data; the same console renders your fleet's real run.
Why a panel? The Inference Index research put identical outputs in front of 10 independent judges: preservation scores ranged from 61% to 97%. One judge is an opinion. A panel with a disclosed spread is a measurement.
| Candidate | Tool Fidelity | Arg Validity | Schema | Refusals | Equivalence | Latency | Verdict |
|---|---|---|---|---|---|---|---|
| loading report… | |||||||
CERTIFIED = tool fidelity ≥95% · equivalence median ≥4 · zero refusals · FAIL = tool fidelity <80% · CAUTION = everything between. The CLI exits non-zero on any FAIL — drop it in CI ahead of every routing change.
Agent Verify speaks the protocol your fleet already speaks: any OpenAI-compatible endpoint. Zero dependencies, one command, report in minutes.
git clone https://github.com/AmperesAI/inference-index && cd inference-index python3 pipeline/agent_verify.py \ --tasks pipeline/agent_tasks.jsonl \ --model "baseline|https://api.openai.com/v1|gpt-5|OPENAI_API_KEY" \ --model "swap-1|https://api.tokenfactory.nebius.com/v1|meta-llama/Meta-Llama-3.1-8B-Instruct-fast|NEBIUS_API_KEY" \ --model "swap-2|https://<workspace>.cloud.databricks.com/serving-endpoints|databricks-meta-llama-3-1-8b-instruct|DATABRICKS_TOKEN" \ --judge "judge-1|https://generativelanguage.googleapis.com/v1beta/openai|gemini-2.5-pro|GEMINI_API_KEY" \ --judge "judge-2|https://api.tokenfactory.nebius.com/v1|Qwen/Qwen3-235B-A22B-Instruct-2507|NEBIUS_API_KEY" \ --judge "judge-3|https://api.mistral.ai/v1|mistral-large-3|MISTRAL_API_KEY" \ --out docs/agent-verify-report.json
Single stdlib-only Python file — no pip install. A 2-candidate, 3-judge run ≈ 100 small API calls, a few dollars, ~2–5 minutes. Repeat --judge to grow the panel; verdicts always report median + spread.
Ten agent-shaped tasks ship in the suite — tool calls, multi-arg bookings, JSON extraction, a no-tool distractor. One JSONL line per task; your existing tool schemas paste straight in:
{"id":"weather-tool","user":"What's the weather in San Francisco right now?",
"tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather for a city",
"parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
"expect":{"tool":"get_weather","required_args":["city"]}}
Swap in traces from your own agents for a fleet-specific certification. The index prices the market. Agent Verify certifies the swap. Same panel methodology.