Inference Index  ·  Mission: Model Swap Certification

Agent
Verify

Same agent. Different model. Different behavior. Agent fleets swap models constantly — cost, availability, routing. Agent Verify certifies the swap before your users find out the hard way.

Certification Report
▼ SCROLL

The Problem

Swaps ship blind

Multi-model agent platforms re-route the model beneath an agent every day. The prompt is identical — the behavior is not: different tools called, malformed arguments, broken JSON, silent refusals. And the research is clear on why single-judge spot-checks don't catch it:

61–97%
quality-preservation range across 10 judges on identical outputs
10
judges validated the methodology — Inference Index research
0
standard, portable pre-swap certification today — every team hand-rolls its own
5
behavioral dimensions certified per swap candidate

Live Simulation

Mission Control

One task suite. One baseline. Three swap candidates racing in parallel. Watch every tool call, argument, and schema check — then the panel verdict. This replay runs on labeled sample data; the same console renders your fleet's real run.

T+00:00.0 STANDBY — awaiting launch command SIMULATION · SAMPLE DATA

Methodology

Five dimensions. One verdict.

01
Tool Fidelity
Calls the same tools the baseline called — and no phantom ones. The distractor task catches over-eager callers.
02
Arg Validity
Tool arguments parse as JSON and contain every required key.
03
Schema Validity
JSON-output tasks return parseable, correctly-shaped objects.
04
Non-Refusal
No refusals or empty outputs on benign tasks.
05
Equivalence · Panel
A panel of judge models — recommend ≥3, from different families — scores candidate vs baseline 1–5, reported as median + spread, never one judge.

Why a panel? The Inference Index research put identical outputs in front of 10 independent judges: preservation scores ranged from 61% to 97%. One judge is an opinion. A panel with a disclosed spread is a measurement.


Output

Swap Certification Report

CandidateTool FidelityArg ValiditySchema RefusalsEquivalenceLatencyVerdict
loading report…

CERTIFIED = tool fidelity ≥95% · equivalence median ≥4 · zero refusals  ·  FAIL = tool fidelity <80%  ·  CAUTION = everything between. The CLI exits non-zero on any FAIL — drop it in CI ahead of every routing change.


Integration

Point it at your fleet

Agent Verify speaks the protocol your fleet already speaks: any OpenAI-compatible endpoint. Zero dependencies, one command, report in minutes.

DATABRICKS SERVING ENDPOINTSNEBIUS TOKEN FACTORY OPENAIGROQTOGETHERvLLM / SELF-HOSTED
git clone https://github.com/AmperesAI/inference-index && cd inference-index

python3 pipeline/agent_verify.py \
  --tasks pipeline/agent_tasks.jsonl \
  --model "baseline|https://api.openai.com/v1|gpt-5|OPENAI_API_KEY" \
  --model "swap-1|https://api.tokenfactory.nebius.com/v1|meta-llama/Meta-Llama-3.1-8B-Instruct-fast|NEBIUS_API_KEY" \
  --model "swap-2|https://<workspace>.cloud.databricks.com/serving-endpoints|databricks-meta-llama-3-1-8b-instruct|DATABRICKS_TOKEN" \
  --judge "judge-1|https://generativelanguage.googleapis.com/v1beta/openai|gemini-2.5-pro|GEMINI_API_KEY" \
  --judge "judge-2|https://api.tokenfactory.nebius.com/v1|Qwen/Qwen3-235B-A22B-Instruct-2507|NEBIUS_API_KEY" \
  --judge "judge-3|https://api.mistral.ai/v1|mistral-large-3|MISTRAL_API_KEY" \
  --out docs/agent-verify-report.json

Single stdlib-only Python file — no pip install. A 2-candidate, 3-judge run ≈ 100 small API calls, a few dollars, ~2–5 minutes. Repeat --judge to grow the panel; verdicts always report median + spread.

Ten agent-shaped tasks ship in the suite — tool calls, multi-arg bookings, JSON extraction, a no-tool distractor. One JSONL line per task; your existing tool schemas paste straight in:

{"id":"weather-tool","user":"What's the weather in San Francisco right now?",
 "tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather for a city",
   "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
 "expect":{"tool":"get_weather","required_args":["city"]}}

Swap in traces from your own agents for a fleet-specific certification. The index prices the market. Agent Verify certifies the swap. Same panel methodology.