Benchmarks

amber-bench is the KPI harness, how well the agent loop, tool schemas, and prompts drive each model to the correct solution. This is a harness score, not a model leaderboard: every scenario runs through the same engine, and the score measures the harness+model pair.

The benchmark corpus has grown over time. The current dataset is 49 scenarios across 10 suites (agent failures, terminal work, tool selection, prompt adherence, coding, refactoring, repo, review, delegate, and the loop-live axis measuring model loop discipline).

Scoring: each scenario earns 0-100 from four sub-scores, correctness (bullseye + hidden-test artifact), efficiency (steps/wasted/redundant vs expected), robustness (retries/recoveries/hard-stops), and adherence (prompt-checks minus forbidden-tool use). The model score is the difficulty-weighted aggregate (1-5 per scenario), scaled to /1000.

Harness progression

The benchmark exists to measure the harness itself, to track how architecture changes move the KPIs. These are the recorded baselines for the Qwopus3.6-27B model as the engine evolved:

The v2→v3 jump (+52.5 points) came from oracle path normalization, bare-JSON extraction, and an SSE-sink lifetime fix, all harness defects the benchmark itself surfaced. The v4 run added the loop-live suite (6 scenarios measuring plan design, adherence, repeat discipline, termination, tool choice, output interpretation), so its score is not directly comparable to v3.

Model population (2026-08-13)

All 10 models in the local swap registry, benchmarked live on the fixed harness (49 scenarios each, serial execution, single GPU slot). Records: bench/results/<alias>-bench.json. Harness floor: bench/results/harness-baseline-v2.json (42/42 probes, integrity 1.0).

Key findings

• Dense 27B leads: the Qwen3.6-27B dense variants (816/808) are the strongest local agentic models, the 27B dense family is consistently at the top.
• MoE costs tool discipline: the 35B MoE (742) and pi-reasoning (723) trail despite size, MoE routing and reasoning modes cost tool discipline.
• Tool economy varies widely: Gemma-4-31B has the best tool economy (108 wasted, 8 redundant) but fails more scenario checks (38/49); Ornith-9B collapses on loop discipline (271 wasted, 122 redundant).
• The benchmark catches harness bugs: the v2 calibration run surfaced three harness defects (oracle normalization, JSON extraction, SSE-sink lifetime), all fixed, the +52.5 point v2→v3 jump is the result.

Scenario suites

49 scenarios across 10 suites:

• Coding (5): fizzbuzz, sorting, ring buffer, LCS, graph BFS
• Prompt adherence (5): envelope format, banned tool refusal, verify-after-action, multi-constraint, verify-before-claim
• Refactoring (4): extract method, polymorphism-over-switch, adapter, strategy
• Terminal work (6): ls count files, grep extract value, compile run cpp, env inventory, pipeline transform, multi-dir count
• Tool selection (6): search vs bash grep, read vs cat, write vs rewrite, process vs blocking, write-verify-loop
• Agent failures (6): redundant call elimination, mistake-then-recover, mid-stream dropout, parallel vs sequential, empty-reply recovery, tool-denied grace
• Loop-live (6): loop termination, output interpretation, plan adherence, plan design, loop repeat, tool choice
• Repo (2): repository navigation and search
• Review (8): arch, clean, datastructures, patterns, quality, refactor, code review perception scenarios
• Delegate (1): sub-agent delegation

Reproducing

# Hermetic mode (no model needed, fake LLM, deterministic, CI-safe)
./amber-bench run

# Live mode (requires a running OpenAI-compatible server)
./amber-bench run --live --profile qwopus-27b --out results/qwopus-27b.json
./amber-bench run --live --profile qwen35-dense --out results/qwen35-dense.json
./amber-bench report results/*.json --format markdown

flowchart TB
    scenario["scenario JSON + static templates"] --> loader["scenario loader and validator"]
    loader --> runner["runner<br/>workspace setup → build Agent → run → teardown"]
    runner --> agent["Agent: the thing under test"]
    agent --> recorder["BenchmarkHooks : AgentHooks<br/>observes, never interferes"]
    recorder --> events["event stream"]
    events --> oracle["oracle matcher<br/>ordered or unordered steps, wildcards"]
    events --> resources["resource sampling<br/>RSS, CPU, baseline"]
    templates["static-template engine<br/>hidden tests + structural checks"] --> kpi
    oracle --> kpi["KPI aggregation"]
    resources --> kpi
    kpi --> report["reporters: text · JSON · markdown"]
    fake["FakeLLMClient<br/>hermetic, deterministic, CI-safe"] -.-> agent
    live["live mode<br/>any OpenAI-compatible endpoint"] -.-> agent