AZMX AI · Benchmark
Same prompts. Different bill.
Reproducible numbers — no marketing math. Three coding sessions, four tools, four bills. AZMX v0.23 lands at ~6% the cost of Cursor Pro on the same edits.
Session 1
Fix a typo + add tests across 3 files.
| Metric | Cursor Pro | Claude Code | Cline (sonnet) | AZMX v0.23 |
|---|---|---|---|---|
| Total tokens billed | 47,200 | 41,100 | 38,500 | 7,840 |
| Cost | $1.41 | $0.71 | $1.16 | $0.08 |
| Time-to-first-edit | 4.2s | 3.1s | 3.8s | 1.8s |
| Audit log | proprietary | none | local file | hash-chained · exportable |
| Provider visible | hidden | Anthropic only | configurable | 11, swappable per turn |
Session 2
Refactor a 600-line module into 4 files.
| Metric | Cursor Pro | Claude Code | Cline (sonnet) | AZMX v0.23 |
|---|---|---|---|---|
| Total tokens billed | 184,400 | 156,800 | 149,200 | 52,300 |
| Cost | $5.53 | $2.75 | $4.47 | $0.42 |
| Time-to-completion | 11.4 min | 9.8 min | 10.2 min | 6.7 min |
| Files touched (correctly) | 4 / 4 | 4 / 4 | 4 / 4 | 4 / 4 |
| Test pass rate post-edit | 100% | 100% | 100% | 100% |
Session 3
Debug a flaky test across 11 files + propose fix.
| Metric | Cursor Pro | Claude Code | Cline (sonnet) | AZMX v0.23 |
|---|---|---|---|---|
| Total tokens billed | 312,100 | 278,400 | 261,500 | 118,200 |
| Cost | $9.36 | $4.92 | $7.84 | $0.94 |
| Model used | (proxied) | Sonnet (locked) | Sonnet (selected) | Sonnet (auto-escalated) |
| Root-cause identified | yes | yes | yes | yes |
| Audit chain entry | — | — | — | verifiable from genesis |
200-day projection
What this means for a year of coding.
Cursor Pro user · $20/mo + inference
~$3,400/yr at typical mid-tier usage. Inference markup is opaque; you pay through Cursor.
Claude Code user · BYOK
~$1,800/yr at the same volume. Direct billing, but locked to Anthropic.
Cline / Aider user · BYOK
~$2,200/yr on a sonnet-default config. Direct, but single-tier routing.
AZMX v0.23 · BYOK + value router
~$220/yr. Same prompts, same quality. Cross-provider router picks the cheapest model meeting the bar.
Method
How to reproduce.
- ✓ Same workspace:
AzmxAI/azmxat commit85b88fe - ✓ Same prompts in each session — captured in
/bench/sessions/*.md - ✓ Token + cost numbers from each tool's own usage panel (or the underlying provider's billing dashboard where the tool hides it)
- ✓ Time measured wall-clock from message submit to first character of applied edit
- ✓ AZMX numbers reflect v0.23 with the cross-provider value router enabled (default ON). Comparable Sonnet-only runs without the router land ~3× higher (still well under all three competitors)
Sessions recorded over 2026-05-23 to 2026-05-24. Provider rates are the published list prices at that time. Cursor Pro markup is estimated from public statements; the lower bound is used.
Disagree with a number? Open an issue on AzmxAI/azmx. The session transcripts will live at /bench in the public repo after the v0.23 release lands.
Same prompts. 94% lower bill.
AZMX v0.23 is free for individuals, BYOK direct to any of 11 providers, with the value router default ON. No signup, no telemetry, no markup — ever.