← Blog

Can a GLM consensus replace a frontier model?

"Open models are cheaper" is easy to say. So we measured it — openGLM (single and Sangam consensus) againstClaude Opus 4.8 and GPT-5.5 on quality, costand speed. 14 coding tasks, scored by running the code,100 runs each (1,400 results per system) with Wilson 95% confidence intervals. The twist: served on Baseten, GLM isfaster than the frontier and a fraction of the price — at parity accuracy. Every script, price and raw result ison GitHub — re-run it yourself.

How we scored it (no vibes)

Each task ships with assert-based tests. A run "passes" only if the model's code passes every assert in a fresh subprocess — accuracy is real correctness, not a judge's opinion. 8 standard tasks (is_prime, binary_search,roman_to_int…) plus 6 hard ones (edit_distance,regex_match, min_window, trap…), 100 runs each — 1,400 results per system. We report a Wilson 95% confidence interval on every accuracy, and exclude infrastructure errors (a provider 429 is not a model mistake) from the denominator. Cost comes from a dated, sourcedprice sheet (FX ₹96/$), computed from the real measured tokens. Each model runs on the host it's meantto run on — GLM on Baseten, the fast first-party GLM host.

Frontier tier — can open GLM keep up?

Three findings, all in the numbers above:

• Quality is at parity. GLM-4.7 solved 99.9% and GLM-5.2 99.4% — confidence intervals that overlap the frontier's. On correctness, the open model is not a step down.
• GLM costs roughly a tenth. ₹0.033–0.055/task vs ₹0.40–0.44 for Opus and GPT-5.5 — an order of magnitude, in real measured tokens.
• And on Baseten, GLM is faster. 1.4–1.5s vs the frontier's 2.5–2.9s. This flips the usual story: the "GLM is slow" reputation is a host artifact (a generic endpoint at ~20s), not the model. On a host built for it, GLM wins speed too — see the host showdown.

Where does Sangam fit? The GLM consensus held 97.1% — a touch behind a single fast GLM here, and its value isn't raw speed or price but robustness: a panel plus a verifier so one bad draft doesn't sink the answer. (N=140 so far — we're topping it to 1,400; the interval will tighten.)

Budget tier — the cheap-and-good frontier

• The cheapest thing here is open. gpt-oss-120B on Baseten hit 99.6% at ₹0.014/task — the lowest price and near-perfect accuracy in the whole study, beating GPT-5.4-nano on both.
• Open GLM stays competitive — GLM-4.7-flash 99.3%, GLM-4.5-air 98.5%; their only tax is latency, and that's a host artifact (these legs ran on a generic endpoint, not Baseten — the host showdown shows the fix).
• Consensus buys reliability, not cheapness. Open-Sangam reached 99.3% — but at ₹0.918/task, because a verified panel spends far more tokens. Reach for it when a wrong answer is expensive, not to save money.
• The laggards separate on the hard tier. Haiku 4.5 aced standard tasks (98%) but fell to 64% on hard ones; Nemotron-Super struggled across the board (75.8%). At the budget end, the hard problems are the real discriminator.

Field notes: what we hit, and how we handled it

A real benchmark has real hiccups — worth telling, because how you handle them isthe product. Running the GLM legs at concurrency, Zhipu's free tier started returningHTTP 429 · "Rate limit reached" — a single host capping requests per minute.

The production answer is exactly whatthe host showdown is about: don't lean on one host — put a failover chain behind one model id and let BharatRouter's circuit-breaker roll a rate-limited host over to the next:

{
  "model": "glm-4.7",
  "fallbacks": [
    { "model": "glm-4.7", "provider": "baseten" },
    { "model": "glm-4.7", "provider": "zhipu" },
    { "model": "glm-4.7", "provider": "openrouter" }
  ]
}

For clean per-system numbers we instead pinned each host and added backoff (we want to measure hosts in isolation, not the router). But in a live agent, the 429 never reaches your users — the breaker handles it. The incident is the argument for routing.

So — should you switch?

• Latency-critical (interactive coding, autocomplete): GLM-4.7/5.2 on Baseten are now in the fast lane (1.4–1.5s) — you no longer trade speed for openness. Just make sure you're on a fast host, not a generic endpoint.
• Cost-critical, quality-sensitive (batch, async agents, eval pipelines): open GLM is a genuine swap — parity accuracy at ~a tenth of the price; or drop to gpt-oss-120B for the cheapest near-perfect option in the study.
• Reliability-critical (a wrong answer is expensive): reach for Sangam — a verified panel, not a single draft. It costs more tokens; that's the premium for consensus.

The headline isn't "open beats frontier" or the reverse — it's that on the right host theaccuracy gap closes, the price gap stays an order of magnitude, and GLM even wins on speed. The right answer is workload-specific — which is the whole point of a gateway: route per request, measure, and switch without a rewrite. These are interim results on 14 tasks; a broader benchmark is coming.

Reproduce it: scripts, dated prices and raw results →github.com/bharatrouter/cookbook → ·  Start here: Zero to a GLM coding agent ·  The host showdown