# When open weights beat proprietary: 3 workloads where the 13-point BenchLM gap disappears

If you only look at the overall ranking column on [BenchLM](https://benchlm.ai/), proprietary models occupy the top spots while open-weight models appear to trail behind. Once you open the task-level tables inside BenchLM and [Artificial Analysis](https://artificialanalysis.ai/), however, the picture flips on everyday company work: open-weight workhorses like **Qwen3.8-27B**, **GLM-5.2**, and **MiMo-V2.6-Pro** match or even beat the #1 proprietary model on structured PDF extraction, customer-support policy rules, and multi-app tool calling at a tiny fraction of the token price.

Why does a model trailing by 26 points on the global index beat the #1 model in the world on document extraction? Because composite leaderboards blend Olympiad math and 25-turn autonomous terminal loops with routine production tasks.

In this guide, we break down the three production workloads where the 13-point gap disappears, the one workload where proprietary models still earn their price tag, and how to route between them on our infrastructure.

 BENCHMARK · BENCHLM & ARTIFICIAL ANALYSIS (OCT 27, 2026) 

### Proprietary leads the global composite by 12.8 points; open weights tie or win on extraction, tool policies, and PR code checks.

Comparing BenchLM global composite scores against task-level benchmarks for document extraction, tool execution, and output token pricing (October 27, 2026 snapshot).

 Open-weight models (our API & frontier open) Proprietary frontier APIs (Anthropic & OpenAI) 

BenchLM global composite score (716 records)

Opus 5.586.66 (#1)

GPT-6 Astra84.43 (#2)

MiMo-V2.673.85 (#13)

GLM-5.260.78 (#57)

 

Workload 1: ExtractBench PDF extraction (LlamaIndex)

Qwen3.8-27B89.75%

Opus 5.586.06%

Opus ParseB78.89%

Qwen ParseB69.72%

 

Workload 2: tool calling (tau2-bench & Toolathlon)

GLM-5.2 tau299.1%

Qwen3.8 IFB79.5%

Opus Tool77.8%

MiMo Tool76.9%

 

Output price per 1M tokens (lower is better)

Core plan~€0.065

MiMo-V2.6$0.87

Opus 5.5$20.00

GPT-6 Astra$50.00

 

 

**+3.69 pts**Qwen3.8-27B lead over Claude Opus 5.5 on ExtractBench
(89.75% vs 86.06%, LlamaIndex Oct 25, 2026 snapshot)

**99.1%**GLM-5.2 accuracy on Artificial Analysis tau2-bench
(customer-ops policy & tool-calling benchmark)

**23× to 300×**lower output token cost with open weights
($0.87/1M MiMo, ~€0.065/1M on our Core plan vs $20/1M Opus 5.5)

 

Sources: BenchLM global leaderboard (230 ranked models across 716 benchmark records, verified October 27, 2026), LlamaIndex ExtractBench (October 25, 2026), and Artificial Analysis v4.3.2.

 

## The models in plain numbers: proprietary frontier vs open-weight workhorses

Before looking at individual benchmarks, compare the five models side by side: on [BenchLM](https://benchlm.ai/), the overall score is a weighted composite across 8 categories (coding, agentic workflows, reasoning, mathematics, knowledge, instruction following, multimodal understanding, and multilingual accuracy).

A model that excels at PhD-level research math (`FrontierMath v2 Tier 4`) gets a massive boost in that global average, even if your company only uses AI to parse invoices, route support tickets, or review pull requests.

### At a glance: proprietary #1 vs open-weight leaders on BenchLM

Data from BenchLM (October 27, 2026) and Artificial Analysis v4.3.2. Green pills mark the best result in each row.

 

 

 | What matters | Claude Opus 5.5Anthropic (Proprietary) | MiMo-V2.6-ProXiaomi (Open weight #1) | GLM-5.2Z.AI (On our API) | Qwen3.8-27BAlibaba (On our API) | GPT-OSS 120BOpenAI (On our API) |
|---|---|---|---|---|---|
| BenchLM compositeOverall score across 8 categories | 86.66 (#1 overall)86 benchmarks covered | **73.85 (#13 overall)**\#1 open-weight model | **60.78 (#57 overall)**49 benchmarks covered | **59.83 (#60 overall)**71 benchmarks covered | **37.47 (#156 overall)**Non-reasoning fast model |
| Document extractionLlamaIndex ExtractBench | **86.06%**78.89% on ParseBench | Not reported19.2% on GDP.pdf | Not reported91.0% MMAnswerBench | 89.75% (winner)91.1% OmniDocBench 1.5 | Not reportedText & code only |
| Tool & policy executiontau2-bench & Toolathlon | 77.8% Toolathlon69.5% AA AutomationBench | 76.9% Toolathlon58.6% AA AutomationBench | 99.1% tau2-bench76.8% MCP Atlas | **79.5% IFBench**84.3% OSWorld-Verified | **65.8% tau2-bench**69.0% AA-IFBench |
| Single-pass codingLiveCodeBench & SWE-bench | 89.9% SWE-bench Pro74.2% DeepSWE | **71.9% DeepSWE**94.0% CyberGym | **82.8% SWE-bench (Vals)**68.8% AA Coding Index | 90.3% LiveCodeBench v686.0% SWE-bench (Vals) | 87.8% AA LiveCodeBench62.4% SWE-bench Verified |
| Output speedTokens per second on AA | **Moderate**Proprietary API | **Fast MoE**Open weights | **98.0 tok/s**1M context window | **Fast 27B dense**262K context window | 237.1 tok/s (fastest)$0.11 per AA task |
| Output price / 1MList API rate vs our Core plan | $20.00 / 1M out$4.00 / 1M input | $0.87 / 1M out23× cheaper than Opus | ~€0.065 / 1M Core$4.40/1M list on AA | ~€0.065 / 1M Core€0.50 in / €2.10 out PAYG | ~€0.065 / 1M Core$0.15 in / $0.60 out on AA |

  SOVEREIGN EUROPEAN INFERENCE 

### Run GLM, Qwen, Gemma, and Mistral in Europe with zero data retention

Get 600 million tokens on our Core plan (€39/mo flat, ~€0.065/1M). Switch endpoints in one line of code with full OpenAI SDK compatibility on 100% green European datacenters.

[Start 30-day free trial (no card required) →](https://regolo.ai/pricing/?utm_source=blog&utm_medium=bento_cta&utm_campaign=open-weights-vs-proprietary-benchlm)

Free credits included · 100% green energy · ✓ EU Green Datacenters ✓ Certified ZDR

## Workload 1: PDF, invoice, and structured data extraction

Consider a typical back-office pipeline: your system receives thousands of supplier invoices, customs declarations, or KYC forms in PDF format and needs to extract structured JSON fields (tax IDs, line-item totals, contract dates) cleanly into PostgreSQL.

When LlamaIndex tested this exact job on [`ExtractBench`](https://benchlm.ai/benchmarks/extractbench) and [`OmniDocBench 1.5`](https://benchlm.ai/benchmarks/omnidocbench15), compact open-weight **Qwen3.8-27B** came out ahead of Anthropic’s top-ranked **Claude Opus 5.5**, as the chart below shows:

 BENCHMARK · DOCUMENT EXTRACTION ON BENCHLM### Document extraction accuracy (Oct 2026)

Qwen3.8-27B beats #1 proprietary Claude Opus 5.5 by 3.69 points on LlamaIndex ExtractBench.

Qwen Omni91.10%

Qwen Extr89.75%

Opus Extr86.06%

Opus ParseB78.89%

LlamaIndex ExtractBench and ParseBench snapshots (Oct 25, 2026) and Qwen3.8-27B model card (OmniDocBench 1.5), reported via BenchLM (Oct 27, 2026).

Why does a compact open model outperform a giant proprietary flagship on document parsing? Because turning a messy PDF into valid JSON depends on vision-encoder resolution and strict schema obedience ([`IFBench`](https://benchlm.ai/benchmarks/ifbench)) rather than open-ended creative reasoning. Running high-volume invoice extraction on **Qwen3.8-27B** via our flat Core plan replaces thousands of dollars in metered proprietary API bills while keeping regulated customer documents inside our European datacenters.

## Workload 2: customer support policies, MCP app routing, and multi-step tool calling

The second daily workload is connecting an assistant to internal APIs: looking up a customer subscription in Stripe, checking return eligibility against company policy, and opening a ticket in Jira or Zendesk over MCP (Model Context Protocol).

When teams evaluate this workload on [`tau2-bench`](https://benchlm.ai/benchmarks/tau2-bench) (Artificial Analysis’s multi-turn customer service and policy-compliance benchmark) and [`Toolathlon-Verified`](https://benchlm.ai/benchmarks/toolathlonverified) (multi-step tool execution across real software environments), the gap between proprietary and open weights vanishes:

 BENCHMARK · TOOL CALLING & POLICY COMPLIANCE### Tool calling & policy accuracy

GLM-5.2 hits 99.1% on tau2-bench; MiMo-V2.6-Pro matches Opus 5.5 on Toolathlon-Verified.

GLM-5.2 tau299.1%

Qwen3.8 IFB79.5%

Opus Tool77.8%

MiMo Tool76.9%

BenchLM records (Oct 27, 2026): Artificial Analysis tau2-bench (GLM-5.2), IFBench (Qwen3.8-27B), and Toolathlon-Verified (Claude Opus 5.5 vs MiMo-V2.6-Pro).

- **Policy-bound customer agents:** **GLM-5.2** leads `tau2-bench` and performs strongly on `MCP Atlas`, following multi-step refund and account-change rules reliably without inventing policy exceptions.
- **Multi-app tool chains:** On `Toolathlon-Verified`, open-weight **MiMo-V2.6-Pro** virtually ties **Claude Opus 5.5** across complex software workflows while costing a fraction of the proprietary output rate.
- **High-speed triage and live chat:** When a customer is waiting in a live chat widget, **GPT-OSS 120B** and **Mistral Small 4** on our API deliver instant responses and rock-bottom per-task costs.

## Workload 3: single-pass code generation, PR reviews, and security scans

Software teams run two very different kinds of coding tasks. The first kind is single-pass engineering: generating a function from a spec, writing unit tests in CI, reviewing a pull request diff, or scanning nightly commits for known vulnerability patterns.

On single-pass coding and vulnerability benchmarks tracked by BenchLM on October 27, 2026, open-weight models match proprietary accuracy at a fraction of the bill:

 BENCHMARK · CODING & SECURITY WORKLOADS### Single-pass coding & security scans

Qwen3.8-27B hits 90.3% on LiveCodeBench v6; GPT-OSS 120B hits 87.8% at 237 tok/s.

MiMo Cyber94.0%

Qwen3.8 LCB690.3%

Opus SWE-Pro89.9%

GPT-OSS LCB87.8%

BenchLM coding records (Oct 27, 2026): CyberGym (MiMo-V2.6-Pro), LiveCodeBench v6 (Qwen3.8-27B), SWE-bench Pro (Claude Opus 5.5), and AA LiveCodeBench (GPT-OSS 120B).

- **Algorithmic code generation and IDE autocomplete:** **Qwen3.8-27B** leads single-pass coding on `LiveCodeBench v6`, while **GPT-OSS 120B** combines high coding accuracy with ultra-fast token generation for real-time IDE suggestions and inline pull-request comments.
- **Repository bug fixing:** **GLM-5.2** pairs strong single-patch resolution on `SWE-bench` with deep scientific and mathematical reasoning when a bug touches complex business logic.
- **Nightly security and vulnerability audits:** Open-weight checkpoints like **MiMo-V2.6-Pro** top `CyberGym`, letting AppSec teams run thousands of unattended sandbox scans overnight without sending proprietary source code outside their own environment.

## The honest caveat: where proprietary models still earn $20 to $50 per million tokens

If open weights win or tie on document extraction, tool calling, and single-pass coding, why does **Claude Opus 5.5** still sit at the top of the global BenchLM leaderboard? Because of two specialized frontier workloads: **long-horizon terminal recovery loops** and **research-level mathematics**.

 BENCHMARK · THE TERMINAL-BENCH 2.1 VS 4.0 CLIFF### Terminal-Bench 2.1 vs 4.0 cliff

Open weights saturate old Terminal-Bench 2.1 (89.9%), while Opus 5.5 leads new v4.0 (66.4%).

MiMo TB 2.189.9%

GLM TB 2.181.0%

Opus TB 4.066.4%

MiMo TB 4.034.9%

BenchLM agentic records (Oct 27, 2026). Terminal-Bench 2.1 is saturated across open models; strict Terminal-Bench 4.0 separates 25-turn autonomous recovery.

### Where open weights beat or match proprietary vs where proprietary leads

Task-by-task comparison from BenchLM (October 27, 2026). Match your budget to the exact test that mirrors your workload.

 

 

 | Workload & benchmark | Proprietary leader (Claude Opus 5.5) | Best open-weight result | Production verdict |
|---|---|---|---|
| **Enterprise PDF extraction**LlamaIndex ExtractBench | 86.06% ($20.00/1M out) | 89.75% (Qwen3.8-27B)~€0.065/1M on our Core plan | Open weight wins (+3.69 pts)Higher accuracy + EU data residency |
| **Customer ops & tool policy**tau2-bench & Toolathlon-Verified | 77.8% Toolathlon ($20.00/1M out) | 99.1% tau2 (GLM-5.2)76.9% Toolathlon (MiMo-V2.6-Pro) | On par (-0.9 pts Toolathlon)23× to 300× lower token cost |
| **Single-pass code & PR checks**LiveCodeBench v6 & SWE-bench (Vals) | 89.9% SWE-bench Pro ($20.00/1M out) | 90.3% LCB v6 (Qwen3.8-27B)87.8% AA LCB at 237 tok/s (GPT-OSS) | Open weight wins speed/costIdeal for CI checks & IDE completion |
| **25-turn terminal refactoring**Terminal-Bench 4.0 (strict) | 66.4% (59.6% AA strict)26.9 avg turns on Toolathlon | 34.9% (MiMo-V2.6-Pro)5.6% (Qwen3.8-27B) | Proprietary leads (+31.5 pts)Route only multi-file loops here |
| **Research math & abstract puzzles**FrontierMath v2 Tier 4 & ARC-AGI-2 | 95.0% FrontierMath Tier 491.7% ARC-AGI-2 | 29.3% Tier 4 (GLM-5.2)59.2% Tiers 1-3 (GLM-5.2) | Proprietary leads (+65.7 pts)Rarely needed in B2B software |

Notice the cliff between `Terminal-Bench 2.1` and `Terminal-Bench 4.0`.

Open-weight models have largely saturated the older `Terminal-Bench 2.1` suite, but on the much harder `Terminal-Bench 4.0` (where an agent has to explore an unfamiliar environment, recover from broken builds, and retry across 25 turns), proprietary flagships like **Claude Opus 5.5** still hold a wide lead. **That gap on long-horizon shell recovery and research math is what pulls open-weight models down in the global BenchLM composite,** even though the vast majority of enterprise API calls never touch a 25-turn bash loop or a Tier-4 math proof.

 ZERO DATA RETENTION 

### Get engineering notes and model releases in your inbox.

Sovereign AI inference, OpenAI-compatible, EU-hosted. Benchmarks, new models (Mistral, Qwen, GLM), and cost insights. One email, no spam.

✓

#### Thank you for subscribing!

Please check your inbox to confirm your subscription.

 

   Subscribe Free 

   I accept the [Privacy Policy](https://regolo.ai/privacy-policy/) \*  

    100% European infrastructure · Zero spam · Unsubscribe anytime.

 

## Visual guide: how to route workloads between open weights and frontier APIs

Use the decision map below to match each daily engineering activity to the model that wins that specific benchmark:

 DECISION MAP · WORKLOAD & BENCHMARK → MODEL → TARGET ICP### Which model fits your workload? Follow the task-level numbers

Route the 85% of document, tool, and single-pass coding jobs to open weights on our EU infrastructure, and reserve $20/1M proprietary tokens for 25-turn terminal loops.

 

1. Your daily activity & benchmark proof

2. Model to pick

3. Ideal customer profile (ICP)

 

Activity · Enterprise PDF & invoice extraction

Parse 10,000+ invoices, contracts, and charts into strict JSON schemas

Proof: **89.75%** ExtractBench (beats Opus 5.5 at 86.06%) · **91.1%** OmniDocBench 1.5

 

→

Extraction winner

Qwen3.8-27B

 ~€0.065/1M Core 262K ctx Apache 2.0 

 

→

Target ICP · FinTech, KYC & document ops

Regulated EU teams processing private customer PDFs

You beat #1 proprietary extraction accuracy by 3.69 points while keeping documents inside EU Zero Data Retention datacenters.

 

 

Activity · Customer ops & MCP tool workflows

Multi-turn support agents calling CRM, billing, and SQL tools under strict policy rules

Proof: **99.1%** tau2-bench · **76.8%** MCP Atlas · **82.8%** SWE-bench (Vals)

 

→

Policy & tool workhorse

GLM-5.2

 ~€0.065/1M Core 98 tok/s 1M ctx 

 

→

Target ICP · SaaS & enterprise AI teams

Teams connecting assistants to 10+ internal tools via MCP

You get 99.1% policy compliance and 1M context on our API without paying $20/1M output tokens on routine ticket resolution.

 

 

Activity · Real-time IDE completion & CI PR checks

Inline code suggestions, unit test generation, and high-volume ticket routing

Proof: **87.8%** AA LiveCodeBench · **237.1 tok/s** · **$0.02 to $0.11** per AA task

 

→

Fastest throughput

GPT-OSS 120B / Mistral Small 4

 237 tok/s $0.02 to $0.11/task OpenAI API 

 

→

Target ICP · DevEx & platform engineering

Engineering orgs running thousands of automated CI reviews per day

Sub-second time to first token and 237 tok/s keep developers in flow while cutting CI token bills by over 90%.

 

 

Activity · 25-turn autonomous repo refactoring

Unattended terminal agent editing 15+ files and recovering from shell test failures

Proof: **66.4%** Terminal-Bench 4.0 (Opus 5.5) vs **34.9%** (MiMo-V2.6-Pro)

 

→

Hybrid semantic routing

Brick Router + Frontier

 85% EU Open 15% Frontier 

 

→

Target ICP · AI-native product teams

CTOs balancing autonomous coding quality against cloud spend

Use our Brick semantic router to keep 85% of prompts on fast EU open weights and escalate only deep multi-file loops.

 

 

 

Tip: use our **Brick** semantic router to classify prompt complexity in milliseconds and route extraction, tool calls, and single-file patches to **Qwen3.8-27B**, **GLM-5.2**, or **GPT-OSS 120B**.

As we showed in our guide on [why hybrid routing is the default](https://regolo.ai/hybrid-routing-is-the-default-why-managed-open-weights-beat-self-hosting-and-closed-apis/), no engineering team needs to pay $20.00 per million output tokens to extract an invoice total or write a unit test. Through [our platform](https://regolo.ai/pricing/), you can call **Qwen3.8-27B**, **GLM-5.2**, **GPT-OSS 120B**, **Gemma 4 31B**, and **Mistral Small 4** through one OpenAI-compatible EU endpoint with certified Zero Data Retention, or deploy any custom Hugging Face checkpoint on our dedicated European GPUs.

  SOVEREIGN EUROPEAN INFERENCE 

###  Run GLM, Qwen, Gemma, and Mistral in Europe with zero data retention 

 Get 600 million tokens on our Core plan (€39/mo flat, ~€0.065/1M). Switch endpoints in 1 line of code with full OpenAI SDK compatibility on 100% green datacenters.

 [ Start 30-day trial → ](https://regolo.ai/pricing/?utm_source=blog&utm_medium=bento_cta&utm_campaign=deepseek-flash-mid)  Free credits included · Live in 60s  

  ✓ 100% EU Green Datacenters   ✓ Certified ZDR  

 

 

## Frequently asked questions

**Why does Claude Opus 5.5 lead BenchLM overall if Qwen3.8-27B beats it on ExtractBench?**
BenchLM’s global score averages eight broad categories, including PhD-level research math (`FrontierMath v2 Tier 4`) and 25-turn autonomous bash recovery (`Terminal-Bench 4.0`), where proprietary flagships still dominate. On focused enterprise tasks like structured PDF extraction (`ExtractBench`), compact open-weight models like Qwen3.8-27B already outperform the top proprietary model, as shown in Chart 1.

**Which open-weight model on our API is best for MCP tool calling and customer support bots?**
GLM-5.2 is the strongest choice we serve for policy-bound tool calling and MCP workflows, leading Artificial Analysis’s `tau2-bench` with a 1M token context window. For live chat widgets where instant response speed is the top priority, GPT-OSS 120B and Mistral Small 4 deliver our highest token throughput.

**How much do Qwen3.8-27B and GLM-5.2 cost on our platform compared to proprietary APIs?**
While frontier proprietary APIs charge $20 to $50 per million output tokens, our flat Core subscription gives you a monthly pool of up to 600 million tokens across our open-weight catalog for €39 per month (€11.70 per month for the first 3 months), cutting token spend by over 90% on workloads where open weights match proprietary accuracy.

**Why do open-weight models score so high on Terminal-Bench 2.1 but drop on Terminal-Bench 4.0?**
`Terminal-Bench 2.1` is an older suite that modern open-weight models have largely saturated. `Terminal-Bench 4.0` introduces much harder, multi-step environment recovery tasks that separate long-horizon terminal agents from single-pass coders, so always check which version a benchmark chart is quoting.

**How do I switch my existing OpenAI or Anthropic SDK code to Regolo?**
Point your OpenAI client `base_url` to `https://api.regolo.ai/v1`, set your `REGOLO_API_KEY`, and select a model such as `qwen3.8-27b`, `glm-5.2`, `gpt-oss-120b`, or `mistral-small-4`. You can also place our Brick semantic router in front of your gateway to route simple and medium tasks to open weights in Europe while escalating only 25-turn refactoring jobs.

    PRODUCTION-READY EU INFERENCE · ZERO DATA RETENTION  OpenAI-Compatible API 

## Ship Private AI. Not Infrastructure.

 You have the architecture and the open-source workflow. Now give it a sovereign European inference layer built for production: fast access to **GLM-5.2, Qwen3.8-27B, GPT-OSS 120B, Gemma 4, Mistral**, and **Brick** with zero data retention and no new SDK to learn.

 DROP-IN ENDPOINT `export OPENAI_BASE_URL="https://api.regolo.ai/v1"` Keep your existing SDK & agent setup 

 

  [    30-DAY FREE TRIAL 

### Start your 30-day free trial →

Build, test, and deploy with no infrastructure to maintain. Core plans from €11.70/mo (70% off first 3 months).

 ](https://regolo.ai/pricing/?utm_source=blog&utm_medium=footer_stub&utm_campaign=tutorials)  [    OPEN-SOURCE CODE 

### Clone the GitHub repository →

Get the full implementation, configuration scripts, evaluation benchmarks, and ready-to-run templates.

 ](https://github.com/regolo-ai/tutorials/)  [    BUILDER COMMUNITY 

### Join the Regolo Discord →

Meet engineers building with coding agents, private RAG, local LLMs, and production open-weight models.

 ](https://discord.gg/bqGrVJHeF)  [    ARCHITECTURE REVIEW 

### Talk to an AI Infrastructure Engineer →

Scaling beyond a proof of concept or running regulated EU workloads? Get a tailored architecture and commercial plan.

 ](https://regolo.ai/contacts/?utm_source=blog&utm_medium=footer_stub&utm_campaign=tutorials) 

 **Private AI should not require a private data center.** We give your team an EU-native path from local experimentation to production-grade inference. 

 [API Docs](https://docs.regolo.ai) · [GitHub](https://github.com/regolo-ai/tutorials/) · [Discord](https://discord.gg/bqGrVJHeF) · [X (@regolo\_ai)](https://x.com/regolo_ai) · [Reddit](https://www.reddit.com/r/regolo_ai/) · [Contact](https://regolo.ai/contacts/) 

Built by the Regolo team · 100% EU Sovereign GPU