If you only look at the overall ranking column on BenchLM, proprietary models occupy the top spots while open-weight models appear to trail behind. Once you open the task-level tables inside BenchLM and Artificial Analysis, however, the picture flips on everyday company work: open-weight workhorses like Qwen3.8-27B, GLM-5.2, and MiMo-V2.6-Pro match or even beat the #1 proprietary model on structured PDF extraction, customer-support policy rules, and multi-app tool calling at a tiny fraction of the token price.
Why does a model trailing by 26 points on the global index beat the #1 model in the world on document extraction? Because composite leaderboards blend Olympiad math and 25-turn autonomous terminal loops with routine production tasks.
In this guide, we break down the three production workloads where the 13-point gap disappears, the one workload where proprietary models still earn their price tag, and how to route between them on our infrastructure.
Proprietary leads the global composite by 12.8 points; open weights tie or win on extraction, tool policies, and PR code checks.
Comparing BenchLM global composite scores against task-level benchmarks for document extraction, tool execution, and output token pricing (October 27, 2026 snapshot).
(89.75% vs 86.06%, LlamaIndex Oct 25, 2026 snapshot)
(customer-ops policy & tool-calling benchmark)
($0.87/1M MiMo, ~€0.065/1M on our Core plan vs $20/1M Opus 5.5)
Sources: BenchLM global leaderboard (230 ranked models across 716 benchmark records, verified October 27, 2026), LlamaIndex ExtractBench (October 25, 2026), and Artificial Analysis v4.3.2.
The models in plain numbers: proprietary frontier vs open-weight workhorses
Before looking at individual benchmarks, compare the five models side by side: on BenchLM, the overall score is a weighted composite across 8 categories (coding, agentic workflows, reasoning, mathematics, knowledge, instruction following, multimodal understanding, and multilingual accuracy).
A model that excels at PhD-level research math (FrontierMath v2 Tier 4) gets a massive boost in that global average, even if your company only uses AI to parse invoices, route support tickets, or review pull requests.
At a glance: proprietary #1 vs open-weight leaders on BenchLM
Data from BenchLM (October 27, 2026) and Artificial Analysis v4.3.2. Green pills mark the best result in each row.
| What matters | Claude Opus 5.5Anthropic (Proprietary) | MiMo-V2.6-ProXiaomi (Open weight #1) | GLM-5.2Z.AI (On our API) | Qwen3.8-27BAlibaba (On our API) | GPT-OSS 120BOpenAI (On our API) |
|---|---|---|---|---|---|
| BenchLM compositeOverall score across 8 categories | 86.66 (#1 overall)86 benchmarks covered | 73.85 (#13 overall)#1 open-weight model | 60.78 (#57 overall)49 benchmarks covered | 59.83 (#60 overall)71 benchmarks covered | 37.47 (#156 overall)Non-reasoning fast model |
| Document extractionLlamaIndex ExtractBench | 86.06%78.89% on ParseBench | Not reported19.2% on GDP.pdf | Not reported91.0% MMAnswerBench | 89.75% (winner)91.1% OmniDocBench 1.5 | Not reportedText & code only |
| Tool & policy executiontau2-bench & Toolathlon | 77.8% Toolathlon69.5% AA AutomationBench | 76.9% Toolathlon58.6% AA AutomationBench | 99.1% tau2-bench76.8% MCP Atlas | 79.5% IFBench84.3% OSWorld-Verified | 65.8% tau2-bench69.0% AA-IFBench |
| Single-pass codingLiveCodeBench & SWE-bench | 89.9% SWE-bench Pro74.2% DeepSWE | 71.9% DeepSWE94.0% CyberGym | 82.8% SWE-bench (Vals)68.8% AA Coding Index | 90.3% LiveCodeBench v686.0% SWE-bench (Vals) | 87.8% AA LiveCodeBench62.4% SWE-bench Verified |
| Output speedTokens per second on AA | ModerateProprietary API | Fast MoEOpen weights | 98.0 tok/s1M context window | Fast 27B dense262K context window | 237.1 tok/s (fastest)$0.11 per AA task |
| Output price / 1MList API rate vs our Core plan | $20.00 / 1M out$4.00 / 1M input | $0.87 / 1M out23× cheaper than Opus | ~€0.065 / 1M Core$4.40/1M list on AA | ~€0.065 / 1M Core€0.50 in / €2.10 out PAYG | ~€0.065 / 1M Core$0.15 in / $0.60 out on AA |
Run GLM, Qwen, Gemma, and Mistral in Europe with zero data retention
Get 600 million tokens on our Core plan (€39/mo flat, ~€0.065/1M). Switch endpoints in one line of code with full OpenAI SDK compatibility on 100% green European datacenters.
Start 30-day free trial (no card required) →
Free credits included · 100% green energy · ✓ EU Green Datacenters ✓ Certified ZDR
Workload 1: PDF, invoice, and structured data extraction
Consider a typical back-office pipeline: your system receives thousands of supplier invoices, customs declarations, or KYC forms in PDF format and needs to extract structured JSON fields (tax IDs, line-item totals, contract dates) cleanly into PostgreSQL.
When LlamaIndex tested this exact job on ExtractBench and OmniDocBench 1.5, compact open-weight Qwen3.8-27B came out ahead of Anthropic’s top-ranked Claude Opus 5.5, as the chart below shows:
Document extraction accuracy (Oct 2026)
Qwen3.8-27B beats #1 proprietary Claude Opus 5.5 by 3.69 points on LlamaIndex ExtractBench.
LlamaIndex ExtractBench and ParseBench snapshots (Oct 25, 2026) and Qwen3.8-27B model card (OmniDocBench 1.5), reported via BenchLM (Oct 27, 2026).
Why does a compact open model outperform a giant proprietary flagship on document parsing? Because turning a messy PDF into valid JSON depends on vision-encoder resolution and strict schema obedience (IFBench) rather than open-ended creative reasoning. Running high-volume invoice extraction on Qwen3.8-27B via our flat Core plan replaces thousands of dollars in metered proprietary API bills while keeping regulated customer documents inside our European datacenters.
Workload 2: customer support policies, MCP app routing, and multi-step tool calling
The second daily workload is connecting an assistant to internal APIs: looking up a customer subscription in Stripe, checking return eligibility against company policy, and opening a ticket in Jira or Zendesk over MCP (Model Context Protocol).
When teams evaluate this workload on tau2-bench (Artificial Analysis’s multi-turn customer service and policy-compliance benchmark) and Toolathlon-Verified (multi-step tool execution across real software environments), the gap between proprietary and open weights vanishes:
Tool calling & policy accuracy
GLM-5.2 hits 99.1% on tau2-bench; MiMo-V2.6-Pro matches Opus 5.5 on Toolathlon-Verified.
BenchLM records (Oct 27, 2026): Artificial Analysis tau2-bench (GLM-5.2), IFBench (Qwen3.8-27B), and Toolathlon-Verified (Claude Opus 5.5 vs MiMo-V2.6-Pro).
- Policy-bound customer agents: GLM-5.2 leads
tau2-benchand performs strongly onMCP Atlas, following multi-step refund and account-change rules reliably without inventing policy exceptions. - Multi-app tool chains: On
Toolathlon-Verified, open-weight MiMo-V2.6-Pro virtually ties Claude Opus 5.5 across complex software workflows while costing a fraction of the proprietary output rate. - High-speed triage and live chat: When a customer is waiting in a live chat widget, GPT-OSS 120B and Mistral Small 4 on our API deliver instant responses and rock-bottom per-task costs.
Workload 3: single-pass code generation, PR reviews, and security scans
Software teams run two very different kinds of coding tasks. The first kind is single-pass engineering: generating a function from a spec, writing unit tests in CI, reviewing a pull request diff, or scanning nightly commits for known vulnerability patterns.
On single-pass coding and vulnerability benchmarks tracked by BenchLM on October 27, 2026, open-weight models match proprietary accuracy at a fraction of the bill:
Single-pass coding & security scans
Qwen3.8-27B hits 90.3% on LiveCodeBench v6; GPT-OSS 120B hits 87.8% at 237 tok/s.
BenchLM coding records (Oct 27, 2026): CyberGym (MiMo-V2.6-Pro), LiveCodeBench v6 (Qwen3.8-27B), SWE-bench Pro (Claude Opus 5.5), and AA LiveCodeBench (GPT-OSS 120B).
- Algorithmic code generation and IDE autocomplete: Qwen3.8-27B leads single-pass coding on
LiveCodeBench v6, while GPT-OSS 120B combines high coding accuracy with ultra-fast token generation for real-time IDE suggestions and inline pull-request comments. - Repository bug fixing: GLM-5.2 pairs strong single-patch resolution on
SWE-benchwith deep scientific and mathematical reasoning when a bug touches complex business logic. - Nightly security and vulnerability audits: Open-weight checkpoints like MiMo-V2.6-Pro top
CyberGym, letting AppSec teams run thousands of unattended sandbox scans overnight without sending proprietary source code outside their own environment.
The honest caveat: where proprietary models still earn $20 to $50 per million tokens
If open weights win or tie on document extraction, tool calling, and single-pass coding, why does Claude Opus 5.5 still sit at the top of the global BenchLM leaderboard? Because of two specialized frontier workloads: long-horizon terminal recovery loops and research-level mathematics.
Terminal-Bench 2.1 vs 4.0 cliff
Open weights saturate old Terminal-Bench 2.1 (89.9%), while Opus 5.5 leads new v4.0 (66.4%).
BenchLM agentic records (Oct 27, 2026). Terminal-Bench 2.1 is saturated across open models; strict Terminal-Bench 4.0 separates 25-turn autonomous recovery.
Where open weights beat or match proprietary vs where proprietary leads
Task-by-task comparison from BenchLM (October 27, 2026). Match your budget to the exact test that mirrors your workload.
| Workload & benchmark | Proprietary leader (Claude Opus 5.5) | Best open-weight result | Production verdict |
|---|---|---|---|
| Enterprise PDF extractionLlamaIndex ExtractBench | 86.06% ($20.00/1M out) | 89.75% (Qwen3.8-27B)~€0.065/1M on our Core plan | Open weight wins (+3.69 pts)Higher accuracy + EU data residency |
| Customer ops & tool policytau2-bench & Toolathlon-Verified | 77.8% Toolathlon ($20.00/1M out) | 99.1% tau2 (GLM-5.2)76.9% Toolathlon (MiMo-V2.6-Pro) | On par (-0.9 pts Toolathlon)23× to 300× lower token cost |
| Single-pass code & PR checksLiveCodeBench v6 & SWE-bench (Vals) | 89.9% SWE-bench Pro ($20.00/1M out) | 90.3% LCB v6 (Qwen3.8-27B)87.8% AA LCB at 237 tok/s (GPT-OSS) | Open weight wins speed/costIdeal for CI checks & IDE completion |
| 25-turn terminal refactoringTerminal-Bench 4.0 (strict) | 66.4% (59.6% AA strict)26.9 avg turns on Toolathlon | 34.9% (MiMo-V2.6-Pro)5.6% (Qwen3.8-27B) | Proprietary leads (+31.5 pts)Route only multi-file loops here |
| Research math & abstract puzzlesFrontierMath v2 Tier 4 & ARC-AGI-2 | 95.0% FrontierMath Tier 491.7% ARC-AGI-2 | 29.3% Tier 4 (GLM-5.2)59.2% Tiers 1-3 (GLM-5.2) | Proprietary leads (+65.7 pts)Rarely needed in B2B software |
Notice the cliff between Terminal-Bench 2.1 and Terminal-Bench 4.0.
Open-weight models have largely saturated the older Terminal-Bench 2.1 suite, but on the much harder Terminal-Bench 4.0 (where an agent has to explore an unfamiliar environment, recover from broken builds, and retry across 25 turns), proprietary flagships like Claude Opus 5.5 still hold a wide lead. That gap on long-horizon shell recovery and research math is what pulls open-weight models down in the global BenchLM composite, even though the vast majority of enterprise API calls never touch a 25-turn bash loop or a Tier-4 math proof.
Get engineering notes and model releases in your inbox.
Sovereign AI inference, OpenAI-compatible, EU-hosted. Benchmarks, new models (Mistral, Qwen, GLM), and cost insights. One email, no spam.
Thank you for subscribing!
Please check your inbox to confirm your subscription.
100% European infrastructure · Zero spam · Unsubscribe anytime.
Visual guide: how to route workloads between open weights and frontier APIs
Use the decision map below to match each daily engineering activity to the model that wins that specific benchmark:
Which model fits your workload? Follow the task-level numbers
Route the 85% of document, tool, and single-pass coding jobs to open weights on our EU infrastructure, and reserve $20/1M proprietary tokens for 25-turn terminal loops.
Tip: use our Brick semantic router to classify prompt complexity in milliseconds and route extraction, tool calls, and single-file patches to Qwen3.8-27B, GLM-5.2, or GPT-OSS 120B.
As we showed in our guide on why hybrid routing is the default, no engineering team needs to pay $20.00 per million output tokens to extract an invoice total or write a unit test. Through our platform, you can call Qwen3.8-27B, GLM-5.2, GPT-OSS 120B, Gemma 4 31B, and Mistral Small 4 through one OpenAI-compatible EU endpoint with certified Zero Data Retention, or deploy any custom Hugging Face checkpoint on our dedicated European GPUs.
Run GLM, Qwen, Gemma, and Mistral in Europe with zero data retention
Get 600 million tokens on our Core plan (€39/mo flat, ~€0.065/1M). Switch endpoints in 1 line of code with full OpenAI SDK compatibility on 100% green datacenters.
Frequently asked questions
Why does Claude Opus 5.5 lead BenchLM overall if Qwen3.8-27B beats it on ExtractBench?
BenchLM’s global score averages eight broad categories, including PhD-level research math (FrontierMath v2 Tier 4) and 25-turn autonomous bash recovery (Terminal-Bench 4.0), where proprietary flagships still dominate. On focused enterprise tasks like structured PDF extraction (ExtractBench), compact open-weight models like Qwen3.8-27B already outperform the top proprietary model, as shown in Chart 1.
Which open-weight model on our API is best for MCP tool calling and customer support bots?
GLM-5.2 is the strongest choice we serve for policy-bound tool calling and MCP workflows, leading Artificial Analysis’s tau2-bench with a 1M token context window. For live chat widgets where instant response speed is the top priority, GPT-OSS 120B and Mistral Small 4 deliver our highest token throughput.
How much do Qwen3.8-27B and GLM-5.2 cost on our platform compared to proprietary APIs?
While frontier proprietary APIs charge $20 to $50 per million output tokens, our flat Core subscription gives you a monthly pool of up to 600 million tokens across our open-weight catalog for €39 per month (€11.70 per month for the first 3 months), cutting token spend by over 90% on workloads where open weights match proprietary accuracy.
Why do open-weight models score so high on Terminal-Bench 2.1 but drop on Terminal-Bench 4.0?Terminal-Bench 2.1 is an older suite that modern open-weight models have largely saturated. Terminal-Bench 4.0 introduces much harder, multi-step environment recovery tasks that separate long-horizon terminal agents from single-pass coders, so always check which version a benchmark chart is quoting.
How do I switch my existing OpenAI or Anthropic SDK code to Regolo?
Point your OpenAI client base_url to https://api.regolo.ai/v1, set your REGOLO_API_KEY, and select a model such as qwen3.8-27b, glm-5.2, gpt-oss-120b, or mistral-small-4. You can also place our Brick semantic router in front of your gateway to route simple and medium tasks to open weights in Europe while escalating only 25-turn refactoring jobs.
Ship Private AI. Not Infrastructure.
You have the architecture and the open-source workflow. Now give it a sovereign European inference layer built for production: fast access to GLM-5.2, Qwen3.8-27B, GPT-OSS 120B, Gemma 4, Mistral, and Brick with zero data retention and no new SDK to learn.
Start your 30-day free trial →
Build, test, and deploy with no infrastructure to maintain. Core plans from €11.70/mo (70% off first 3 months).
Clone the GitHub repository →
Get the full implementation, configuration scripts, evaluation benchmarks, and ready-to-run templates.
Join the Regolo Discord →
Meet engineers building with coding agents, private RAG, local LLMs, and production open-weight models.
Talk to an AI Infrastructure Engineer →
Scaling beyond a proof of concept or running regulated EU workloads? Get a tailored architecture and commercial plan.