When Artificial Analysis replaced its fourth-generation benchmark suite with Intelligence Index v5.0.1 at the end of October 2026, every model’s headline score dropped while average cost and runtime per task jumped. The updated index swaps out saturated single-turn exams for nine multi-step evaluations, including Terminal-Bench 4.0, FrontierCode 2.0, AA-Briefcase v1.1, and AutomationBench-AA. On that harder scale, Kimi K3.1 Pro (2.8 trillion total parameters, 104 billion active per token) took the #1 open-weight spot at 46 points and climbed to 74.61 on BenchLM.
Why does a 2.8-trillion-parameter model that matches Claude Opus 5.5 on Terminal-Bench 4.0 lose to a 16B-active model on repository bug fixing and cost six times more per task?
In this guide, we compare the November 2026 numbers for Kimi K3.1 Pro, DeepSeek V4.1 Flash 1017, GLM-5.3 (alongside GLM-5.2 on our API), and Qwen3.8-27B, and show how we route daily engineering workloads across our European infrastructure to avoid 36-minute waits and runaway token bills.
Kimi K3.1 Pro leads the AA Index v5 at 46 points ($3.68/task); DeepSeek 1017 wins DeepSWE v1.1 at 77.0% for $0.60/task.
Comparing Artificial Analysis Intelligence Index v5.0.1 composite scores, DeepSWE v1.1 coding pass rates, output throughput, and weighted cost per task (verified November 5, 2026).
(45× to 150× lower per 1M output tokens than $3 to $10 PAYG)
(174 tok/s on 16B-active DeepSeek vs 33 tok/s on 104B-active Kimi)
(served on our flat €39/mo European Core API pool)
Sources: Artificial Analysis Intelligence Index v5.0.1 model pages, BenchLM BenchAlign v5.8 leaderboard (1,014 tracked models, verified November 5, 2026), and moonshotai/Kimi-K3.1-Pro Hugging Face evaluation tables.
Active parameters, streaming speed, and hardware footprint across the four models
Before looking at individual benchmark scores, look at how many parameters each checkpoint executes per token. Total parameters determine how much GPU memory (VRAM) you need to hold the weights, while active parameters per token determine how fast the model streams tokens and how long an agent waits between tool calls.
Kimi K3.1 Pro scales its Mixture-of-Experts (MoE) architecture to 2.8 trillion total parameters and activates 104 billion parameters on every token across 896 routed experts. Even with quantization-aware training in 4-bit MXFP4, holding 2.8 trillion weights requires roughly 1.4 TB of VRAM across a multi-node GPU cluster, and computing 104 billion active weights per token holds output throughput to 33 tokens per second.
Open-weight and proprietary benchmark ledger (November 2026)
Artificial Analysis Intelligence Index v5.0.1, BenchLM v5.8, and Moonshot / DeepSeek evaluation tables
| Metric | Kimi K3.1 Pro2.8T / 104B MoE | DeepSeek 1017552B / 16B MoE | GLM-5.3 / 5.2753B / 40B MoE | Qwen3.8-27B27B Dense (Core API) | Claude Opus 5.5Proprietary ref. |
|---|---|---|---|---|---|
| Global composite scoreAA Index v5.0.1 / BenchLM | 46 pts (#1 Open)BenchLM: 74.61 (#13) | 33 ptsBenchLM: 65.20 (#30) | 32 pts (GLM-5.3)BenchLM: 61.00 (#50) | 20 ptsBenchLM: 54.30 (#80) | 59 pts (#2 All)BenchLM: 86.29 (#2) |
| Repository bug fixingDeepSWE v1.1 (multi-turn) | 75.5%Claude Code harness | 77.0% (#1)Beats Opus 5.5 & Kimi | 66.9%SWE-bench Vals: 95.4% | 42.5%SWE-bench Pro: 61.7% | 74.2%SWE-bench Pro: 89.9% |
| CLI & office agentsTerminal-Bench 4.0 & OfficeQA | 65.7% TB4.0OfficeQA Pro v2: 64.4% | 42.3% TB4.0OfficeQA Pro v2: 40.4% | 41.8% TB4.0TB 2.1: 88.2% | 6.3% TB4.0TB 2.1: 73.0% | 66.4% TB4.0OfficeQA Pro v2: 62.2% |
| Schema & policy accuracyExtractBench & tau2-bench | 73.3% AutoBenchGDP.pdf w/ py: 85.7% | 66.6% AutoBenchAPEX-Agents: 69.8% | 99.1% tau2 (GLM-5.2)Toolathlon: 73.0% | 89.75% ExtractBenchBeats Opus 5.5 (86.06%) | 86.06% ExtractBenchAutoBench: 69.5% |
| Speed & task durationOutput tok/s & AA v5 min/task | 33 tok/s (36.0 min)First token: 153.5s | 174 tok/s (13.3 min)First token: 50.8s | 65 tok/s (30.7 min)First token: 96.0s | 46 tok/s (39.8 min)Low effort: 26.1 min | 98 tok/s (26.2 min)First token: 345.8s |
| Price per 1M tokensRegolo Core vs AA pay-per-token | $2.00 in / $10.00 out113M output tok on AA v5 | $0.30 in / $1.20 out174M output tok (-98% cache) | ~€0.065/1M on CoreAA PAYG: $1.40 in / $4.40 out | ~€0.065/1M on CoreAA PAYG: $0.50 in / $3.00 out | $4.00 in / $20.00 outProprietary metered API |
| Cost per AA v5 taskPay-per-token vs Regolo flat pool | $3.68 / taskMetered per-token API | $0.60 / task6.1× cheaper than Kimi | $5.01 / task (PAYG)GLM-5.2 in €39/mo Core | €39/mo flat (600M tok)vs $3.53/task on AA PAYG | $10.04 / task$4/$20 per 1M tokens |
Output speed (tokens/second)
Active parameters per token drive streaming speed across the 4 models
Artificial Analysis throughput (TTFT: 50.8s DeepSeek 1017, 96.0s GLM-5.3, 122.7s Qwen3.8-27B xhigh, 153.5s Kimi K3.1 Pro). Sources: Artificial Analysis Index v5.0.1 (Nov 5, 2026). Sources: Artificial Analysis Index v5.0.1 & BenchLM v5.8 (verified Nov 5, 2026)
Get 600 million monthly tokens on our flat Regolo Core plan (€39/mo, ~€0.065/1M tokens) or deploy any Hugging Face checkpoint on dedicated European GPUs. Switch endpoints in one line of code with full OpenAI SDK compatibility.
Start 30-day free trial (no card required) →
Free credits included · 100% green energy · ✓ EU Green Datacenters ✓ Certified Zero Data Retention
30-step terminal recovery vs repository bug patches and schema extraction
When an engineering team asks which open-weight model is “best for coding,” the answer depends on whether the agent is fixing a bug inside an existing Git repository or recovering from broken environment state across 30 shell commands. In our earlier breakdown of DeepSeek 1017, Muse Spark 816B, and Beam 501B, we saw how single-turn and multi-turn harnesses split coding rankings. The Artificial Analysis Index v5.0.1 data makes that split even clearer with Kimi K3.1 Pro.
However, when the workload shifts to pure software engineering inside a codebase, the ranking reverses. On DeepSWE v1.1, DeepSeek V4.1 Flash 1017 scores 77.0%, edging out both Kimi K3.1 Pro (75.5%) and Claude Opus 5.5 (74.2%). And as we showed in our analysis of where open weights beat proprietary models on BenchLM, smaller workhorses win on focused enterprise tasks: Qwen3.8-27B leads LlamaIndex’s ExtractBench at 89.75% (ahead of Claude Opus 5.5 at 86.06%), and GLM-5.2 hits 99.1% on Artificial Analysis’s tau2-bench customer-operations policy benchmark.
DeepSWE v1.1 pass rate
Multi-turn repository bug fixing: 16B-active DeepSeek vs 104B-active Kimi
DeepSeek V4.1 Flash 1017 (552B total, 16B active) beats Kimi K3.1 Pro (2.8T total, 104B active) and Claude Opus 5.5 on DeepSWE v1.1 while Kimi leads Terminal-Bench 4.0 at 65.7%. Sources: Artificial Analysis Index v5.0.1 & BenchLM v5.8 (verified Nov 5, 2026)
The 36-minute latency trap and hidden reasoning token bills on AA Index v5
The biggest surprise in the Artificial Analysis Intelligence Index v5.0.1 release is how much wall-clock time and hidden reasoning verbosity cost when models run at maximum thinking effort. Because reasoning models bill internal scratchpad tokens (reasoning_content) at the full output token rate, generating 113 million to 174 million output tokens across the suite pushes Artificial Analysis’s pay-per-token cost to $3.53 per task on Qwen3.8-27B ($0.50 in / $3.00 out per 1M), $3.68 per task on Kimi K3.1 Pro ($2.00 in / $10.00 out), and $5.01 per task on GLM-5.3 ($1.40 in / $4.40 out), versus $0.60 per task on DeepSeek V4.1 Flash 1017 ($0.30 in / $1.20 out).
With our flat Regolo Core subscription (€39/month for 600 million tokens), the effective price drops to roughly €0.065 per million tokens across our Core models (including Qwen3.8-27B and GLM-5.2), cutting the per-million output token rate by 45× to 150× compared to metered pay-per-token APIs.
Output price per 1M tokens
Regolo Core subscription (€39/mo for 600M tok) vs AA pay-per-token rates
Our €39/mo Core plan includes 600M tokens (~€0.065/1M on Qwen3.8-27B & GLM-5.2) vs AA pay-per-token output rates ($3.00/1M Qwen3.8, $4.40/1M GLM-5.3, $10.00/1M Kimi K3.1). Sources: Artificial Analysis Index v5.0.1 & BenchLM v5.8 (verified Nov 5, 2026)
Latency follows the active parameter count and reasoning effort settings just as strictly. Waiting 36.0 minutes per task on Kimi K3.1 Pro (with a 153.5-second wait before the first answer token appears) or 30.7 minutes on GLM-5.3 is acceptable for an overnight evaluation batch, but it stalls interactive IDE sessions in OpenCode or Roo Code where a developer is waiting to review a diff. DeepSeek V4.1 Flash 1017 cuts that average task duration to 13.3 minutes at maximum effort (and 10.3 minutes at high effort for $0.46 per task).
Minutes per AA Index v5 task
Average wall-clock execution time per benchmark task (lower is faster)
Activating 104B parameters per token slows Kimi K3.1 Pro to 33 tok/s (153.5s TTFT) and 36.0 minutes per task, vs 174 tok/s and 13.3 minutes on 16B-active DeepSeek 1017. Sources: Artificial Analysis Index v5.0.1 & BenchLM v5.8 (verified Nov 5, 2026)
Running Qwen3.8-27B at xhigh reasoning effort generates 128 million output tokens across the suite (122.7s time to first answer token, 39.8 minutes per task). Stepping down to medium or low effort cuts time to first answer token to 35.6 seconds and 18.3 seconds respectively, while switching to non-reasoning mode drops time to first token to 5.5 seconds for structured JSON extraction and single-pass code edits.
Similarly, stepping GLM-5.3 from max to low effort halves task duration from 30.7 minutes to 15.5 minutes (13.0s time to first token) and lowers task cost from $5.01 to $1.57, while GLM-5.3-Flash (320B total, 18B active) reaches 28 points on AA Index v5 at $0.58 per task.
How to choose between Kimi K3.1 Pro, DeepSeek 1017, GLM-5.3/5.2, and Qwen3.8-27B
No engineering team needs a 2.8-trillion-parameter model running at $3.68 and 36 minutes per task to parse an invoice, enforce a customer-support tool policy, or patch a TypeScript function. In production coding harnesses like OpenCode, Roo Code, and Kilo Code, roughly 70% to 80% of agent turns are routine file reads, schema extractions, and test checks, while only 20% to 30% require deep multi-file reasoning.
On our European infrastructure, you can run Qwen3.8-27B and GLM-5.2 inside our flat €39/month Core subscription (600 million monthly tokens), pair them with our open-source Brick semantic router (brick-v1-beta) so each prompt is routed by complexity, and deploy frontier MoE checkpoints like DeepSeek-V4.1-Flash-1017 or Kimi-K3.1-Pro on dedicated European GPUs through our custom models service under the same Zero Data Retention architecture.
Which open-weight model fits your engineering workflow
Match your daily engineering task to the benchmark-proven checkpoint, latency budget, and European deployment path.
Sources: Artificial Analysis Intelligence Index v5.0.1, BenchLM v5.8 (Nov 5, 2026), and official Hugging Face model cards.
Explore related benchmarks and harness guides on Regolo: Read our deep dive into every benchmark where Qwen3.8-27B beats frontier proprietary models, check our tested harness configs for Pi, OpenCode, and Kilo Code, compare open-weight vs proprietary models across 716 BenchLM evaluations, or inspect our full European open-weight model catalog and flat-rate subscription tiers.
Sovereign AI inference, OpenAI-compatible, EU-hosted. Benchmarks, new models (Mistral, Qwen, GLM), and cost insights. One email, no spam.
Please check your inbox to confirm your subscription.
100% European infrastructure · Zero spam · Unsubscribe anytime.
Frequently asked questions
Why did Kimi K3.1 Pro and Claude Opus 5.5 scores change on Artificial Analysis Intelligence Index v5?
On October 27, 2026, Artificial Analysis replaced Index v4 with Intelligence Index v5.0.1, introducing nine harder, agentic evaluations including Terminal-Bench 4.0, FrontierCode 2.0, AA-Briefcase v1.1, and AutomationBench-AA. On this stricter scale, Claude Opus 5.5 (max) scores 59 points ($10.04/task) and Kimi K3.1 Pro leads open-weight models at 46 points ($3.68/task).
Is Kimi K3.1 Pro better than DeepSeek V4.1 Flash 1017 for software engineering?
It depends on the harness and latency budget. Kimi K3.1 Pro beats DeepSeek V4.1 Flash 1017 on Terminal-Bench 4.0 (65.7% vs 42.3%) and scanned document QA (OfficeQA Pro v2, 64.4% vs 40.4%), but DeepSeek V4.1 Flash 1017 scores higher on multi-turn repository bug fixing (DeepSWE v1.1, 77.0% vs 75.5%), streams 5.3 times faster (174 tok/s vs 33 tok/s), and costs 6.1 times less per task ($0.60 vs $3.68).
How much GPU memory (VRAM) does Kimi K3.1 Pro require compared to Qwen3.8-27B?
Kimi K3.1 Pro has 2.8 trillion total parameters (104 billion active per token) and requires roughly 1.4 TB of VRAM even with its native 4-bit MXFP4 weights, meaning it must run across a multi-node GPU cluster. Qwen3.8-27B has 27 billion parameters and fits on a single 80 GB GPU in BF16 (~54 GB) or a 24 GB GPU in INT4 (~16 GB), while beating both models on LlamaIndex’s ExtractBench (89.75%).
How do reasoning effort settings affect Qwen3.8-27B and GLM-5.3 latency on AA Index v5?
At maximum thinking effort (xhigh / max), Qwen3.8-27B generates 128 million output tokens (122.7s time to first answer token, 39.8 minutes per task) and GLM-5.3 generates 135 million output tokens (96.0s time to first answer token, 30.7 minutes per task, $5.01/task). Lowering effort to low cuts GLM-5.3 task duration in half to 15.5 minutes ($1.57/task) and reduces Qwen3.8-27B time to first token to 18.3 seconds (or 5.5 seconds in non-reasoning mode).
How can teams run GLM-5.2 and Qwen3.8-27B in Europe with predictable monthly costs?
We serve GLM-5.2 and Qwen3.8-27B directly on our flat €39/month Core subscription (600 million monthly tokens, ~€0.065/1M tokens) inside 100% green European datacenters with certified Zero Data Retention, and support custom Hugging Face checkpoints on dedicated European GPUs.
You have the architecture and the open-source workflow. Now give it a sovereign European inference layer built for production: fast access to GLM-5.2, Qwen3.5, Mistral, Llama, DeepSeek, and Brick Complexity Pro with zero data retention and no new SDK to learn.
Build, test, and deploy with no infrastructure to maintain. Core plans from €11.70/mo (70% off first 3 months).
Get the full implementation, configuration scripts, evaluation benchmarks, and ready-to-run templates.
Meet engineers building with coding agents, private RAG, local LLMs, and production open-weight models.
Scaling beyond a proof of concept or running regulated EU workloads? Get a tailored architecture and commercial plan.