Harness engineering for Qwen3.8 27B: tested configs for Pi, OpenCode, and Kilo Code

You already know Qwen3.8 27B is the first open-weight model at this size that holds up on long-horizon agentic coding — 61.7 on SWE-bench Pro, 42.2 on DeepSWE 1.1 in Qwen’s official eval. But independent benchmarks tell the more useful story:

Setup (same checkpoint)	DeepSWE 1.1
Qwen official Claude Code (temp 1.0, top_p 0.95, 256K ctx)	42.2
Best reproducible Claude Code run (community benchmark)	35.2
Pi with effort xhigh + tool-error handling fixes	46.0 (52/113 tasks)

Same weights, 7-point drop in one harness, 4 points above Qwen’s own published score in another. ~10-point spread = pure harness engineering. This guide covers the levers that produce it, in order of measured impact, with working configs for Pi, OpenCode, and Kilo Code (VS Code). All configs point to Regolo as the serving endpoint — FP8 precision, no self-hosting required.

---

Prerequisite: the serving contract

Every agent below talks to an OpenAI-compatible endpoint. Two requirements are non-negotiable — every subsequent optimization assumes them:

1. Working tool calling. On vLLM that means --enable-auto-tool-choice --tool-call-parser hermes. If tool calls don’t work, nothing else in this guide matters — the agent can’t act.

2. Reasoning content exposed in the API response. The endpoint must return the model’s thinking (typically reasoning_content field, vLLM/Qwen style) instead of discarding it, so the harness can replay it on the next turn.

Regolo: both requirements are handled — tool calling works out of the box, reasoning content is exposed in the response. Skip the vLLM flags section if you’re using the API.

Sampling: temperature 1.0, top_p 0.95 in thinking mode — the exact config behind Qwen’s official DeepSWE eval. Qwen’s model card recommends it explicitly. Lower temperatures degrade agentic behavior on this model family.

---

Lever 1: Preserve reasoning across turns

The data: Early Claude Code runs with this model scored 35.2 instead of Qwen’s 42.2. Closing that gap required fixing how the harness handled reasoning traces between turns. Turning preservation off makes runs at effort xhigh perform like medium — while consuming more tokens, because the model re-derives conclusions it had already reached. Bonus mechanism: preserved thinking blocks stay identical → better KV-cache reuse.

How to configure:

Agent	Configuration
Pi	"reasoning": true in the model entry of ~/.pi/agent/models.json
OpenCode	compatibility.reasoningField: "reasoning_content" in provider config
Kilo Code	Verify in a multi-turn session — if the model re-explains itself, reasoning is being stripped in the replay path

The OpenCode silent failure: without reasoningField, reasoning vanishes silently between turns. No error, no warning — the agent just gets dumber without telling you.

---

Lever 2: Calibrate reasoning effort per task

Qwen3.8 replaces the binary thinking on/off toggle with graded effort levels with calibration:

Effort	When to use	Impact
xhigh	Multi-file refactors, debugging with unclear symptoms, long-horizon tasks	Highest completion rate (46.0 DeepSWE)
medium	Daily default	Near-xhigh accuracy, fraction of the token budget
low	Mechanical edits, fast completions	Fast, low cost

The mistake: running everything at maximum effort doubles token spend without measurable gains on simple tasks, while with preserved thinking, every extra reasoning token stays in context for the rest of the session → compounding cost.

Configuration:

Agent	How to set it
Pi	/model picker; if your server doesn’t understand reasoning_effort, set "supportsReasoningEffort": false in compat
OpenCode (v2)	Model variant with model#high / model#medium syntax or settings.reasoningEffort
Kilo Code	Two separate provider profiles (one per effort level), manual switch

---

Lever 3: Context compaction — biggest lever after reasoning

Even the 262K window fills on long agentic tasks, when it does, either the session dies or the harness must summarize.

# The general formula 
effective context = contextWindow − reserveTokens − responseRunwayCode language: PHP (php)

Compaction triggers when contextTokens > contextWindow − reserveTokens. Lossy by design: older messages get summarized, recent ones stay verbatim (Pi keeps the last ~20K tokens; default reserve = 16384).

The full history survives in the session’s JSONL, but the model never sees it again.

Pi — the most mature stack:

Command/Extension	What it does
/compact [instructions]	Manual compaction; instructions focus the summary
/autocompact	Toggle automatic threshold (on by default)
@pi-unipi/compactor	Lossless with zero-LLM sentinel; /unipi:session-recall <query> for BM25/regex recall
pi-compact-plus	Tool-output pruning — single most effective lever for coding agents (tool outputs fill the window, not conversation text)
pi-context-tools	Exposes context_info and compact_context as tools — agent self-compacts mid-task

OpenCode and Kilo Code — curate instead of compress:

Agent	Mechanism
OpenCode	AGENTS.md in project root (generate with /init). Lookup order: AGENTS.md → CLAUDE.md, first match wins. Durable knowledge for a few hundred tokens
Kilo Code	Memory Bank — structured markdown in .kilo/rules/memory-bank/. Switch to Architect mode with your strongest model, have it analyze the repo and write the bank files

The logic: compaction is reactive (you lose detail when the window fills); rules and memory banks are proactive (durable knowledge enters context cheaply and on purpose).

---

Lever 4: Persistent memory — options honestly evaluated

The myth to correct: @modelcontextprotocol/server-memory (the one with MEMORY_FILE_PATH) is not an embedding database. It’s a plain JSON knowledge graph (entities, relations, observations) with exact-match retrieval. The file path is the entire storage story. Fine for small, structured fact sets, zero dependencies. But it can’t answer “what did we decide about the retry logic?” unless the phrasing matches closely.

For real recall, you want an embedding-backed server:

Server	Architecture	Pros
mcp-memory-service (doobidoo)	SQLite + sqlite-vec + local embeddings (all-MiniLM-L6-v2 via ONNX)	No API calls, data never leaves the machine, ~5ms read latency
@provos/memory-mcp-server	sqlite-vec cosine + FTS5 BM25 + optional LLM summarization	Hybrid retrieval — technically correct choice for coding

Why hybrid beats pure vector: embeddings handle paraphrase (“retry logic” ≈ “backoff strategy”) but fail on exact identifiers — a function named compact_context, an error string, a config key. BM25 catches those precisely. Coding memory is identifier-heavy → fusion is the correct choice, not a nice-to-have.

Project-scoped config:

// .opencode/opencode.json
{
  "mcp": {
    "memory": {
      "type": "local",
      "command": ["uvx", "mcp-memory-service"],
      "environment": {
        "MCP_MEMORY_STORAGE_PATH": "/absolute/path/to/project-memory.db"
      },
      "enabled": true
    }
  }
}Code language: JSON / JSON with Comments (json)

// .kilo/kilo.jsonc
{
  "mcp": {
    "memory": {
      "type": "local",
      "command": ["uvx", "mcp-memory-service"],
      "environment": { "MCP_MEMORY_STORAGE_PATH": "/absolute/path/to/project-memory.db" },
      "enabled": true,
      "timeout": 10000
    }
  }
}Code language: JSON / JSON with Comments (json)

Pi: pi install npm:pi-mcp-adapter, then declare in ~/.pi/agent/mcp.json with "lifecycle": "keep-alive" — server stays warm, no process-startup latency per query.

How they compose: AGENTS.md / Memory Bank = project knowledge (what this codebase is, what was decided). Memory MCP = cross-session working knowledge (recurring patterns, your preferences, bugs already diagnosed). Different problems, both belong in a serious setup.

---

Lever 5: Tool reliability and error recovery

The single largest measured gain in the independent DeepSWE evals — going from 42.2 to 46.0 — came from handling non-zero tool exit codes gracefully in Pi; not from prompts, not from more compute: from the agent loop surviving failures, recovering incomplete sessions, and committing patches reliably.

What this means in practice:

• Harnesses that retry or contextualize tool failures instead of aborting the loop. In Pi, controllable via extensions — the community has documented specific fixes that push scores above Qwen’s official results.
• OMP (a Pi variant) reports slightly better results while consuming 2–3× the tokens. Trade-off only sensible for benchmarks or high-stakes one-off tasks.
• Custom tool wrappers: return structured, informative errors (exit code + stderr + what the agent should try next). An error the model can read is a recoverable state; a stack trace dump is context pollution.

---

Appendix: base configurations

All configs below use Regolo as the endpoint, replace <KEY> with your Regolo API key.

Pi — ~/.pi/agent/models.json

{
  "providers": {
    "regolo": {
      "baseUrl": "https://api.regolo.ai/v1",
      "api": "openai-completions",
      "apiKey": "<KEY>",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false
      },
      "models": [
        {
          "id": "Qwen/Qwen3.8-27B",
          "name": "Qwen3.8 27B",
          "reasoning": true,
          "input": ["text", "image"],
          "contextWindow": 262144,
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
        }
      ]
    }
  }
}Code language: JSON / JSON with Comments (json)

supportsDeveloperRole: false — many Qwen-compatible endpoints reject the developer role; without the flag, requests fail with unhelpful errors. Reload is automatic via /model.

OpenCode — ~/.config/opencode/opencode.json

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "regolo": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Qwen3.8 (Regolo)",
      "options": {
        "baseURL": "https://api.regolo.ai/v1",
        "apiKey": "{env:REGOLO_API_KEY}"
      },
      "models": {
        "Qwen/Qwen3.8-27B": {
          "name": "Qwen3.8 27B",
          "limit": { "context": 131072, "output": 32768 }
        }
      }
    }
  },
  "model": "regolo/Qwen/Qwen3.8-27B",
  "small_model": "regolo/Qwen/Qwen3.8-27B"
}Code language: JSON / JSON with Comments (json)

Kilo Code — VS Code extension

1. Kilo panel → Settings → Providers → new profile
2. API Provider: OpenAI Compatible
3. Base URL: https://api.regolo.ai/v1
4. API Key: your Regolo key
5. Context Window Size (num_ctx) ≥ 32768 — below 32K quality degrades significantly
6. Note: Kilo CLI does not support local Ollama models; local models are an extension feature

---

Quick-reference matrix

Lever	Pi	OpenCode	Kilo Code
Reasoning preservation	reasoning: true (native)	compatibility.reasoningField	Verify provider replay
Effort calibration	Native, up to xhigh	model#variant, reasoningEffort	Separate profiles
Compaction	Core /compact + extensions (UniPi, compact-plus)	AGENTS.md curation	Memory Bank curation
Persistent memory	pi-mcp-adapter, keep-alive	mcp key, project-scoped	kilo.jsonc + native Memory Bank
Tool reliability	Extension-controlled, best documented	Depends on provider	Depends on provider
Token budgets	contextWindow in models.json	Explicit limit.*	num_ctx ≥ 32K
Precision	FP8 via Regolo (no config needed)	FP8 via Regolo (no config needed)	FP8 via Regolo (no config needed)

---

---

FAQ

Why do benchmark scores for the same model vary so much across tools?

Because you’re not just benchmarking the model — you’re benchmarking the model plus the harness. With the identical Qwen3.8 27B checkpoint, DeepSWE 1.1 scores ranged from 35.2 (Claude Code, before reasoning-preservation fixes) to 46.0 (Pi with tool-error handling fixes), against Qwen’s own published 42.2. That ~10-point spread is entirely harness engineering: reasoning replay, effort calibration, context management, and tool-error recovery.

What exactly is “preserve thinking” and why does it matter so much?

Qwen3.8 was trained to keep its reasoning blocks in the conversation across turns. When a harness strips them between turns, the model re-derives conclusions it had already reached — generating more tokens while performing worse: runs at effort xhigh without preserved reasoning drop to medium-level results. It also hurts performance indirectly, since preserved thinking blocks enable KV-cache prefix reuse on the serving side. In Pi it’s the reasoning: true flag; in OpenCode it’s compatibility.reasoningField: "reasoning_content".

Why temperature 1.0? Isn’t lower temperature more precise?

Not for this model family in thinking mode. Qwen’s model card explicitly recommends temperature=1.0, top_p=0.95, and that’s the exact configuration behind the official DeepSWE 1.1 evaluation. Chat-tuning intuition (“lower temp = more deterministic = better”) measurably degrades agentic behavior here. If you’re seeing repetitive or degraded outputs at temp 1.0, the problem is almost always the harness, not the sampling.

Which agent should I pick — Pi, OpenCode, or Kilo Code?

• Pi if you want maximum control over the agent loop: native effort levels up to xhigh, the richest compaction extension ecosystem, and the best-documented tool-error handling (it’s the harness behind the 46.0 DeepSWE score).
• OpenCode if you live in the terminal and want AGENTS.md project rules plus easy MCP integration with a large catalog of providers.
• Kilo Code if you work inside VS Code: it’s the only one of the three with a native Memory Bank (structured project memory as markdown in .kilo/rules/memory-bank/), and its Marketplace installs agents, skills, and MCP servers preconfigured.

All three talk to the same OpenAI-compatible endpoint, so switching between them is cheap.

Is the official @modelcontextprotocol/server-memory good enough for persistent memory?

Only for small, structured fact sets. It’s a plain JSON knowledge graph with exact-match retrieval — no embeddings, no semantic search. That’s why its config is just a MEMORY_FILE_PATH. For real recall (“what did we decide about the retry logic?”), use an embedding-backed server: mcp-memory-service (SQLite + sqlite-vec, local ONNX embeddings, ~5ms reads, nothing leaves your machine) or @provos/memory-mcp-server (hybrid: vector search fused with BM25 keyword ranking).

Why hybrid retrieval instead of pure vector search for coding memory?

Because embeddings are great at paraphrase and bad at exact identifiers. A vector store will connect “retry logic” with “backoff strategy,” but it won’t reliably find compact_context or an exact error string — and coding memory is identifier-heavy. BM25 keyword search handles those precisely, so fusing both (like @provos/memory-mcp-server does with sqlite-vec + FTS5) gives you the technically correct retrieval for this domain.

Should I scope the memory MCP globally or per project?

Per project. A single global memory store silently cross-contaminates repos — the agent recalls decisions from a different codebase and applies them to yours. Put the mcp block in the project-level config (.opencode/opencode.json, .kilo/kilo.jsonc) with a per-project storage path, not in the global config.

Do I need both a Memory Bank / AGENTS.md and a memory MCP?

They solve different problems and compose well. AGENTS.md (OpenCode) and the Memory Bank (Kilo Code) hold project knowledge — architecture, conventions, work state — that you curate deliberately and that enters context cheaply. A memory MCP holds cross-session working knowledge — recurring patterns, preferences, already-diagnosed bugs. The curated layer delays the first compaction; the MCP layer prevents re-learning across sessions.

What does compaction actually lose, and can I avoid the loss?

Compaction summarizes older messages to free context (Pi keeps the last ~20K tokens verbatim; auto-compaction triggers when contextTokens > contextWindow − reserveTokens, default reserve 16384). The summary is lossy by design — full history survives in the session JSONL, but the model never sees it again. Mitigations: focus the summary with /compact [instructions], use @pi-unipi/compactor for lossless compaction (pre-compaction messages stay searchable via BM25 recall), and use pi-compact-plus for tool-output pruning, since tool outputs — not conversation text — are what actually fill the window.

How much context do I really need? The model supports 262K.

Kilo Code’s own docs set the practical floor at 32K — below that, quality degrades significantly. Budget upward from there based on your VRAM, not the theoretical maximum. Two failure modes to avoid: declaring more context than your server actually allocates (silent mid-task truncation in OpenCode, which assumes 200K/32K for unknown models unless you set limit.* explicitly), and undersized output budgets (reasoning gets cut mid-thought, and with preserved thinking the model re-reads its own truncated reasoning next turn).

---