# The Build-Verify Loop: stop your AI agent from claiming victory before the tests pass

Your coding agent just reported success. "All done — the fix is implemented and working." You open the pull request. The tests fail. The file it edited isn't even the one the bug lives in.

Sound familiar? It should. This is the single most common failure mode of autonomous agents, and it has nothing to do with model intelligence. The agent genuinely *believed* it was finished, because nothing in its execution environment forced it to check.

This is the first article in our Harness Engineering Patterns series. Each piece takes one pattern from our benchmark-backed guide to harness engineering and turns it into a working implementation. Today: the **build-verify loop** — a core harness pattern behind open-source coding agents achieving ~42.5% accuracy on Terminal Bench 2.0 (matching closed agents like Claude Code), by ensuring task execution is rigorously verified against executed evidence.

We'll build a verification-gated agent in production using OpenCode as the coding agent and an OpenAI-compatible European endpoint. No hyperscaler APIs required.

> **A build-verify loop forces an agent to plan, implement, verify its output against executed evidence, and fix failures — as a control-flow requirement, not a polite suggestion in a prompt.**

## Why Agents Lie About Success (Without Meaning To)

LLMs are trained to produce plausible completions. When an agent finishes writing code, the most plausible next token sequence is a confident summary of the work. The model has no internal signal that distinguishes "code that runs" from "code that reads well."

Three failure modes follow:

1. **Premature termination** — the agent stops after the first plausible implementation, never running a single test.
2. **Favourable self-reading** — when it does glance at output, it interprets ambiguity in its own favour. A stack trace becomes "a minor warning."
3. **Spec drift** — after several edits, the agent optimizes for its own rewritten understanding of the task, not the original specification.

None of these are reasoning failures. They're *architecture* failures: the system allowed exit without evidence. The fix is twofold — explicit plan-build-verify-fix guidance in the system prompt, plus execution sandboxes (such as Harbor or containerized CI runners) and verification gates that intercept exit attempts and demand executed test evidence. Prompt alone isn't enough; the execution gate makes it structural.

That's the design principle to steal: **verification must be a gate in the control flow, not a paragraph in the prompt.**

## The Pattern at a Glance

| Stage | What happens | What can go wrong without it |
|---|---|---|
| Plan & discover | Inspect task, environment, constraints, available test commands | Solving the wrong problem convincingly |
| Build | Smallest viable implementation, interfaces preserved | Over-engineered or brittle output |
| Verify | Execute tests, read full output, compare against the *original* spec | Fluent but broken "completion" |
| Fix | Diagnose root cause, modify, re-verify | Infinite "looks done to me" loop |

One detail from LangChain's write-up deserves its own paragraph: they told the agent *how it would be evaluated* — programmatic tests, strict file paths, edge cases rather than happy paths. This shifts agent behaviour from "produce something reasonable" to "produce an artefact that survives an external check." It costs one sentence in the prompt. Use it.

## Hands-On: A Verification-Gated Agent

Here's a real production setup. A six-person European team runs a B2B SaaS for electronic invoicing (FatturaPA, the Italian e-invoicing standard). Fifteen to twenty-five bug-fix and small-feature issues land in their backlog every week. They use **OpenCode** — the open-source, terminal-native coding agent — as an autonomous issue resolver wired into GitHub Actions, with **Regolo** as the LLM endpoint. Before adding the verification gate, 34% of agent-opened PRs failed CI on the first run.

> **How OpenCode and GitHub Actions connect:** OpenCode is not an external cloud SaaS or desktop app; it is a CLI tool (`npm install -g opencode`). GitHub Actions serves as the headless execution environment: when triggered by a comment, the CI runner spins up, clones the repo, installs the `opencode` CLI directly inside the runner, and executes `opencode run` locally. OpenCode edits files in the runner's workspace and sends HTTPS requests to Regolo's OpenAI-compatible endpoint. The verification gate then runs on the exact same machine before opening a PR.

The architecture:

```
GitHub Issue (/opencode comment)
        │
        ▼
GitHub Actions runner (self-hosted, EU)
        │
        ├── OpenCode agent
        │       ├── Provider: Regolo (OpenAI-compatible)
        │               ├── Model: qwen3-coder-next
        │       └── Prompt: build-verify rules
        │
        ├── Verification gate (CI script)
        │       ├── Full pytest run, mandatory
        │       ├── Forbidden-path check
        │       ├── Secret scan on diff
        │       └── PR blocked if any gate fails
        │
        └── PR opened → CI pipeline → human review
Code language: Bash (bash)
```

### Step 1 — Configure OpenCode with the OpenAI-compatible endpoint

`~/.config/opencode/opencode.json`:

```
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "regolo": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Regolo EU",
      "options": {
        "baseURL": "https://api.regolo.ai/v1",
        "apiKey": "{env:REGOLO_API_KEY}"
      },
      "models": {
        "qwen3-coder-next": {
          "name": "Qwen3 Coder Next (EU)"
        },
        "Llama-3.3-70B-Instruct": {
          "name": "Llama 3.3 70B Instruct"
        }
      }
    }
  },
  "model": "regolo/qwen3-coder-next",
  "small_model": "regolo/Llama-3.3-70B-Instruct",
  "agent": {
    "build": {
      "mode": "primary",
      "model": "regolo/qwen3-coder-next",
      "prompt": "{file:./prompts/build-verify.md}"
    }
  }
}
Code language: JavaScript (javascript)
```

Two models, one endpoint. The strong coding model does the work; the smaller one handles classification and auxiliary calls. Swapping either is a config change, not a code change — that's the whole point of an OpenAI-compatible gateway.

### Step 2 — The verification-aware agent prompt

`prompts/build-verify.md`:

```
You are an autonomous coding agent fixing GitHub issues for a FatturaPA SaaS.

WORKFLOW — mandatory, no shortcuts:
1. READ the issue and the relevant source files.
2. RUN the existing test suite: `pytest tests/ -x -q`. Note which tests pass BEFORE your change.
3. IMPLEMENT the smallest fix. Do not refactor unrelated code.
4. RE-RUN the full test suite. Read the complete output.
5. If any test fails: diagnose the root cause, fix, and re-run.

COMPLETION RULES:
- You will be evaluated by CI. The pipeline runs `pytest tests/ -v --tb=short` and checks exact file paths.
- Write or update at least one test that covers your change.
- NEVER open a PR where `pytest` exits non-zero.
- If you cannot make tests pass after 5 fix attempts, stop and write a comment
  on the issue explaining what you tried and what is blocked.

FORBIDDEN:
- Do not modify files in `deploy/` or `.github/`.
- Do not commit secrets, API keys, or `.env` files.
- Do not change the database schema without a migration file.
Code language: Bash (bash)
```

Notice the two structural moves from the pattern. The evaluation method is disclosed ("evaluated by CI... exact file paths"). And there's a disclosure path for genuinely blocked work — an agent that *can't* declare a task unresolvable will eventually learn to lie about the blockers. I've seen it happen, and the resulting PRs are worse than honest silence.

### Step 3 — The verification gate in CI

The prompt sets expectations; the gate enforces them. OpenCode executes tool calls, but PR creation passes through a wrapper script that cannot be persuaded by confident prose.

`scripts/verify_gate.sh`:

```
#!/bin/bash
# Verification gate — runs inside the GitHub Actions runner
# Blocks PR creation if any check fails

set -euo pipefail

WORKSPACE="${1:-$PWD}"
LOG_FILE="$WORKSPACE/.verify_gate.log"

echo "=== BUILD-VERIFY GATE ===" | tee "$LOG_FILE"

# --- Gate 1: tests must pass ---
echo "[GATE 1] Running full test suite..." | tee -a "$LOG_FILE"
if ! pytest "$WORKSPACE/tests/" -v --tb=short >> "$LOG_FILE" 2>&1; then
    echo "BLOCKED: pytest exited non-zero. PR creation cancelled." | tee -a "$LOG_FILE"
    tail -30 "$LOG_FILE"
    exit 1
fi
echo "[GATE 1] PASSED" | tee -a "$LOG_FILE"

# --- Gate 2: at least one test file touched ---
CHANGED_TESTS=$(git diff --name-only HEAD | grep -c "tests/" || true)
if [ "$CHANGED_TESTS" -eq 0 ]; then
    echo "WARNING: no test files modified — flagged for review" | tee -a "$LOG_FILE"
fi

# --- Gate 3: forbidden paths untouched ---
FORBIDDEN=$(git diff --name-only HEAD | grep -E "(deploy/|\.github/|\.env)" || true)
if [ -n "$FORBIDDEN" ]; then
    echo "BLOCKED: forbidden files modified: $FORBIDDEN" | tee -a "$LOG_FILE"
    exit 1
fi

# --- Gate 4: no secrets in the diff ---
if git diff HEAD | grep -iE "(api_key|secret|password|token).*=.*['\"][^'\"]{8,}" > /dev/null; then
    echo "BLOCKED: potential secret detected in diff" | tee -a "$LOG_FILE"
    exit 1
fi

echo "=== ALL GATES PASSED ===" | tee -a "$LOG_FILE"
exit 0
Code language: Bash (bash)
```

Four gates, each executable and each independent of the model's opinion. Gate 1 is the core of the pattern: executed test evidence or no PR. Gates 3 and 4 are policy enforcement — cheap to write, expensive to skip.

### Step 4 — The GitHub Actions workflow

`.github/workflows/opencode-agent.yml`:

```
name: OpenCode Agent

on:
  issue_comment:
    types: [created]

jobs:
  agent-fix:
    if: contains(github.event.comment.body, '/opencode')
    runs-on: self-hosted  # EU runner

    steps:
      - uses: actions/checkout@v4
        with:
          ref: main
          fetch-depth: 0

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt pytest
          npm install -g opencode

      - name: Run OpenCode agent
        env:
          REGOLO_API_KEY: ${{ secrets.REGOLO_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          ISSUE_BODY=$(gh issue view ${{ github.event.issue.number }} --json title,body -q '.title + "\n" + .body')
          PROMPT_RULES=$(cat prompts/build-verify.md)
          opencode run "$PROMPT_RULES" "$ISSUE_BODY" --agent build

      - name: Verification gate
        run: bash scripts/verify_gate.sh "${{ github.workspace }}"

      - name: Create PR
        if: success()
        uses: peter-evans/create-pull-request@v6
        with:
          title: "fix: ${{ github.event.issue.title }} [agent]"
          body: |
            ## Automated fix by OpenCode agent
            - Issue: #${{ github.event.issue.number }}
            - Model: qwen3-coder-next (EU, OpenAI-compatible endpoint)
            - Verification gate: PASSED
            See `.verify_gate.log` for full test output.
          branch: agent/fix-${{ github.event.issue.number }}
          labels: agent-generated, needs-review
Code language: YAML (yaml)
```

The workflow step ordering *is* the pattern: GitHub Actions spins up an isolated runner, installs OpenCode CLI, lets the agent fix the code locally in the runner, evaluates the gate, and opens the PR only on success. The `if: success()` on the PR step means a blocked gate produces a failed workflow run — visible, logged, and impossible to confuse with a completed task.

### Impact on Agent Reliability

By enforcing a deterministic verification gate, the system shifts from relying on prompt compliance to structural enforcement:

1. **Elimination of Broken PRs**: The CI runner blocks any PR creation if `pytest` exits non-zero, ensuring that non-compiling code or failing test cases never reach human review.
2. **Internal Self-Correction**: When tests fail during the run, the error trace feeds directly back into the agent's loop, allowing it to diagnose and patch its own mistakes before concluding the task.
3. **Optimized Engineering Efficiency**: While executing test suites consumes additional LLM tokens during the run, it eliminates the major bottleneck of human developers manually debugging broken agent outputs.

---

## What changes in practice

**Detail one: define what counts as a test.** The bash gate runs `pytest` because in this case it's a python project, instead if part of your stack is Go or TypeScript, extend the gate to the real toolchain or declare the test command explicitly in the agent's startup context.

**Detail two: watch for gaming.** Agents occasionally satisfy gates in letter rather than spirit — running one trivial test file instead of the suite, or committing a test that asserts nothing. Mitigations: require the full output in the log artifact, check that changed source files have corresponding changed tests (Gate 2 exists for this), and review every blocked run. Those blocks are your best failure data. Cluster them after a month and you'll see your agent's favourite evasion strategies, ranked.

---

## Generalizing beyond coding

The loop transfers to any agent whose output can be checked against external evidence:

| Agent type | Build | Verify | Evidence |
|---|---|---|---|
| Coding agent | Write the patch | Run tests, check exact paths | Exit code 0, artefacts exist |
| Document processor | Extract fields to JSON | Schema validation, confidence threshold | `jsonschema` passes, required fields present |
| RAG assistant | Draft grounded answer | Claim-to-source mapping | Every claim carries a retrieved source ID |
| API automation | Prepare the mutation | Dry-run or read-back | Read-after-write confirms intended state |

The coding version is easiest because the feedback channel — a test suite — already exists. For document and RAG agents you'll build the checker yourself: a schema validator, a citation-coverage function, a read-back call. Build it anyway. An agent without an executable check is an agent you're reviewing by hand, which defeats the purpose.

---

## FAQ

### What is a build-verify loop in AI agents?

A build-verify loop is a harness pattern that forces an agent through four control-flow stages — plan, build, verify against executed evidence, and fix — before it may complete a task. Verification is enforced by a gate in the execution environment, not merely suggested in the prompt.

### Why do AI agents claim success when tests fail?

Language models generate plausible completions, and a confident success summary is the most plausible text after writing code. Without an external evidence requirement in the execution loop, the agent has no signal separating "code that reads well" from "code that runs."

### How do you enforce verification in a coding agent?

Put the gate where the agent cannot bypass it: a CI script that runs the full test suite and blocks PR creation on failure, plus forbidden-path checks and secret scanning. The prompt declares the rules; the pipeline enforces them.

### Does a verification loop increase agent cost?

Per attempted task, yes — typically 20–30% more tokens for test execution and output inspection. However, total engineering cost drops substantially because human developers no longer spend time debugging broken or non-working agent PRs.

### Can the build-verify loop work outside coding?

Yes. Document processors use JSON Schema validation, RAG assistants use claim-to-source mapping, and API automation uses read-after-write confirmation. The evidence type changes; the gate structure does not.

### Which tools and models support this pattern without hyperscaler APIs?

OpenCode supports any OpenAI-compatible provider through the `@ai-sdk/openai-compatible` package, including EU-hosted endpoints serving open-weight models. The verification gate lives in your CI, so it works identically regardless of which model sits behind the endpoint.

---

## Ship Private AI. Not Infrastructure.

You have the private AI App architecture, bow give it an inference layer built for production.

**Regolo** gives European teams fast, OpenAI-compatible access to Mistral, Llama, Qwen, DeepSeek, GLM, and more — with zero data retention, EU data residency, and no new SDK to learn.

Change your `base_url`. Keep your LangChain code. Start shipping.

### 🚀 [Start your 30-day free trial →](https://regolo.ai/?utm_source=blog&utm_medium=cta&utm_campaign=private-rag)

Build, test, and deploy with no infrastructure to maintain.
**No credit card. No migration project. No compromise on data control.**

### 💬 [Join the Regolo Discord →](https://discord.gg/bqGrVJHeF)

Meet builders working on private RAG, local LLMs, LangChain, Ollama, and production AI systems. Share your setup, get feedback from the community, and speak directly with the Regolo team.

### 🤝 [Talk to an AI Infrastructure Engineer →](https://regolo.ai/contact?utm_source=blog&utm_medium=cta&utm_campaign=private-rag)

Running a sensitive workload, scaling beyond a proof of concept, or assessing a managed EU inference provider? Get a tailored architecture and commercial proposal for your team.

### 📂 [Clone the GitHub repository →](https://github.com/regolo-ai/tutorials/)

Get the full implementation from this guide: ingestion scripts, ChromaDB setup, hybrid retrieval, the **30-Question RAG Floor**, evaluation examples, and deployment configuration.

> **Private AI should not require a private data center.**
> Regolo gives your team an EU-native path from local experimentation to production-grade inference.

---

### Build with Regolo

- **Discord:** [Join the community →](https://discord.gg/bqGrVJHeF)
- **GitHub:** [Explore open-source workflows →](https://github.com/regolo-ai/tutorials/)
- **X / Twitter:** [Follow @regolo\_ai →](https://x.com/regolo_ai)
- **Reddit:** [Join the community →](https://www.reddit.com/r/regolo_ai/)
- **Documentation:** [Read the API docs →](https://docs.regolo.ai)
- **Contact:** [Talk to the team →](https://regolo.ai/contact)

---

*Built with ❤️ by the Regolo team. Questions? [regolo.ai/contact](https://regolo.ai/contact)* or chat with us on [Discord](https://discord.gg/bqGrVJHeF)