If you want to make a coding LLM more reliable, start by upgrading the harness instead of swapping models. LangChain’s middleware system lets you control agent execution with retries, limits, and human approval, while external checkers such as pytest turn “looks correct” into “verified correct.” docs.

This guide shows how to build a real Python CI repair agent with a self-verification loop, then benchmark the harness so you can prove whether it actually improves results.

The target use case is painfully common: a Python team has a growing pile of flaky or broken CI runs, engineers waste time opening AI-generated patches that still fail tests, and nobody trusts auto-fix agents enough to let them touch production repos. That pain is not mainly a model problem. It is a completion-gate problem.

What this agent fixes
Why this use case matters for companies
Project structure
Requirements
How the self-verification loop works
Deterministic completion gate
Runnable benchmark
What to measure
FAQ

What this agent fixes

The demo agent is built for a real operational pain: an API endpoint returns the wrong status code or crashes when an optional field is missing, the test suite already exposes the bug, and the team wants an agent to propose the smallest safe fix without shipping a fake “done.”

In this design, the LLM is the maker, but `pytest`, linting, and repository policies are the checker. The model is never the final authority on completion. docs.

A self-verification loop means the agent writes code, runs checks, reads the exact failure output, retries with specific feedback, and escalates cleanly when the retry budget is exhausted.

LangChain supports this pattern because middleware runs inside the compiled agent loop and can enforce limits, early termination, and other runtime rules. docs.

Target pain	What the code does	Practical benefit
AI says “fixed” but tests still fail	Runs `pytest` and `ruff` outside the LLM before accepting completion	Reduces false-complete PRs
Agent retries blindly	Injects the exact command, exit code, stdout, and stderr into the next attempt	Faster, more targeted corrections
Agent loops and burns budget	Caps model calls, tool calls, and repair attempts	Predictable cost and latency
Agent changes risky files	Blocks CI, infrastructure, migration, and lock-file paths	Lower blast radius
Failed attempts disappear	Returns the failed checks and final `git diff` for human escalation	Debuggable handoff instead of a dead end

Why this use case matters for companies

The highest-friction failure mode in AI coding teams is false completeness: the patch looks plausible, the model explains it confidently, and the CI still fails. LangChain’s public harness-engineering material explicitly highlights self-verification as a practical lever for improving agent outcomes without changing the underlying model.

The important shift is that “done” becomes a system decision, not a model statement. The LLM remains useful for diagnosis and patch generation, but the harness validates the claim against the actual repository state.

The use case also maps cleanly onto measurable business pain, every false-positive fix wastes reviewer time, burns inference budget, and reduces trust in the system.

Project structure

ci-repair-agent/
├── agent.py
├── tools.py
├── evaluation.py
├── requirements.txt
├── benchmarks/
│   └── tasks.json
└── tests/
    ├── test_evaluation.py
    └── test_benchmark.py

The goal is not to build a giant platform, but it’s to give you a harness you can run, inspect, and extend in one sitting.

How the self-verification loop works

The loop is simple enough to reason about and strict enough to stop nonsense:

Read the bug report and relevant files.
Generate the smallest safe patch.
Run pytest.
Run ruff.
If all required checks pass, accept the result.
If any check fails, feed the exact failure output back to the agent.
Retry within a fixed attempt budget.
If the budget is exhausted, return a structured failure and escalate.

LangChain middleware exists specifically to control this kind of loop, including limits and early termination behavior.

On the right side the complete working schema of this agent that you can see how it works for each step and edit as you need.

                    ┌───────────────────────────────┐
                    │ Bug report / failing CI run    │
                    │ e.g. POST /users returns 500   │
                    └───────────────┬───────────────┘
                                    │
                                    v
                    ┌───────────────────────────────┐
                    │ LangChain agent               │
                    │ Reads files and failing tests │
                    │ Produces smallest safe patch  │
                    └───────────────┬───────────────┘
                                    │
                                    v
                    ┌───────────────────────────────┐
                    │ Controlled repository tools    │
                    │ read_file / write_file         │
                    │ protected-path policy          │
                    └───────────────┬───────────────┘
                                    │
                                    v
              ┌────────────────────────────────────────┐
              │ Deterministic completion gate           │
              │ 1. python -m pytest                     │
              │ 2. python -m ruff check .               │
              │ 3. optional: python -m mypy src         │
              └───────────────┬────────────────────────┘
                              │
              ┌───────────────┴─────────────────┐
              │                                 │
              v                                 v
┌───────────────────────────────┐   ┌─────────────────────────────────┐
│ All checks pass               │   │ One or more checks fail          │
│                               │   │                                 │
│ Return verified diff          │   │ Compact error report:             │
│ Status: verified_success      │   │ command + exit code + trace      │
└───────────────────────────────┘   └───────────────┬─────────────────┘
                                                     │
                                                     v
                                    ┌────────────────────────────────┐
                                    │ Retry within hard limits        │
                                    │ Max attempts: 3                 │
                                    │ Model/tool call budgets         │
                                    └───────────────┬────────────────┘
                                                    │
                              ┌─────────────────────┴─────────────────────┐
                              │                                           │
                              v                                           v
              ┌──────────────────────────┐             ┌──────────────────────────┐
              │ Checks pass on retry     │             │ Budget exhausted         │
              │ Return verified diff     │             │ Human-review escalation  │
              └──────────────────────────┘             │ diff + logs + failure    │
                                                       └──────────────────────────┘Code language: PHP (php)

Deterministic completion gate

It decides whether the agent is actually allowed to say “done”, based on hard, reproducible signals from the environment instead of the model’s own judgment. It is “deterministic” because, given the same repository state and the same test commands, it will always produce the same pass/fail result, with no randomness and no LLM in the loop.

What it does

In this codebase, the completion gate is implemented by the verify_completion function plus the logic that uses its output in repair_ci_failure. Together, they do three things:

Run external checks
When the agent thinks it has fixed the bug, the harness calls deterministic tools (run_pytest, run_ruff, optionally run_typecheck) against the target repository. Each tool returns a structured result: command, exit code, a boolean passed, and truncated stdout/stderr.
Decide pass/fail without the model
verify_completion iterates over the required checks, builds a checks dictionary, and sets passed = all(result["passed"] for result in checks.values()). This single boolean is the gate: if it is false, the agent is not allowed to complete; if it is true, the harness can safely treat the run as verified.
Drive the self-verification loop
repair_ci_failure uses the gate’s output to control the retry loop. After each agent run:
- If report["passed"] is true, it returns status="verified_success" along with the final git diff and checker results.
- If report["passed"] is false, it builds a compact failure report (command, exit code, stdout/stderr per failing checker), injects that back into the next agent prompt as specific feedback, and repeats, up to a fixed MAX_ATTEMPTS.
- If the loop exhausts its budget, it returns status="needs_human_review" and includes all the failure evidence.

This matches the idea of a verification gate described in harness-engineering patterns: the agent will say it is done, but the harness uses deterministic checks such as tests and build exit codes to decide whether “done” is actually true.vincentvandeth+1

Why it matters

The completion gate solves the most painful failure mode: false completeness, instead of trusting the LLM when it says “tests pass” or “bug fixed”, the gate:

runs pytest and ruff on the repo,
looks only at exit codes and passed booleans,
ignores the model’s narration if the suite is still red,
and pushes the agent back to work with precise error output.

That way, “done” becomes “all required checks succeeded”, not “the agent said it was done”.

Retry loop

This is the entire self-verification loop in practical form – the model acts, deterministic tools verify, the harness compresses the failure signal, and the next attempt is bounded. If the fix still fails after three tries, the system escalates instead of pretending the work is complete.

if __name__ == "__main__":
    result = repair_ci_failure(
        issue="""
        POST /users returns HTTP 500 when the request has a valid email
        but omits the optional display_name field. Fix the implementation.
        Do not change tests or API contracts.
        """,
        thread_id="ci-repair-demo-001",
    )

    print(result["status"])
    print(result["diff"])
Code language: PHP (php)

The Output

Do not publish a single vanity score, It publish the system delta on the same model, the same tasks, and the same budget. That is the cleanest way to show whether the harness did useful work.

Metric	Baseline agent	Verified harness	Why it matters
Verified task success rate	__%	__%	Counts only tasks that pass real checks
False completion rate	__%	__%	Measures fake “done” claims
Median attempts	__	__	Shows convergence quality
Median duration	__s	__s	Captures latency cost
Model calls per task	__	__	Proxy for spend and loopiness
Escalation rate	__%	__%	Shows controlled failures

In the output it shows a table to see if it’s within the cost limits (environment setup variable) and how much does costs each task.

Github Repo

You can download the codes on our Github repo, just download and follow the README steps. If need help you can always reach out our team on Discord 🤙

Download the codes

FAQ

What is a self-verification loop in LangChain?

It is an agent pattern where the model proposes a change, deterministic tools such as pytest validate the result, and middleware plus outer control logic bound retries and completion. LangChain middleware is designed to control agent execution, limits, and termination behavior at runtime.

Why use pytest instead of asking the LLM to judge its own patch?

pytest is deterministic and tied to the repository’s acceptance criteria, while an LLM judging its own work can still return confident but wrong completions. A self-verification loop works best when the checker is external to the model.

What should be limited first in LangChain?

Start with model calls and tool calls. LangChain provides built-in middleware for both, and these limits are the simplest way to stop runaway loops and uncontrolled cost.

Is pytest-benchmark enough for evaluation?

No. It is useful for repeated timing measurements, but you still need task-level evaluation such as verified task success rate, false completion rate, and escalation rate to understand whether the harness is actually improving outcomes.

Start your free 30-day trial at regolo.ai and deploy LLMs with complete privacy by design. If need help you can always reach out our team on Discord 🤙

👉 Talk with our Engineers or Start your 30 days free →

Discord – Share your thoughts
GitHub Repo – Code of blog articles ready to start
Follow Us on X @regolo_ai
Open discussion on our Subreddit Community

Built with ❤️ by the Regolo team. Questions? regolo.ai/contact or chat with us on Discord

LangChain Self-Verification Loop: Python + pytest Demo for a CI Repair Agent

Table of contents

What this agent fixes

In this design, the LLM is the maker, but `pytest`, linting, and repository policies are the checker. The model is never the final authority on completion. docs.

Why this use case matters for companies

Project structure

How the self-verification loop works

On the right side the complete working schema of this agent that you can see how it works for each step and edit as you need.

Deterministic completion gate

What it does

Why it matters

Retry loop

The Output

Github Repo

FAQ

What is a self-verification loop in LangChain?

Why use pytest instead of asking the LLM to judge its own patch?

What should be limited first in LangChain?

Is pytest-benchmark enough for evaluation?

Table of contents

What this agent fixes

In this design, the LLM is the maker, but pytest, linting, and repository policies are the checker. The model is never the final authority on completion. docs.

Why this use case matters for companies

Project structure

How the self-verification loop works

On the right side the complete working schema of this agent that you can see how it works for each step and edit as you need.

Deterministic completion gate

What it does

Why it matters

Retry loop

The Output

Github Repo

FAQ

What is a self-verification loop in LangChain?

Why use pytest instead of asking the LLM to judge its own patch?

What should be limited first in LangChain?

Is pytest-benchmark enough for evaluation?

In this design, the LLM is the maker, but `pytest`, linting, and repository policies are the checker. The model is never the final authority on completion. docs.