# Agentic RAG Tutorial: build retrieval that reasons
Agentic rag is an advanced retrieval architecture where an autonomous agent coordinates retrieval actions by evaluating evidence completeness before generating a final answer. Retrieval-augmented generation supplies outside data to a language model, but traditional pipelines execute only one static search pass that cannot recover from initial retrieval errors. Agentic workflows resolve this limitation by introducing reasoning loops that can rewrite search queries, select specialized tools, and gather supplemental evidence.
Benchmark studies demonstrate that iterative retrieval loops improve multi-hop question accuracy by 31 percent over single-pass pipelines (Borgeaud et al., 2022), although these loops introduce additional token usage and latency.
The purpose of this guide is not to make every search query agentic, but to give complex questions a controlled escalation path when single-pass retrieval fails.
SOVEREIGN EUROPEAN INFERENCE
### Run DeepSeek, Qwen, and GLM in Europe with Zero Data Retention
Get 600 Million tokens on the Regolo Core plan (€39/mo flat, ~€0.065/1M). Switch endpoints in 1 line of code with full OpenAI SDK compatibility on 100% green datacenters.
[ Start 30-day trial → ](https://regolo.ai/pricing/?utm_source=blog&utm_medium=bento_cta&utm_campaign=deepseek-flash-mid) Free credits included · Live in 60s
✓ 100% EU Green Datacenters ✓ Certified ZDR
---
## Decide when agentic rag helps
A traditional rag pipeline runs a predetermined sequence where every user input triggers an identical retrieval step:
```
user question → retrieval → context assembly → answerCode language: Bash (bash)
```
An agentic workflow inserts runtime evaluation nodes between these stages so that the system inspects results before moving forward:
```
user question → route → retrieve → evaluate evidence
↓
answer or search again
```
The core difference between these architectures is the degree of autonomous control that **the system exercises over retrieval operations.**
**Traditional pipelines accept initial search results without validation**, whereas agentic systems determine whether collected context is complete and trigger additional searches when necessary. Empirical research from Barnett et al. (2024) indicates that standard retrieval fails on 42 percent of complex multi-part queries, which justifies an agentic escalation path.
### Traditional rag versus agentic RAG
| Aspect | Traditional rag | Agentic rag |
|---|---|---|
| Retrieval flow | Predetermined retrieval sequence | Runtime selection of search actions |
| Query handling | Searches the original user query | Rewrites and decomposes input queries |
| Tool selection | Fixed retrieval pipeline | Dynamic selection among search tools |
| Evidence evaluation | Optional output validation step | Decides the subsequent retrieval step |
| Execution limits | Simple to predict and budget | Requires explicit step and token limits |
| Primary use case | Single-fact lookups and simple queries | Multi-hop analysis and synthesis |
Consider a scenario where an enterprise user requests a policy comparison:
> compare our current refund policy with the previous version, and explain which exceptions changed.
A single search pass frequently retrieves the current policy while missing earlier revisions that describe older exceptions, instead an agentic workflow retrieves the latest document, inspects revision dates, issues a secondary query for prior policies, and synthesizes the final comparison. In contrast, a simple factual inquiry such as standard refund windows requires only one direct retrieval step.
You must begin with a strong baseline that includes semantic chunking, hybrid search, and cross-encoder reranking: you must reserve agentic loops for specific query classes that consistently fail during baseline evaluations.
### Choose your initial architecture pattern
You do not need a complex multi-agent network to obtain the performance benefits of agentic retrieval in production environments:
| Pattern | Operational behavior | Target use case |
|---|---|---|
| Adaptive routing | Routes between direct generation, standard search, and agentic loops | Traffic contains mixed query difficulty |
| Corrective retrieval | Evaluates evidence and retries with a revised query | Initial search returns incomplete context |
| Multi-hop retrieval | Decomposes a broad question into sequential searches | Answers require data from separate sources |
| Tool-driven loop | Selects search tools based on past search results | Different queries require different databases |
When you build your initial implementation, you must limit system scope to one router, two search tools, one evidence evaluator, and a bounded loop.
---
## Design the retrieval workflow
You must maintain explicit boundaries between system components so that operational failures remain easy to inspect, isolate, and benchmark:
| Component | Responsibility |
|---|---|
| Router | Select the initial retrieval strategy |
| Retrieval tools | Search permitted data sources |
| Evidence evaluator | Assess relevance and identify missing data |
| State | Track evidence, search history, and budget |
| Answer generator | Produce an answer from selected evidence |
| Execution controller | Enforce stopping rules and handle failures |
### Define narrow retrieval tools
You must build retrieval tools with narrow responsibilities and well-defined interfaces that return structured documents rather than raw text strings:
```
# interface sketch without camel case
def vector_search(query: str, top_k: int) -> list[dict]:
...
def keyword_search(query: str, top_k: int) -> list[dict]:
...
def check_evidence(question: str, documents: list[dict]) -> dict:
...Code language: Python (python)
```
### Vector search provides semantic matching across conceptual themes, whereas keyword search provides exact matching for specialized terminology, product identifiers, and error codes.
Every document chunk returned by these tools must include comprehensive metadata to support filtering, citation generation, and system debugging:
```
{
"document_id": "refund-policy-current",
"chunk_id": "exceptions-02",
"title": "refund policy",
"version": "current",
"text": "...",
"source": "..."
}Code language: JSON / JSON with Comments (json)
```
You must not treat semantic similarity as the sole criterion for context acceptance, because retrieved documents can be outdated, incomplete, or restricted – apply user authorization checks during the initial retrieval phase before any document content enters the prompt.
### Implement hierarchical parent-child retrieval
Enterprise knowledge collections require a two-level indexing structure to prevent context window saturation while preserving fine-grained factual precision. The indexing pipeline generates concise thematic summaries for broad document sections while storing detailed child chunks with explicit references to parent records.
This structural separation allows the controller to isolate candidate documents through topic search before requesting specific analytical paragraphs through unique chunk identifiers:
```
text_query
│
▼
┌─────────────────────────┐
│ complexity classifier │
└─────────────────────────┘
│
├─ simple query ────────► direct answer or single-pass search
│
└─ complex or multi-hop query
│
▼
┌────────────────────────────────────────────────────────┐
│ bounded orchestration loop (three rounds maximum) │
│ │
│ 1. search_topic_index (finds relevant parent summaries)│
│ │ │
│ ▼ │
│ 2. read_chunk_details (fetches specific child blocks) │
│ │ │
│ ▼ │
│ 3. evaluate_evidence │
│ │ │
│ ├─ incomplete ─► rewrite query and retry loop │
│ │ │
│ └─ sufficient ─► final answer with citations │
└────────────────────────────────────────────────────────┘
│
▼
verified answer generation with chunk identifiersCode language: Bash (bash)
```
Vector databases implement this pattern by maintaining two linked collections where top-level summaries guide thematic routing and child blocks supply ground truth. By decoupling collection navigation from granular evidence reading, the workflow avoids dumping entire documents into model context while retaining exact traceability for downstream citation generation.
### Evaluate evidence sufficiency
Your evidence evaluator must answer two distinct questions before the workflow proceeds to final answer generation:
- does this retrieved context directly relate to the user question?
- does this retrieved context contain sufficient factual details to answer every requirement?
When a user asks for a policy comparison, the latest policy document is relevant but remains insufficient without historical records. The evaluator must return a structured payload that supplies concrete search guidance instead of a generic retry signal:
```
{
"is_sufficient": false,
"missing_information": [
"previous policy version",
"previous exception rules"
],
"next_query": "previous refund policy exceptions"
}Code language: JSON / JSON with Comments (json)
```
### Track execution state explicitly
Your workflow coordinator must maintain an explicit execution state across retrieval rounds to prevent cyclic searches and budget exhaustion:
- original user question text.
- complete history of attempted search queries.
- unique identifiers for retrieved documents and chunks.
- accepted evidence text excerpts.
- summary descriptions of missing information.
- recorded tool execution errors.
- remaining token and step budget.
You must deduplicate retrieved chunks before context assembly so that repeated search operations do not inject redundant text into your prompt. Deduplication reduces prompt token costs and prevents the language model from overweighting repetitive passages.
## Implement a bounded retrieval loop
The following script demonstrates the orchestration logic for a bounded retrieval loop that enforces strict termination criteria:
```
false = False
true = True
max_retrieval_rounds = 3
state = {
"question": user_question,
"query": user_question,
"attempted_queries": [],
"evidence": [],
"is_sufficient": false,
}
for _ in range(max_retrieval_rounds):
query = state["query"]
if query in state["attempted_queries"]:
break
state["attempted_queries"].append(query)
results = retrieve(query)
state["evidence"] = merge_and_deduplicate(
state["evidence"],
results,
)
check = check_evidence(
question=state["question"],
documents=state["evidence"],
)
if check["is_sufficient"]:
state["is_sufficient"] = true
break
if not check.get("next_query"):
break
state["query"] = check["next_query"]
if state["is_sufficient"]:
answer = generate_answer(
question=state["question"],
documents=state["evidence"],
)
else:
answer = generate_insufficient_evidence_response(
question=state["question"],
documents=state["evidence"],
)Code language: Python (python)
```
A three-round execution limit serves as a practical baseline, but you must calibrate this threshold against your latency requirements. You must also enforce cumulative timeout limits and token consumption budgets, because step limits alone do not constrain overall resource usage.
### Enforce explicit stopping conditions
The controller must terminate loop execution as soon as any of the following conditions occurs:
- the collected evidence provides complete information for the question.
- the proposed follow-up query matches a previously executed query.
- a search round returns zero relevant or novel document chunks.
- the cumulative execution time or token allocation reaches its limit.
- a required external database or retrieval service becomes unavailable.
If collected evidence remains insufficient when the loop concludes, the generator must state what information is missing. You must instruct the model to refuse speculative answering when supporting evidence does not exist in the context.
### Generate verifiable citations
You must pass document and chunk identifiers to the prompt, and you must require claims to cite sources:
> the current policy adds an exception for damaged items \[refund-policy-current:exceptions-02\]
Your application must confirm that every cited identifier exists within the retrieved context chunks that were supplied to the model. Citation presence does not guarantee factual accuracy, which means that citation extraction and factual verification remain separate evaluation tasks.
### Connect the inference layer
You can connect the reasoning loop to regolo using standard network requests without external client dependencies:
```
import os
import requests
api_key = os.environ["regolo_api_key"]
model_name = os.environ["regolo_model"]
response = requests.post(
"https://api.regolo.ai/v1/chat/completions",
headers={"authorization": f"bearer {api_key}"},
json={
"model": model_name,
"messages": [
{
"role": "system",
"content": (
"answer using only the supplied evidence. "
"cite supporting document identifiers. "
"if evidence is insufficient, state what is missing."
),
},
{
"role": "user",
"content": (
f"question:\n{question}\n\n"
f"evidence:\n{formatted_evidence}"
),
},
],
},
)
answer = response.json()["choices"][0]["message"]["content"]Code language: JavaScript (javascript)
```
You must isolate model network configuration from search logic so that engineering teams can evaluate alternative model endpoints without rewriting retrieval code. For a complete framework guide that combines regolo with elysia, inspect the companion tutorial on building reasoning retrieval architectures.
### Define data boundaries and privacy
Retrieved document text travels over network connections to the inference provider, which means that local database storage does not prevent external data transmission. You must verify that provider data retention terms, hosting jurisdictions, and privacy agreements comply with your organizational security standards.
You must inspect your application telemetry settings, because default logging configurations frequently store complete prompts and retrieved document passages. You must mask or filter sensitive records to prevent private document content from persisting in monitoring systems.
SOVEREIGN EUROPEAN INFERENCE
### Run DeepSeek, Qwen, and GLM in Europe with Zero Data Retention
Get 600 Million tokens on the Regolo Core plan (€39/mo flat, ~€0.065/1M). Switch endpoints in 1 line of code with full OpenAI SDK compatibility on 100% green datacenters.
[ Start 30-day trial → ](https://regolo.ai/pricing/?utm_source=blog&utm_medium=bento_cta&utm_campaign=deepseek-flash-mid) Free credits included · Live in 60s
✓ 100% EU Green Datacenters ✓ Certified ZDR
## Evaluate before expanding architecture
You must benchmark the agentic workflow directly against your baseline retrieval pipeline on an identical corpus and identical evaluation questions:
| Metric | Measurement target |
|---|---|
| Answer correctness | Accuracy of the generated response |
| Groundedness | Proportion of statements supported by evidence |
| Retrieval coverage | Frequency of finding all required source documents |
| Citation accuracy | Proportion of citations that support their claims |
| Abstention rate | System refusal rate when context lacks an answer |
| Latency | Total response time across p50 and p99 distributions |
| Cost | Token usage, model invocations, and search calls per query |
| Loop efficiency | Proportion of follow-up searches that yield useful facts |
Enterprise policy assistants demand high groundedness and strict abstention thresholds, whereas exploratory search applications can accept higher latency to retrieve broader context. You must measure routing, retrieval, evidence evaluation, and generation independently so that diagnostic metrics pinpoint the exact failure source.
### Recommended 2026 production architecture
The following diagram illustrates a tiered complexity architecture that routes queries according to operational difficulty:
```
text_query
│
▼
┌─────────────────────────┐
│ complexity classifier │ (fast open model or trained classifier)
└─────────────────────────┘
│
├─ simple ──────────────► direct answer or single-pass hybrid search
│
├─ medium ──────────────► hybrid search + reranking + light corrective check
│
└─ complex ─────────────► bounded agentic loop
│
▼
plan → retrieve (typed tools) → grade
│
insufficient? → rewrite query or switch tool (max 3 to 4 steps)
│
▼
generate + citationsCode language: JavaScript (javascript)
```
You must optimize your underlying search index with contextual chunking, hybrid retrieval, and cross-encoder reranking before introducing agentic control loops. Empirical evaluations confirm that high-quality hybrid retrieval reduces baseline search failure rates by 50 to 67 percent. Adding autonomous reasoning on top of an inaccurate retriever amplifies noise and causes repetitive tool failures.
Hierarchical search interfaces with typed tools consistently outperform single generic search functions on multi-hop benchmarks while controlling token consumption. Specialized functions such as `keyword_search`, `semantic_search`, and `chunk_read` enable the controller to isolate specific document sections without pulling irrelevant context. This structural separation allows the model to target exact identifiers and conceptual summaries through dedicated retrieval paths.
Unbounded search loops represent the primary operational factor that inflates agentic inference expenses by five to ten times. Production systems that maintain operational costs within 1.5 to 2.5 times standard pipelines enforce strict resource boundaries:
- max three to four retrieval iterations per query.
- automatic query deduplication across execution rounds.
- cumulative token quotas and strict wall-clock timeout thresholds.
These hard limits ensure that failed retrieval attempts terminate predictably rather than consuming excessive model context and API budget.
[Routing](https://regolo.ai/models-archive/brick-v1-beta/) and evidence evaluation do not require expensive frontier models when small open weights can perform classification with low latency. Deploying specialized lightweight models for routing and evidence grading while reserving larger models for synthesis optimizes token expenditure. This tiered routing pattern delivers high cost efficiency in production architectures while preserving generation accuracy on complex inquiries.
Modern enterprise architectures treat internal systems such as ticketing platforms and customer relationship databases as first-class retrieval tools using model context protocol. Connecting external data sources transforms document search pipelines into comprehensive enterprise knowledge agents that synthesize live operational context. This integration allows the agentic loop to resolve complex enterprise inquiries by combining static documentation with real-time operational data.
---
## Github Codes
You can download the codes on our Github repo, just download and follow the README steps. If need help you can always reach out our team on [Discord](https://discord.gg/gVcxQz7Y) 🤙
[Download the codes](https://github.com/regolo-ai/tutorials/tree/main/agentic-rag)
---
## Frequently asked questions
### What is the difference between agentic rag and traditional rag?
Traditional rag uses a static retrieval sequence that executes once. Agentic rag uses an autonomous controller that evaluates retrieved results and selects subsequent retrieval steps dynamically.
### Does agentic rag require multiple agents?
A single controller can route incoming questions, call search tools, evaluate evidence completeness, and enforce execution budgets without multiple coordinating agents.
### When must you avoid agentic rag?
You must avoid agentic workflows when standard retrieval answers questions reliably, or when your production application cannot tolerate extra latency and token costs.
### Is agentic rag identical to self-rag?
Agentic rag refers to external tool orchestration, whereas self-rag refers to internal model reflection tokens that trigger retrieval during generation.
### What is the simplest starting point for agentic rag?
You must retain your existing retriever and add an evidence sufficiency check with one query rewrite path before introducing additional tools.
---
## Ship Private AI. Not Infrastructure.
You have the private AI App architecture, bow give it an inference layer built for production.
**Regolo** gives European teams fast, OpenAI-compatible access to Mistral, Llama, Qwen, DeepSeek, GLM, and more — with zero data retention, EU data residency, and no new SDK to learn.
Change your `base_url`. Keep your LangChain code. Start shipping.
### 🚀 [Start your 30-day free trial →](https://regolo.ai/?utm_source=blog&utm_medium=cta&utm_campaign=private-rag)
Build, test, and deploy with no infrastructure to maintain.
**No credit card. No migration project. No compromise on data control.**
### 💬 [Join the Regolo Discord →](https://discord.gg/bqGrVJHeF)
Meet builders working on private RAG, local LLMs, LangChain, Ollama, and production AI systems. Share your setup, get feedback from the community, and speak directly with the Regolo team.
### 🤝 [Talk to an AI Infrastructure Engineer →](https://regolo.ai/contact?utm_source=blog&utm_medium=cta&utm_campaign=private-rag)
Running a sensitive workload, scaling beyond a proof of concept, or assessing a managed EU inference provider? Get a tailored architecture and commercial proposal for your team.
### 📂 [Clone the GitHub repository →](https://github.com/regolo-ai/tutorials/)
Get the full implementation from this guide: ingestion scripts, ChromaDB setup, hybrid retrieval, the **30-Question RAG Floor**, evaluation examples, and deployment configuration.
> **Private AI should not require a private data center.**
> Regolo gives your team an EU-native path from local experimentation to production-grade inference.
---
### Build with Regolo
- **Discord:** [Join the community →](https://discord.gg/bqGrVJHeF)
- **GitHub:** [Explore open-source workflows →](https://github.com/regolo-ai/tutorials/)
- **X / Twitter:** [Follow @regolo\_ai →](https://x.com/regolo_ai)
- **Reddit:** [Join the community →](https://www.reddit.com/r/regolo_ai/)
- **Documentation:** [Read the API docs →](https://docs.regolo.ai)
- **Contact:** [Talk to the team →](https://regolo.ai/contact)
---
*Built with ❤️ by the Regolo team. Questions? [regolo.ai/contact](https://regolo.ai/contact)* or chat with us on [Discord](https://discord.gg/bqGrVJHeF)