Most LangChain tutorials default to OpenAI. This one doesn’t — and it is written for a specific reader: EU-based engineering teams in regulated industries (legal, healthcare, fintech, public sector, consulting) who cannot send internal documents to external AI APIs and need a retrieval system that runs entirely inside their own security perimeter. If your DPO just blocked a ChatGPT proof of concept, this guide is for you.
You will build a deployable internal document assistant using Mistral via Ollama, ChromaDB as the vector store, and LangChain as the orchestrator — then see where Regolo fits when you need managed, EU-hosted, Mistral-compatible inference without rewriting the application.
The core claim of this guide: a fully private RAG system — embeddings, vector storage, and generation all on your own server — requires no OpenAI dependency, no per-token API costs, and no data leaving your infrastructure. Everything below is the working proof.
Table of Contents
- Architecture
- Ollama vs Regolo vs OpenAI
- Install and run
- Ingest documents
- Build hybrid retrieval
- Generate grounded answers
- Evaluate with the 30-Question RAG Floor
- Frequently Asked Questions
Architecture
A private RAG system is an application that retrieves relevant internal documents before asking a language model to answer, keeping the source corpus, the embeddings, and the inference layer inside infrastructure you control.
The local version uses Ollama for generation and embeddings, ChromaDB for persistent vector storage, and LangChain as the orchestration layer connecting ingestion, retrieval, prompting, and responses.
PDFs, Markdown, DOCX
|
v
LangChain loaders
|
v
Chunking + metadata
|
+----------------------------------+
| |
v v
Chroma vector index BM25 keyword index
| |
+-------- Hybrid retrieval --------+
|
v
Mistral via Ollama
|
v
Answer with source citationsCode language: Bash (bash)Ollama runs open models locally and exposes an embedding endpoint that converts text into vectors suitable for semantic search. Chroma persists those vectors to disk through PersistentClient, although Chroma’s own documentation recommends a client-server deployment for serious production workloads.
Ollama vs Regolo vs OpenAI
The buying decision for this stack usually involves three people — the engineer who builds it, the CTO who signs it off, and the DPO who vetoes it. This table answers all three at once.
| Dimension | Ollama (self-hosted) | Regolo (managed EU) | OpenAI (managed US) |
|---|---|---|---|
| Data leaves your server? | No | Yes, to EU data centers | Yes, to US infrastructure |
| Data residency | Your infrastructure | European Union | United States |
| Zero data retention | By design (local) | Stated zero-retention policy | Subject to API data policies |
| Mistral models | Yes, via ollama pull | Yes, including Mistral Small | No |
| Ops burden | You manage everything | None | None |
| Cost model | Fixed (hardware/VM) | Per-token, EU billing | Per-token, US billing |
| LangChain integration | langchain-ollama | OpenAI-compatible base URL | Native |
| Best for | Strict sovereignty requirements | Managed EU compliance without ops | Teams with no privacy constraints |
Quotable takeaway: self-hosted Ollama is the only option where document content never touches a third-party processor; Regolo is the lowest-friction managed alternative when EU residency and zero retention satisfy your compliance model; OpenAI is the wrong default for regulated European teams, not because of quality, but because of jurisdiction.
Install and run
Create an isolated Python environment, install the LangChain integration packages, and pull one chat model plus one dedicated embedding model.
python -m venv .venv
source .venv/bin/activate
pip install -U \
langchain \
langchain-community \
langchain-chroma \
langchain-ollama \
langchain-text-splitters \
chromadb \
pypdf \
rank_bm25
ollama pull mistral
ollama pull embeddinggemmaCode language: Bash (bash)LangChain’s Ollama integration ships in the langchain-ollama package, and Ollama recommends embedding-focused models such as embeddinggemma, qwen3-embedding, and all-minilm for retrieval workloads.
Use the same embedding model for indexing and querying. Changing it later makes existing Chroma vectors incompatible with newly generated query vectors — a silent failure that looks like “search got worse” and costs hours to diagnose.
private-rag/
├── data/ # Team documents
├── chroma/ # Persistent vector database
├── ingest.py
├── app.py
├── eval.py
└── requirements.txtCode language: Bash (bash)Ingest documents
Start with RecursiveCharacterTextSplitter, LangChain’s recommended default for preserving related text while producing manageable chunks. A practical baseline for policy documents, handbooks, and technical notes is 800 characters per chunk with 120 characters of overlap. Adjust only after measuring retrieval quality, never before.
# ingest.py
from pathlib import Path
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain_chroma import Chroma
from langchain_ollama import OllamaEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
DATA_DIR = Path("data")
PERSIST_DIR = "chroma"
COLLECTION = "team_knowledge"
def load_documents():
documents = []
for path in DATA_DIR.rglob("*"):
if path.suffix.lower() == ".pdf":
documents.extend(PyPDFLoader(str(path)).load())
elif path.suffix.lower() in {".md", ".txt"}:
documents.extend(TextLoader(str(path), encoding="utf-8").load())
for document in documents:
document.metadata["source_file"] = document.metadata.get("source", "unknown")
return documents
documents = load_documents()
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=120,
add_start_index=True,
)
chunks = splitter.split_documents(documents)
embeddings = OllamaEmbeddings(model="embeddinggemma")
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
collection_name=COLLECTION,
persist_directory=PERSIST_DIR,
)
print(f"Indexed {len(chunks)} chunks.")
Code language: Python (python)Chunking is not cosmetic preprocessing. It controls whether a retrieved passage contains enough context to answer a question without burying the useful sentence among unrelated material. I would start with character-based chunks for ordinary business documents, then move to Markdown-header or code-aware splitting when the source structure is reliable.
Chunking rules that work
| Document type | Starting strategy | Why it helps |
|---|---|---|
| Employee handbook | 700–900 characters, 100–150 overlap | A policy clause often needs its surrounding exception or definition |
| API documentation | Split by headings, then 500–800 characters | Endpoint descriptions, parameters, and examples stay together |
| Source code | Language-aware splitter | Function and class boundaries beat arbitrary character counts |
| Contracts | 400–650 characters, 100 overlap | Smaller chunks reduce dilution from unrelated clauses |
| Meeting notes | Split by dates/headings, then 600–900 characters | Metadata enables filtering by team, date, project |
The part most teams skip: version your ingestion configuration alongside the source files. Chunk size, overlap, embedding model, and parser choice all change retrieval behavior. Well, not exactly “change” — they change which evidence the model receives at all, which is worse, because the failure is invisible in the output.
Build hybrid retrieval
Here is the failure mode that sends teams back to keyword search in frustration: vector search alone fails on internal error codes, ticket IDs, product SKUs, and uncommon acronyms, because embeddings treat “ERR-4502” and “ERR-4501” as near-identical strings. Pure semantic retrieval is structurally weak on exact-match tokens.
Hybrid retrieval fixes this by combining both signals — and in internal benchmarks across support and engineering corpora, adding BM25 to vector retrieval typically lifts Recall@5 by double-digit percentages on identifier-heavy queries.
Chroma’s LangChain integration supports hybrid ranking with reciprocal-rank fusion. For a portable starter, pair Chroma similarity search with LangChain’s BM25 retriever through an ensemble retriever.
# app.py
from langchain_chroma import Chroma
from langchain_community.retrievers import BM25Retriever
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain.retrievers import EnsembleRetriever
PERSIST_DIR = "chroma"
COLLECTION = "team_knowledge"
embeddings = OllamaEmbeddings(model="embeddinggemma")
vectorstore = Chroma(
collection_name=COLLECTION,
persist_directory=PERSIST_DIR,
embedding_function=embeddings,
)
vector_retriever = vectorstore.as_retriever(
search_type="similarity",
search_kwargs={"k": 5},
)
all_records = vectorstore.get(include=["documents", "metadatas"])
all_chunks = [
Document(page_content=text, metadata=metadata)
for text, metadata in zip(
all_records["documents"],
all_records["metadatas"],
)
]
bm25_retriever = BM25Retriever.from_documents(all_chunks)
bm25_retriever.k = 5
hybrid_retriever = EnsembleRetriever(
retrievers=[vector_retriever, bm25_retriever],
weights=[0.7, 0.3],
)
Code language: Python (python)A useful starting ratio is 70 percent semantic, 30 percent keyword. I would increase BM25’s weight for support manuals, release notes, legal clauses, and engineering docs — anywhere exact strings decide whether a result is correct.
For small and medium collections, rebuilding the BM25 index at startup is acceptable. At larger scale, run Chroma in server mode, keep retrieval indexes behind an authenticated service, and stop rebuilding sparse structures on every request.
Generate grounded answers
The model must receive only retrieved evidence plus an explicit refusal rule. Otherwise a local Mistral will answer confidently from general training data — the single most dangerous failure mode in an internal Q&A tool, because confident wrong answers erode trust faster than no answers.
llm = ChatOllama(model="mistral", temperature=0)
prompt = ChatPromptTemplate.from_template("""
You answer questions using only the supplied internal context.
If the answer is missing from the context, say:
"I cannot find that in the indexed team documents."
For every factual claim, cite the relevant source file.
Do not invent policies, dates, owners, or technical details.
Question:
{question}
Context:
{context}
""")
def ask(question: str):
documents = hybrid_retriever.invoke(question)
context = "\n\n---\n\n".join(
f"Source: {doc.metadata.get('source_file', 'unknown')}\n{doc.page_content}"
for doc in documents
)
response = (prompt | llm).invoke(
{"question": question, "context": context}
)
return {
"answer": response.content,
"sources": [
doc.metadata.get("source_file", "unknown")
for doc in documents
],
}
if __name__ == "__main__":
result = ask("What is our process for approving production access?")
print(result["answer"])
print("Sources:", result["sources"])
Code language: Python (python)Set temperature to zero for internal Q&A. Factual consistency matters more than verbal variety. Return source names, page numbers when available, and chunk identifiers in the interface — visible evidence makes bad retrieval diagnosable instead of mysterious.
Evaluate with the 30-Question RAG Floor
Do not call a RAG system successful because a few demo questions sound convincing. Use a minimum evaluation gate I call the 30-Question RAG Floor: thirty questions drawn from real team requests, split into 10 direct lookups, 10 multi-step questions, 5 deliberately unanswerable questions, and 5 questions containing exact identifiers (ticket IDs, error codes, policy numbers).
For every item, store the expected source, the expected answer points, and whether the system must refuse. Measure retrieval separately from generation — a perfect model cannot recover evidence the retriever never provided.
| Metric | Minimum target | What failure means |
|---|---|---|
| Retrieval Recall@5 | 85% or higher | The correct chunk never reaches the model |
| Citation accuracy | 95% or higher | Answers cite documents that don’t support the claim |
| Grounded answer score | 4 out of 5 | Missing details or unsupported additions |
| Refusal accuracy | 90% or higher | The system invents answers to unanswerable questions |
| Median response time | Under 8 seconds | Too slow for normal team usage |
Thirty carefully labeled questions reveal more operational truth than three hundred synthetic prompts nobody in the organization would ask. When Recall@5 is weak, inspect chunking and metadata first. When recall is healthy but answers are wrong,
Frequently Asked Questions
What is a private RAG system?
A private RAG (Retrieval-Augmented Generation) system is a document Q&A application where every component — document storage, embeddings, vector database, and language model — runs on infrastructure you control. No document content, prompt, or embedding is sent to an external API. This guide builds one using LangChain (orchestration), ChromaDB (vector store), Ollama (local inference), and Mistral (language model).
Can I use LangChain without OpenAI?
Yes. LangChain is model-agnostic and supports dozens of providers. The langchain-ollama package connects LangChain to locally running open-source models such as Mistral, Llama, and Qwen, providing both chat generation and embeddings with no OpenAI API key, no per-token costs, and no data leaving your server.
How do I build a RAG system with ChromaDB and LangChain?
The pipeline has five steps: (1) load documents with LangChain loaders, (2) split them with RecursiveCharacterTextSplitter at roughly 800 characters with 120 overlap, (3) embed chunks with a local model like embeddinggemma via Ollama, (4) store vectors in a persistent ChromaDB collection, (5) retrieve relevant chunks at query time and pass them to a local Mistral model with a citation-required prompt. Full working code for each step is in this guide.
Is Mistral good enough for document Q&A compared to GPT-4?
For grounded Q&A over retrieved context, Mistral performs well when the prompt enforces strict grounding, temperature is set to zero, and retrieval quality is high. Most perceived “model quality” gaps in RAG systems are actually retrieval failures — the model never received the right chunk. Measure retrieval Recall@5 before blaming the generator.
Why does vector search fail on error codes and ticket IDs?
Embedding models are trained for semantic similarity, not token exactness, so they treat “ERR-4502” and “ERR-4501” as nearly identical. BM25 keyword retrieval scores exact token matches. Hybrid retrieval — combining dense vector search with BM25 through reciprocal-rank fusion or an ensemble retriever — is the standard fix, typically lifting Recall@5 by double digits on identifier-heavy corporate corpora.
What is hybrid search in RAG?
Hybrid search combines semantic vector retrieval (which captures meaning) with keyword retrieval such as BM25 (which captures exact tokens), then merges the ranked results. A practical starting configuration is 70% semantic weight and 30% keyword weight, increasing the keyword share for documentation rich in error codes, SKUs, clause numbers, and internal acronyms.
What chunk size should I use for RAG?
Start with 800 characters and 120 characters of overlap for general business documents. Use smaller chunks (400–650 characters) for contracts to avoid diluting clauses, heading-aware splitting for API documentation, and language-aware splitters for source code. Chunk size is a retrieval-quality decision: measure Recall@5 before and after any change.
Is ChromaDB production-ready?
ChromaDB is production-capable when deployed correctly. The PersistentClient is intended for local development; for production, run Chroma in client-server mode behind authentication, with encrypted backups and tested recovery procedures. For multi-team deployments, enforce document-level authorization with metadata filters at retrieval time.
Where does my data go when using Regolo instead of Ollama?
With Ollama, prompts and documents never leave your server. With Regolo, inference requests are processed in European data centers under a stated zero-data-retention policy — no training on your data, EU jurisdiction, OpenAI-compatible API. Self-hosted Ollama offers maximum sovereignty; Regolo offers managed EU compliance without the operational burden.
Is a local LLM RAG system GDPR-compliant?
A self-hosted stack (Ollama + ChromaDB on your own servers) keeps personal data within your processing boundary, which simplifies GDPR compliance significantly — but compliance also requires access controls, audit logging, retention policies, and documented data flows. Local inference is necessary, not sufficient.
How do I evaluate a RAG system before deploying it to my team?
Use the 30-Question RAG Floor: 30 real team questions — 10 direct lookups, 10 multi-step, 5 deliberately unanswerable, 5 containing exact identifiers — scored against minimum thresholds: Retrieval Recall@5 ≥ 85%, citation accuracy ≥ 95%, refusal accuracy ≥ 90%, median response time under 8 seconds. Measure retrieval and generation separately so failures point to the right layer.
What does a private RAG system cost to run?
Self-hosted costs are fixed: a VM or workstation with 16–32 GB RAM (or a GPU for faster inference) runs Mistral 7B-class models and ChromaDB comfortably. There are no per-token API fees. The real cost is operational time — expect a few hours per month for index updates, model upgrades, and monitoring.
Can I switch from Ollama to a managed provider later without rewriting code?
Yes, if you architect for it. LangChain separates retrieval from generation, so swapping the LLM means changing the model client — with an OpenAI-compatible provider like Regolo, only the base URL and API key change. Your ChromaDB collection, chunking config, retrieval logic, and evaluation suite remain untouched.
Ship Private AI. Not Infrastructure.
You have the private RAG architecture. Now give it an inference layer built for production.
Regolo gives European teams fast, OpenAI-compatible access to Mistral, Llama, Qwen, DeepSeek, GLM, and more — with zero data retention, EU data residency, and no new SDK to learn.
Change your base_url. Keep your LangChain code. Start shipping.
🚀 Start your 30-day free trial →
Build, test, and deploy with no infrastructure to maintain.
No credit card. No migration project. No compromise on data control.
💬 Join the Regolo Discord →
Meet builders working on private RAG, local LLMs, LangChain, Ollama, and production AI systems. Share your setup, get feedback from the community, and speak directly with the Regolo team.
🤝 Talk to an AI Infrastructure Engineer →
Running a sensitive workload, scaling beyond a proof of concept, or assessing a managed EU inference provider? Get a tailored architecture and commercial proposal for your team.
📂 Clone the GitHub repository →
Get the full implementation from this guide: ingestion scripts, ChromaDB setup, hybrid retrieval, the 30-Question RAG Floor, evaluation examples, and deployment configuration.
Private AI should not require a private data center.
Regolo gives your team an EU-native path from local experimentation to production-grade inference.
Build with Regolo
- Discord: Join the community →
- GitHub: Explore open-source workflows →
- X / Twitter: Follow @regolo_ai →
- Reddit: Join the community →
- Documentation: Read the API docs →
- Contact: Talk to the team →
Built with ❤️ by the Regolo team. Questions? regolo.ai/contact or chat with us on Discord