Tracing and evaluating open-weight models on Regolo with Langfuse: a practical guide

Most teams shipping LLM features in Europe hit the same blind spot: when a retrieval-augmented generation (RAG) pipeline returns a wrong answer, nobody can tell without structured traces whether the bug came from document retrieval, prompt formatting, or the model itself. At the same time, sending production prompts to US-hosted endpoints breaks strict data residency rules, and running reasoning models blindly in CI can multiply your evaluation bill by 3x because of hidden thinking tokens.

Pairing Langfuse (MIT-licensed open-source LLM observability and evaluation platform, self-hosted on your own EU servers or on its Frankfurt EU cloud region) with Regolo (European OpenAI-compatible inference with certified Zero Data Retention, meaning prompts and completions are never stored or reused) closes that loop in two lines of code. This guide walks through tracing and evaluating three open-weight models we serve (gemma4-31b, qwen3.8-27b, and glm5.2), or any model on your account via our companion Regolo Eval Studio, across offline datasets, online production traffic, and GitHub Actions CI gates.

Why LLM evaluation is harder than unit testing

Suppose a developer asks your documentation assistant, “What is the daily token limit on the Regolo Core plan?” A traditional unit test checks assert response == "20 million". An LLM assistant might answer “20M tokens per day”, “Up to 20,000,000 tokens daily”, or hallucinate “50 million tokens” because the retriever pulled the Boost plan paragraph instead of the Core plan paragraph.

Scoring only the final string hides whether the retriever fetched the wrong chunk or the generator misread the right chunk. Langfuse solves this by recording a structured trace for every request: the user question, the retrieved chunks from Qwen3-Embedding-8B and Qwen3-Reranker-4B, the prompt sent to the chat model, the output tokens, and the latency of each step.

The models in plain numbers: Pay-As-You-Go vs Core subscription

Before automating evaluations in CI or production, it helps to understand how the two billing options on our platform behave when a test suite runs hundreds of times a week. You can pay per token with Pay-As-You-Go (PAYG) when you only run occasional checks, or use our flat Core subscription when you want predictable monthly spend across all included chat models.

The table above highlights a practical trap that catches many engineering teams off guard: reasoning verbosity. When Artificial Analysis evaluated open-weight models, GLM 5.2 led the Intelligence Index, but it got there by thinking out loud in a long hidden scratchpad before writing its final reply. Because metered APIs bill every word of that internal monologue as output, running a talkative reasoning model on Pay-As-You-Go quickly multiplies your evaluation bill. Moving that same workload onto our flat Core plan removes the per-token penalty completely, since all output comes out of your included monthly pool.

By contrast, Gemma 4 31B and Qwen 3.8 27B go straight to the answer with minimal internal chatter. That makes them fast, predictable choices for everyday RAG generation and LLM-as-a-judge scoring, as shown in the cost comparison below:

Step 1: swap the base URL and trace every Regolo call in open-source Langfuse

Suppose you have a RAG endpoint that answers questions about Regolo documentation. Because Langfuse is MIT-licensed open source, you can self-host the Langfuse v4 server via Docker or Kubernetes on your own EU infrastructure (for example http://localhost:3000 or your internal domain) or use the managed Frankfurt EU cloud (https://cloud.langfuse.com). In both cases, you generate a project public_key and secret_key in the Langfuse settings so your Python service can authenticate when sending traces.

Instead of importing OpenAI from the standard openai package, import the drop-in OpenAI wrapper from langfuse.openai and point base_url at https://api.regolo.ai/v1:

import os
from langfuse.openai import OpenAI

# Self-hosted open-source Langfuse URL (or https://cloud.langfuse.com for EU Cloud)
os.environ["LANGFUSE_HOST"] = "https://langfuse.internal.example.eu"
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."

# Drop-in OpenAI client routed to Regolo EU datacenters
client = OpenAI(
    base_url="https://api.regolo.ai/v1",
    api_key=os.environ["REGOLO_API_KEY"],
)

response = client.chat.completions.create(
    model="gemma4-31b",
    messages=[
        {"role": "system", "content": "Answer using only the provided Regolo documentation."},
        {"role": "user", "content": "Context: Core Plan includes up to 600M tokens/month and 20M tokens/day.\nQuestion: What is the daily token limit on the Core plan?"},
    ],
    name="regolo-docs-rag-generation",
)
print(response.choices[0].message.content)
Code language: Python (python)

Every call to client.chat.completions.create or client.embeddings.create (with Qwen3-Embedding-8B on our API) now appears automatically as a structured observation in your Langfuse dashboard, complete with the prompt, the retrieved context, token usage, and latency.

Step 2: run offline evaluations on a 20-question documentation dataset

Before changing your system prompt or switching from gemma4-31b to qwen3.8-27b, test the change offline against a fixed dataset. Twenty real question-and-answer pairs from your docs are enough to catch regressions. In RAG evaluation, two metrics carry most of the signal:

• Faithfulness: does the answer stick strictly to facts present in the retrieved documentation chunks, without inventing numbers?
• Answer relevance: does the answer directly address the user’s question instead of dumping unrelated paragraphs?

In Langfuse, you combine a fast deterministic code evaluator (free, checks exact facts or keywords in under 2 seconds) with an LLM-as-a-judge evaluator (uses a model on our API to grade faithfulness from 0.0 to 1.0):

from langfuse import get_client, Evaluation
from langfuse.openai import OpenAI

langfuse = get_client()
regolo = OpenAI(base_url="https://api.regolo.ai/v1")

def rag_task(*, item, **kwargs):
    # item.input contains {"question": "...", "context": "..."}
    resp = regolo.chat.completions.create(
        model="gemma4-31b",
        messages=[
            {"role": "system", "content": "Answer strictly from the context."},
            {"role": "user", "content": f"Context: {item.input['context']}\nQuestion: {item.input['question']}"},
        ],
    )
    return resp.choices[0].message.content

def exact_fact_evaluator(*, input, output, expected_output, **kwargs):
    # Deterministic check: did the answer include the expected number or fact?
    passed = expected_output is not None and expected_output.lower() in output.lower()
    return Evaluation(name="contains_expected_fact", value=1.0 if passed else 0.0)

def faithfulness_judge(*, input, output, **kwargs):
    # LLM-as-a-judge running on Regolo (gemma4-31b)
    judge = regolo.chat.completions.create(
        model="gemma4-31b",
        messages=[
            {"role": "system", "content": "Reply with 1 if the answer uses only facts from the context, else 0."},
            {"role": "user", "content": f"Context: {input['context']}\nAnswer: {output}"},
        ],
    )
    score = 1.0 if "1" in judge.choices[0].message.content.strip()[:2] else 0.0
    return Evaluation(name="faithfulness", value=score)

dataset = langfuse.get_dataset("regolo-docs-qa")
result = dataset.run_experiment(
    name="gemma4-31b-rag-v1",
    task=rag_task,
    evaluators=[exact_fact_evaluator, faithfulness_judge],
)
print(result.format())
Code language: Python (python)

Running both the RAG generation task and the faithfulness judge on a concise model like gemma4-31b keeps the entire 20-question experiment fast enough to run on every pull request, while costing only a fraction of a cent per run on our Core plan.

Step 3: score live production traces with 5 percent sampling

Suppose users in production start asking about GPU hourly rates on Regolo (for example, “How much is an NVIDIA H200 per hour?”), a topic that was missing from your initial 20-question offline dataset. Offline tests will stay green while production answers fail.

To catch those live failures without doubling your inference usage, configure an online evaluator in Langfuse: run deterministic code evaluators (regex, JSON schema validation, PII checks) on 100 percent of live observations for free, and set your LLM-as-a-judge evaluator to sample 5 percent of production observations. Attach your online judges directly to the root observation of your RAG workflow (Langfuse sunsets legacy trace-level judges on November 16, 2026).

Step 4: block pull requests on score regressions in CI

When a teammate edits the RAG prompt or swaps the model slug in a pull request, your CI pipeline should block the merge if faithfulness drops below 0.85. Using langfuse/experiment-action in GitHub Actions (requires Langfuse Python SDK v4.6.0+ or JS SDK v5.3.0+), raise a RegressionError when the average score falls below your threshold, and the action will fail the workflow and post the comparison table as a PR comment.

Benchmark any model and export your Langfuse CI workflow locally (Regolo Eval Studio)

Langfuse’s dashboard is great once you have already chosen a model and wired it into your application. When you are still deciding which checkpoint to put into production, however, you usually want to test several models side by side on your own questions, see how much hidden reasoning chatter each one generates, and export a ready-made CI workflow without writing boilerplate scripts from scratch.

To make that effortless, we built Regolo Eval Studio (python3 eval_wizard/server.py), a local companion web app and CLI that works alongside Langfuse:

• Test any model on your account side by side: connects live to our /v1/models endpoint so you can select up to six open-weight models at once and compare their answers on the exact same prompts.
• Bring your own dataset and prompts: upload any .json, .jsonl, or .csv evaluation file, tweak the system prompt and judge rubric in the browser, or start with our built-in documentation benchmark.
• Spot hidden reasoning chatter and subscription break-even: separates visible answer tokens from internal thinking tokens and shows right away when our flat Core subscription becomes cheaper than Pay-As-You-Go for your testing volume.
• Export your Langfuse CI gate in one click: streams traces directly to your Langfuse project and generates a ready-to-commit eval_ci.py script and GitHub Actions workflow tailored to your winning model.

OPEN-SOURCE MULTI-MODEL FINOPS & LANGFUSE CI STUDIO · MIT LICENSE

Clone Regolo Eval Studio on GitHub

Includes the local web app (eval_wizard/server.py), multi-model comparison, custom dataset uploader (.json/.jsonl/.csv), FinOps break-even calculator, Langfuse CI exporter, and unit tests.

Open langfuse-eval-studio on GitHub →

Step 5: route each evaluation workload to the right Regolo model and plan

Use this decision map to match your workload to the right model and billing tier on our platform:

---

ZERO DATA RETENTION

Get engineering notes and model releases in your inbox.

Sovereign AI inference, OpenAI-compatible, EU-hosted. Benchmarks, new models (Mistral, Qwen, GLM), and cost insights. One email, no spam.

✓

Thank you for subscribing!

Please check your inbox to confirm your subscription.

Subscribe Free

I accept the Privacy Policy *

100% European infrastructure · Zero spam · Unsubscribe anytime.

Frequently asked questions

Why does open-source Langfuse still use API keys in the code example?

Even when you self-host the MIT-licensed Langfuse server on your own EU infrastructure, your Python or TypeScript application authenticates to your Langfuse instance with a project LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY generated inside your local dashboard. The keys separate environments (dev, staging, prod) and prevent unauthorized services from writing traces to your database.

Should I run evaluations on Regolo Pay-As-You-Go or on the Core subscription?

If you only run small offline experiments occasionally, Pay-As-You-Go has zero monthly commitment and costs less than a euro per thousand checks on concise models. As soon as you enable continuous CI gates, online production judges, or talkative reasoning models like GLM 5.2, our flat Core subscription shields your budget with a predictable monthly pool, as shown in Chart 1 above.

Why does GLM 5.2 cost 3.3x more per evaluation than Gemma 4 31B on PAYG?

Two factors combine on Pay-As-You-Go: GLM 5.2 is a much larger flagship model with higher per-token rates, and it spends thousands of hidden scratchpad tokens thinking through complex problems before answering. Gemma 4 31B is both lower-priced per token and far more concise, while our flat Core subscription lets you run GLM 5.2 without worrying about reasoning token spikes.

Can I run deterministic code evaluators before paying for an LLM-as-a-judge?

Yes. Langfuse runs Python and TypeScript code evaluators directly in the platform within a 2-second execution limit and without network access. Use code evaluators on 100 percent of traces to check JSON validity, required keywords, or tool-call arguments for free, then sample 5 percent of passing traces for LLM-as-a-judge scoring on our API.

How do I keep both inference and observability data inside the European Union?

Run your models on Regolo, which operates in certified European green datacenters under Zero Data Retention (prompts and outputs are never stored on disk or used for training), and point your Langfuse SDK at a self-hosted Langfuse v4 instance on your own EU servers or the Langfuse EU cloud region in Frankfurt.

---