EmbeddingGemma 2 vs Qwen3-Embedding (0.6B & 8B): Code Retrieval, Matryoshka Compression, and Reranking Benchmarks

When choosing an open-weight embedding model in October 2026, Qwen3-Embedding-8B (served on Regolo’s API at €0.001 per request) and Qwen3-Embedding-0.6B still lead pure text and multilingual retrieval, scoring 70.58 and 64.33 on MTEB Multilingual v2 against 61.36 for Google’s new EmbeddingGemma 2.

Where EmbeddingGemma 2 changes the equation is code retrieval (jumping from 68.76 to 78.68 nDCG@10 on MTEB Code v1), native visual-document and audio embeddings, and a modular 270M-parameter text tower that occupies 542 MB of unified memory on Apple Silicon versus 5.8 GB for Qwen3-Embedding-0.6B at a 32K context window.

In our independent benchmark across 60 code, Italian/English compliance, and architecture passages, pairing local EmbeddingGemma 2 with Qwen3-Reranker-4B via Regolo’s /v1/rerank API lifted overall Recall@5 from 92.22% to 95.56% and code nDCG@10 from 96.51% to 98.42%.

What Google’s EmbeddingGemma 2 model card leaves out

In the original EmbeddingGemma paper (arXiv:2509.20354), Google benchmarked its 308M model directly against Qwen3-Embedding-0.6B, BGE-M3, and jina-embeddings-v3. In the October 2026 EmbeddingGemma 2 model card, external open-weight baselines were removed from the primary benchmark table, leaving only a generational comparison against EmbeddingGemma 1.

Putting the official model-card disclosures from Google and Alibaba side by side explains why:

Three engineering facts stand out from the published numbers:

• Multilingual text retrieval barely moved between EmbeddingGemma 1 and 2: a gain of 0.21 points on MTEB Multilingual v2 (61.15 to 61.36) leaves EmbeddingGemma 2 nearly 3 points behind Qwen3-Embedding-0.6B (64.33) and 9.22 points behind Qwen3-Embedding-8B (70.58).
• Code retrieval jumped by +9.92 points: climbing from 68.76 to 78.68 on MTEB Code v1 brings a 270M text model within 2 points of the 8-billion-parameter Qwen3-Embedding-8B (80.68).
• 740M is the multimodal bundle, not the text footprint: comparing “740M vs 600M” misstates how EmbeddingGemma 2 runs in production. If your pipeline only embeds text and code, you load the 270m-bf16-text checkpoint and skip the vision and audio towers entirely.

Our independent benchmark: Apple M1 Ultra (64 GB) vs Regolo API

Public leaderboards average dozens of datasets with different lengths and domains. To test how these models behave on workloads European engineering teams actually deploy, we ran a controlled benchmark across 60 documents and 30 graded-relevance queries split into three domains:

1. Code retrieval (20 implementations, 10 queries): Python and SQL implementations (Redis Lua token buckets, async full-jitter backoff, AST function chunkers, pgvector HNSW queries, HMAC-SHA256 constant-time verification) paired with near-miss distractors (fixed-interval retries, sliding-window text splitters, unverified JWT decoders).
2. Italian & English compliance & cross-lingual retrieval (20 passages, 10 queries): GDPR Article 28 Zero Data Retention, EU AI Act Article 50 watermarking, DORA ICT exit strategies, Italian Garante Privacy 21-day email metadata limits, FSE 2.0 hospital pseudonymization, and US CLOUD Act exposure. Queries mix Italian questions targeting English documents and English questions targeting Italian regulatory text.
3. Technical & API architecture docs (20 passages, 10 queries): KV cache mechanics, Matryoshka truncation, bi-encoder vs cross-encoder attention, 1-bit binary quantization failure modes, and MoE active-parameter memory footprints.

Hardware, runtime trap, and instruction prefixes

We executed local models on an Apple M1 Ultra with 64 GB of unified memory (macOS arm64) and remote models on Regolo’s production API (https://api.regolo.ai/v1). During setup we hit two operational traps that affect anyone reproducing this stack:

• Ollama version gate for EmbeddingGemma 2: because EmbeddingGemma 2 is built on the Gemma 4 architecture, Ollama 0.21.x rejects the manifest with HTTP 412: The model you are attempting to pull requires a newer version of Ollama. You must run Ollama 0.40.4 or newer, which executes embeddinggemma-2:270m-bf16-text via the Apple mlx backend.
• The 32K KV-cache memory surprise on Qwen3-Embedding-0.6B: while qwen3-embedding:0.6b is a 639 MB download on disk, Ollama allocates its full 32,768-token context window in unified memory at startup, consuming 5.8 GB of GPU memory. By contrast, embeddinggemma-2:270m-bf16-text (8,192-token context) consumes 542 MB, more than 10x less runtime memory.

Each model was evaluated both with its official asymmetric instruction template and with raw unprefixed queries to measure how much accuracy you lose when you drop instruction prefixes:

• EmbeddingGemma 2 & 1 queries: task: code retrieval | query: {query} (code slice) and task: search result | query: {query} (multilingual and docs slices). Documents use title: {title} | text: {text}.
• Qwen3-Embedding (0.6B & 8B) queries: Instruct: {domain_task}\nQuery:{query}. Documents are encoded without instruction tags, as required by the Qwen3 technical report.

Full-dimension retrieval results and latency

What happens if you forget instruction prefixes? When we stripped the Instruct: and task: prefixes and sent raw queries, overall Recall@5 dropped across every model: from 92.22% to 87.22% (-5.0 points) on EmbeddingGemma 2, from 95.00% to 92.22% (-2.78 points) on Qwen3-Embedding-0.6B, and from 98.33% to 93.89% (-4.44 points) on Qwen3-Embedding-8B. On technical documentation queries specifically, omitting the Qwen3 instruction prefix reduced Qwen3-Embedding-8B nDCG@10 from 97.27% to 94.03%.

Matryoshka (MRL) compression curve: 4096d down to 128d

Both EmbeddingGemma 2 and the Qwen3-Embedding family are trained with Matryoshka Representation Learning (MRL), which concentrates the most discriminative variance in the leading vector coordinates. Instead of storing all 4,096 or 768 floats per chunk, you can slice the prefix of the vector and L2-renormalize it, or let Regolo’s API do it server-side by passing "dimensions": 512 or "dimensions": 256 to POST /v1/embeddings.

We tested both client-side MRL slicing across all four checkpoints and native server-side dimensions pass-through on Regolo’s Qwen3-Embedding-8B endpoint. The server-side vectors returned by Regolo at dimensions=512 and dimensions=256 matched our client-side L2-normalized slices to the second decimal place:

The MRL sweet spot for production vector databases

• Qwen3-Embedding-8B truncated to 512d or 256d cuts raw vector RAM by 8x to 16x (from 152.59 GB down to 19.07 GB or 9.54 GB per 10 million chunks) while keeping nDCG@10 above 97.5% and Recall@5 at 95.00%, which still outperforms native 768d EmbeddingGemma 2 (92.22% Recall@5).
• On EmbeddingGemma 2, 512d is the safe floor for single-stage retrieval: while nDCG@10 stays flat at 256d (95.95%), top-5 recall drops from 92.22% (at 768d) to 87.22% (at 256d) and 85.56% (at 128d) as secondary relevant documents fall below the top-5 cutoff. Furthermore, Google’s model card notes that if you use EmbeddingGemma 2 for visual document embeddings (MMEB VisDoc), truncating to 128d degrades NDCG@5 from 67.84 to 62.66 (-5.18 points).

Here is how to request native MRL compression directly from Regolo’s /v1/embeddings endpoint without shipping 4,096 floats over the wire:

import os
import requests

REGOLO_API_KEY = os.environ["REGOLO_API_KEY"]

def embed_qwen3_mrl(texts: list[str], dimensions: int = 512) -> list[list[float]]:
    response = requests.post(
        "https://api.regolo.ai/v1/embeddings",
        headers={
            "Authorization": f"Bearer {REGOLO_API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen3-Embedding-8B",
            "input": texts,
            "dimensions": dimensions,  # Passed directly to the model; returns 512d vectors
        },
        timeout=30,
    )
    response.raise_for_status()
    data = sorted(response.json()["data"], key=lambda item: item["index"])
    return [item["embedding"] for item in data]Code language: Python (python)

Two-stage retrieval: local compact embedding + Qwen3-Reranker-4B on Regolo

What if your architecture requires a fast, 542 MB local embedder like EmbeddingGemma 2 for first-stage candidate generation, but you cannot afford its 6-point Recall@5 gap against Qwen3-Embedding-8B? We tested a two-stage pipeline: retrieve the top 15 candidates locally with EmbeddingGemma 2 (or Qwen3-Embedding-0.6B), then rerank those 15 passages using Qwen3-Reranker-4B on Regolo’s /v1/rerank endpoint (€0.01 per query, up to 14,000 tokens per call).

Critical production finding: the Qwen3-Reranker chat-template trap

During our benchmark, we discovered a subtle but decisive behavior in how Qwen3-Reranker-4B scores candidates. Because Qwen3-Reranker is trained as a causal yes/no judge with a <think>...</think> block, sending bare strings or only <Instruct> / <Query> / <Document> tags without the Qwen ChatML wrapper caused nDCG@10 to drop to 77.98% (as the model’s token probabilities were evaluated without the system judge header and empty thinking block).

When we wrapped the query and documents in the full Qwen3-Reranker ChatML template, top-1 relevance scores snapped from noisy 0.57 probabilities to calibrated 0.9995 vs 0.0000 separations, and overall nDCG@10 jumped by +18.86 points (from 77.98% to 96.84%):

Adding Qwen3-Reranker-4B on top of local EmbeddingGemma 2 brings code retrieval Recall@5 to 100.0% and multilingual Italian/English nDCG@10 to 98.46% (surpassing even standalone Qwen3-Embedding-8B on the multilingual slice). Below is the exact Python helper with the ChatML wrapper required for calibrated scores on https://api.regolo.ai/v1/rerank:

import os
import requests

REGOLO_API_KEY = os.environ["REGOLO_API_KEY"]

def rerank_candidates_qwen3(
    query: str,
    candidate_passages: list[str],
    task: str = "Given a technical or compliance search query, retrieve relevant passages that answer the query",
    top_n: int = 5,
) -> list[dict]:
    formatted_query = (
        "<|im_start|>system\n"
        "Judge whether the Document meets the requirements based on the Query and the Instruct provided. "
        "Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n"
        f"<|im_start|>user\n<Instruct>: {task}\n<Query>: {query}\n"
    )
    formatted_docs = [
        f"<Document>: {doc}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
        for doc in candidate_passages
    ]
    response = requests.post(
        "https://api.regolo.ai/v1/rerank",
        headers={
            "Authorization": f"Bearer {REGOLO_API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen3-Reranker-4B",
            "query": formatted_query,
            "documents": formatted_docs,
            "top_n": top_n,
        },
        timeout=30,
    )
    response.raise_for_status()
    return response.json()["results"]Code language: JavaScript (javascript)

Visual document embeddings (VisDoc) vs DeepSeek-OCR-2 + Qwen3-Embedding-8B

One genuine architectural addition in EmbeddingGemma 2 is its 440M text+vision checkpoint (embeddinggemma-2:440m-bf16-text-vision), which scores 67.84 NDCG@5 on MMEB v2 VisDoc by embedding page screenshots directly into a 768-dimensional vector without running OCR first. Should you replace your OCR pipeline with direct page-image embeddings?

• When direct visual document embeddings work: slide decks, infographics, and visual catalogs where layout, charts, or diagrams carry the primary meaning and you want a quick coarse filter over page thumbnails.
• When single-vector page embeddings fail: multi-column contracts, balance sheets, medical records, and engineering datasheets. Compressing an entire A4 page of 800 words and dense tables into a single 768-float vector loses exact clause numbers, SKUs, and table cell alignment. That loss grows at 128d, where VisDoc NDCG@5 drops to 62.66.

For regulated or dense PDF workflows, our benchmarks in DeepSeek-OCR-2 vs GLM-OCR vs PaddleOCR and our Agentic RAG tutorial show a more reliable pattern: convert pages to structured Markdown with deepseek-ocr-2 on Regolo (€0.02/request, preserving table and layout tags), chunk by section heading, and embed the resulting text with Qwen3-Embedding-8B at dimensions=512.

Decision matrix: which retrieval stack should you run?

git clone https://github.com/regolo-ai/tutorials.git
cd tutorials/embeddinggemma-2-vs-qwen3-embedding
cp .env.example .env && python3 benchmark.py

Frequently asked questions