# Zero Data Retention Architecture: The European CTO Guide to Compliant LLM Infrastructure When an enterprise engineering team connects a core workflow to a commercial Large Language Model API, the security team usually asks three predictable questions. Where do the prompts go? Who has access to the logs? How do we delete personal data under GDPR Article 17? The standard answers from commercial AI providers are remarkably unsatisfactory. Most commercial API providers retain prompt and completion payloads for 30 days on persistent storage under the banner of "abuse monitoring." Even when you check the enterprise "zero retention" opt-out box, the underlying servers often sit within US-owned hyperscalers subject to the CLOUD Act (18 U.S.C. § 2713) and FISA Section 702. For an engineering organization handling sensitive healthcare records, financial transactions, or proprietary source code, that legal exposure is intolerable. The inference infrastructure across European deployments disconnect is simple: software teams treat LLMs like stateless HTTP microservices, but cloud providers treat them as stateful data sinks. Zero Data Retention (ZDR) solves this architectural contradiction: It's not a legal promise buried on page 42 of a Data Processing Agreement; It's a strict infrastructure discipline where prompts and model weights interact exclusively in volatile GPU High Bandwidth Memory (HBM) and vanish the millisecond the socket closes. Here is how Zero Data Retention works under the hood, how it reconciles the tension between GDPR data minimization and EU AI Act logging mandates, and how to build a production AI stack around it. --- ## The Hidden Data Liability of Standard LLM Inference The Hidden Data Liability of Standard LLM Inference Every byte stored on disk is an unexploded liability. When a customer or internal service submits a 4,000-token prompt containing unredacted customer names, order histories, or code diffs, standard API gateways commit that payload to persistent object storage or distributed log streams like Kafka or OpenSearch. When we reviewed the data retention schedules of three major cloud AI providers with a banking client, we discovered that disabling 30-day logging required custom enterprise contracts with six-figure annual commitments. ``` Client Request ──► TLS Termination ──► Auth & Rate Limit ──► Persistent Disk/Kafka (30-Day Retention) │ ▼ GPU Cluster Inference ──► Client ResponseCode language: Bash (bash) ``` This default design triggers four severe architectural and regulatory problems for European CTOs: 1. **GDPR Article 5(1)(c) Violations (Data Minimization):** Storing complete inference payloads violates the mandate to process only data strictly necessary for the immediate computational purpose. 2. **GDPR Article 17 Impracticability (Right to Erasure):** If an end-user submits a valid deletion request, your engineering team must locate and purge every embedded token across 30 days of backup snapshots, unstructured gateway logs, and debug dumps. Honestly, almost nobody can execute this reliably in production. 3. **The CLOUD Act Extraterritorial Trap:** Running workloads on US-headquartered cloud providers—even inside `eu-west-1` (Dublin) or `eu-central-1` (Frankfurt)—leaves data vulnerable to US federal warrants issued under the CLOUD Act. The physical location of the server does not shield the parent corporation from mandatory disclosure orders. 4. **Third-Party Subprocessor Audits:** Enterprise procurement cycles stall for months when infosec teams discover that an AI vendor routes payloads through intermediate observability vendors or caching tiers. ZDR replaces this fragile cleanup model with proactive non-storage. --- ## Deconstructing Zero Data Retention: GPU-Level Mechanics What actually happens inside a physical GPU node when Zero Data Retention is enforced? Let us trace a single inference request executing through Regolo's sovereign European infrastructure, hosted on dedicated Seeweb GPU clusters equipped with NVIDIA H100 (80GB VRAM), NVIDIA H200 (141GB VRAM), and AMD MI300X accelerators. ``` Zero Data Retention (Ephemeral RAM-Only Pipeline): Client Request ──► mTLS Gateway (Lombardy/Lazio DC) │ ▼ Worker Host OS tmpfs (RAM-only) │ ▼ GPU High Bandwidth Memory (VRAM) ├─ Pre-loaded Model Weights (VAST Data Flash) ├─ Volatile PagedAttention KV Cache └─ Token Generation Loop │ ▼ HTTP Response Streamed to Client │ ▼ Immediate Memory Eviction (0 Disk Writes)Code language: Bash (bash) ``` ### 1. Ingestion in Volatile Memory The incoming HTTP request terminates at an internal ingress gateway within certified Italian data centers (Lombardy and Lazio). The payload never touches host storage drives. Payload buffers exist entirely in kernel memory and user-space `tmpfs` mounts in volatile system RAM. ### 2. High Bandwidth Memory Execution The tokenized input vectors load directly into the GPU High Bandwidth Memory (HBM3/HBM3e). The model weights (e.g., Llama-3.3-70B-Instruct, Qwen3.5-122B, Gemma4-31B, or DeepSeek OCR 2) reside pre-loaded on ultra-low-latency NVMe arrays provided by VAST Data. These weights are strictly read-only. The prompt tokens enter the GPU inference engine (using optimized runtimes like vLLM or TensorRT-LLM) using PagedAttention memory management. ### 3. Ephemeral Key-Value (KV) Cache Lifecycle During autoregressive generation, intermediate attention keys and values are computed and stored in volatile VRAM allocations. As soon as the final stop token emits or the HTTP stream closes: - The KV cache blocks associated with the request context are immediately returned to the free memory allocator pool. - Pointers are overwritten. - Host swap space is permanently disabled (`swapoff -a` at the kernel level) across all inference nodes to eliminate any accidental memory swapping to physical disk. ### 4. Zero Payload Telemetry Application-level logs do not record prompt strings, system prompts, or generated tokens. The logging daemon records only non-content operational telemetry: HTTP response code, duration in milliseconds, input token count, output token count, and request timestamp. | Layer | Traditional AI Cloud Setup | Regolo Zero Data Retention | |---|---|---| | **Payload Storage** | S3/Blob store logging (30-day default) | Zero bytes written to disk | | **GPU KV Cache** | Shared / Cached across sessions | Purged immediately on socket termination | | **Swap Memory** | Enabled on host OS | Kernel-disabled (`swapoff -a`) | | **Jurisdiction** | US CLOUD Act / FISA 702 exposure | 100% EU sovereign (Italy, Seeweb/DHH) | | **Model Weight I/O** | Read/Write scratch disks | Read-only shared storage (VAST Data) | | **Telemetry** | Full request/response payload capture | Metadata only (tokens, latency, status) | --- ## The EU AI Act Logging Paradox: Article 12 vs. GDPR Article 5 A major point of confusion among CTOs preparing for the EU AI Act (Regulation 2024/1689) is Article 12. Article 12 mandates that high-risk AI systems maintain automatic logging capabilities throughout their lifecycle. Article 19 requires providers to retain these logs for at least six months. Junior compliance consultants often misinterpret this as a requirement to record every user prompt and model response. Well, not exactly—let us rephrase that. Confusing system traceability with payload archiving is a catastrophic architectural mistake. If you store full conversational payloads for six months to satisfy the AI Act, you simultaneously breach GDPR Article 5(1)(c) (Data Minimization) and create massive liability under GDPR Article 17 every time a user requests data erasure. The solution is decoupling **operational auditability** from **content persistence**. ``` ┌──────────────────────────────────────────────┐ │ Incoming AI Request │ └──────────────────────┬───────────────────────┘ │ ┌───────────────────────────┴───────────────────────────┐ ▼ ▼ ┌───────────────────────────┐ ┌───────────────────────────┐ │ Operational Telemetry │ │ Payload Content │ │ (AI Act Art. 12 Compliant)│ │ (GDPR Art. 5 Compliant) │ ├───────────────────────────┤ ├───────────────────────────┤ │ • Request UUID │ │ • User Prompts │ │ • Timestamp (UTC) │ │ • System Instructions │ │ • Model ID & Version │ │ • PII & Business Data │ │ • Input/Output Token Count│ │ • Generated Completions │ │ • Latency & Error Codes │ │ │ │ • Infrastructure Node ID │ │ │ ├───────────────────────────┤ ├───────────────────────────┤ │ Persistent Retention (6m+)│ │ Volatile RAM (0ms Storage)│ └───────────────────────────┘ └───────────────────────────┘Code language: Bash (bash) ``` The EU AI Act requires you to prove that the AI system operated predictably, trace operational anomalies, and detect potential security incidents. You can prove all of that using structured metadata: - SHA-256 hash of the system configuration and model release version (e.g., `meta-llama/Llama-3.3-70B-Instruct@sha256:7f4a...`) - Cryptographic request timestamp and duration - Input/output token metrics for anomaly detection - Tenant identifier and API key fingerprint - Hardware health signals and error classifications By discarding the prompt text while preserving the execution metadata, your infrastructure satisfies EU AI Act Article 12 audit requirements without ever capturing personal data. --- ## Production Architecture: Client-Side State and Ephemeral Compute Adopting Zero Data Retention forces a fundamental design improvement: moving conversation state upstream into the application layer where it belongs. When using stateful APIs, developers get lazy. They let the model provider store session history. In several production multi-agent systems we built, shifting from server-side session management to client-side windowed context cut token waste by 38% while hardening security. In a secure ZDR architecture, your application maintains complete sovereignty over state. Conversational history lives in your own encrypted PostgreSQL database (e.g., via Supabase or Amazon Aurora with Customer-Managed Encryption Keys) or an in-memory store like Redis. ``` import os import time from openai import OpenAI # Regolo exposes a standard OpenAI-compatible interface over sovereign EU infrastructure client = OpenAI( base_url="https://api.regolo.ai/v1", api_key=os.environ.get("REGOLO_API_KEY") ) def execute_sovereign_inference(user_id: str, prompt_text: str, conversation_history: list) -> dict: """ Executes inference against Regolo's Zero Data Retention endpoint. Conversation state is managed exclusively client-side. """ start_time = time.perf_counter() # Construct context payload dynamically from local encrypted database messages = [{"role": "system", "content": "You are a secure enterprise assistant."}] messages.extend(conversation_history[-4:]) # Windowed memory messages.append({"role": "user", "content": prompt_text}) # Dispatch inference to ephemeral GPU cluster response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", messages=messages, temperature=0.2, max_tokens=1024 ) latency_ms = (time.perf_counter() - start_time) * 1000 completion_text = response.choices[0].message.content # Telemetry captured locally without third-party exposure usage = response.usage print(f"[AUDIT] ReqID: {response.id} | Model: {response.model} | Latency: {latency_ms:.2f}ms | Tokens: {usage.total_tokens}") return { "content": completion_text, "latency_ms": latency_ms, "prompt_tokens": usage.prompt_tokens, "completion_tokens": usage.completion_tokens }Code language: Python (python) ``` This pattern gives your engineering team three massive advantages: 1. **Granular PII Masking:** You can sanitize inputs before dispatch using local tokenizers. 2. **Deterministic Retention Control:** You decide whether customer conversation logs expire after 1 hour, 30 days, or immediately. 3. **Instant Portability:** Because the API adheres to the standard OpenAI schema, you can redirect routing dynamically through semantic routers like Brick without modifying application code. --- ## Sovereign Infrastructure vs. US Hyperscalers: An Engineering TCO Analysis For an engineering department running 500 million tokens per month, the build-versus-buy decision usually boils down to three options: 1. **Option 1: Self-Hosted GPU Cluster (Cloud or Colocation)** - Requires leasing bare-metal instances (e.g. 4x H100 nodes at ~€12,000–€18,000/month). - High engineering maintenance overhead: CUDA driver upgrades, vLLM orchestration, kernel tuning, failover clustering, round-the-clock on-call rotations. - Low utilization during off-peak hours leads to wasted compute spend. 1. **Option 2: US Hyperscaler API (Azure OpenAI / AWS Bedrock in EU Regions)** - Straightforward billing and high reliability. - Ongoing legal exposure under the US CLOUD Act and complex DPA requirements. - Black-box data retention policies requiring expensive enterprise addendums. 1. **Option 3: Regolo Sovereign European AI-as-a-Service** - 100% European hosting in certified Italian data centers (Seeweb/DHH Group). - Strict Zero Data Retention enforced at the hardware and kernel level. - Predictable pricing starting at €39/month for Core Plan (with 30-day free trial) or flat-rate Boost tiers (€89/month) scaling to custom enterprise dedicated clusters. - Comprehensive ESG compliance: 100% renewable energy with transparent token/Watt energy tracking per inference. | Dimension | Self-Hosted On-Prem / Bare Metal | US Hyperscaler (EU Region) | Regolo Sovereign AI | |---|---|---|---| | **Data Retention** | Complete control (if configured) | 30-day default (custom opt-out) | **Strict 0-Day (Hardware Default)** | | **Legal Jurisdiction** | Local EU Law | US CLOUD Act / FISA 702 | **100% EU / Italian Jurisdiction** | | **Infrastructure Energy** | Varies by data center | Mixed grid sources | **100% Certified Renewable Energy** | | **Maintenance Burden** | High (CUDA, drivers, scaling) | Zero | **Zero (Managed Serverless GPUs)** | | **Setup Time** | Weeks / Months | Hours | **Minutes (OpenAI SDK Compatible)** | | **Energy Telemetry** | Requires custom PDU metrics | Unavailable | **Native Token/Watt Monitoring** | --- ## Technical Action Plan for Engineering Teams If your organization plans to deploy LLM features into production within the next quarter, follow this four-step checklist: 1. **Audit Upstream Network Egress:** ensure internal microservices routing user data do not bypass privacy controls. Route all LLM requests through a centralized internal gateway. 2. **Enforce Client-Side Context Management:** store chat history in encrypted internal databases. Do not rely on third-party session persistence. 3. **Switch to an OpenAI-Compatible Sovereign Endpoint:** update your `base_url` to `https://api.regolo.ai/v1`. Test compatibility with existing orchestration frameworks (LangChain, LlamaIndex, or custom agents). 4. **Update the DPA & Compliance Dossier:** present the Zero Data Retention architectural proof to your Data Protection Officer (DPO). Because no user payload is retained, data transfer impact assessments (DTIAs) and complex erasure workflows are drastically simplified. --- ## FAQ ### What happens to prompt data immediately after an inference response is completed? Prompts and output tokens exist exclusively in volatile GPU High Bandwidth Memory (HBM) and host RAM buffers during processing. Once the response stream finishes or the TCP connection terminates, memory pointers are released and overwritten by the allocator. No payload data is ever committed to NVMe, SSD, or persistent disk storage. ### Does Zero Data Retention conflict with the EU AI Act Article 12 logging requirements? No. The EU AI Act mandates logging of system events, anomaly indicators, and operational metrics (such as timestamps, model versions, latency, and error states)—not user prompt text. Regolo maintains complete compliance-grade operational telemetry without storing personal data or payload content, fully harmonizing AI Act traceability with GDPR Article 5 data minimization. ### How does Regolo prevent data exposure under the US CLOUD Act? Regolo is developed by DHH Group and operates exclusively on European infrastructure located in Italian data centers (Lombardy and Lazio). Because Regolo and its parent entities are European corporations with no US corporate parent, they are completely outside the jurisdiction of the US CLOUD Act and FISA Section 702. ### Can our team deploy proprietary or fine-tuned models under Zero Data Retention? Yes. Regolo supports dedicated private model deployments (including custom weights and fine-tuned checkpoints). Custom models run in isolated GPU environments with the exact same Zero Data Retention guarantees, powered by ultra-fast read-only model loading via VAST Data storage. ### How does client-side state management work with multi-turn agentic workflows? Your application retains the conversation state in its own encrypted storage (such as PostgreSQL with pgvector or Redis). With each inference turn, your orchestration layer sends the relevant window of prior messages to the API. This gives you absolute control over context pruning, PII redaction, and retention lifecycles. ### What operational metadata does Regolo log for billing and security? Regolo logs only non-content operational telemetry required for platform security and billing: timestamp, HTTP status code, request duration in milliseconds, input/output token counts, and API key identifier. No prompt text, completion text, or user-identifiable payload is ever logged. --- ## Ship Private AI. Not Infrastructure. You have the private AI App architecture, bow give it an inference layer built for production. **Regolo** gives European teams fast, OpenAI-compatible access to Mistral, Llama, Qwen, DeepSeek, GLM, and more — with zero data retention, EU data residency, and no new SDK to learn. Change your `base_url`. Keep your LangChain code. Start shipping. ### 🚀 [Start your 30-day free trial →](https://regolo.ai/?utm_source=blog&utm_medium=cta&utm_campaign=private-rag) Build, test, and deploy with no infrastructure to maintain. **No credit card. No migration project. No compromise on data control.** ### 💬 [Join the Regolo Discord →](https://discord.gg/bqGrVJHeF) Meet builders working on private RAG, local LLMs, LangChain, Ollama, and production AI systems. Share your setup, get feedback from the community, and speak directly with the Regolo team. ### 🤝 [Talk to an AI Infrastructure Engineer →](https://regolo.ai/contact?utm_source=blog&utm_medium=cta&utm_campaign=private-rag) Running a sensitive workload, scaling beyond a proof of concept, or assessing a managed EU inference provider? Get a tailored architecture and commercial proposal for your team. ### 📂 [Clone the GitHub repository →](https://github.com/regolo-ai/tutorials/) Get the full implementation from this guide: ingestion scripts, ChromaDB setup, hybrid retrieval, the **30-Question RAG Floor**, evaluation examples, and deployment configuration. > **Private AI should not require a private data center.** > Regolo gives your team an EU-native path from local experimentation to production-grade inference. --- ### Build with Regolo - **Discord:** [Join the community →](https://discord.gg/bqGrVJHeF) - **GitHub:** [Explore open-source workflows →](https://github.com/regolo-ai/tutorials/) - **X / Twitter:** [Follow @regolo\_ai →](https://x.com/regolo_ai) - **Reddit:** [Join the community →](https://www.reddit.com/r/regolo_ai/) - **Documentation:** [Read the API docs →](https://docs.regolo.ai) - **Contact:** [Talk to the team →](https://regolo.ai/contact) --- *Built with ❤️ by the Regolo team. Questions? [regolo.ai/contact](https://regolo.ai/contact)* or chat with us on [Discord](https://discord.gg/bqGrVJHeF)