AutoRound Quantization Guide: Local GPU to Production GGUF on Hugging Face

Your RTX 4090 just became a 70B-model machine. Intel’s AutoRound makes it possible — and this guide shows exactly how to quantize, export to GGUF, and publish custom models without vendor lock-in.

Why AutoRound Changes the Game
Prerequisites & Hardware Reality Check
AutoRound-Quantized Models on Hugging Face (Curated)
Installation: CPU, CUDA, Intel GPU, or Gaudi
Core Concepts: Bits, Group Size, and Recipes
CLI Quantization Walkthrough
API Quantization for Pipelines
Exporting GGUF — The Format That Runs Everywhere
Pushing Custom GGUF Models to Hugging Face Hub
Deploy on a Private API endpoint on Regolo
Inference: vLLM, Transformers, llama.cpp
Advanced: Mixed-Bit, VLM, and Calibration Datasets
Troubleshooting & Known Issues
FAQ

Why AutoRound Changes the Game

Most quantization tools force a choice: speed or accuracy, AutoRound uses sign-gradient descent to fine-tune weight rounding and min-max ranges in ~200 iterations — no extra inference overhead, no calibration data engineering unless you want it. The result: INT2 models retaining 97.9% accuracy on DeepSeek-R1 (200 GB compressed to ~50 GB) .

Key differentiators that matter for production teams:

Feature	Why It Matters
2–4 bit support with mixed-bit per layer	Assign 2 bits to insensitive layers, 4 bits to attention — reclaim VRAM surgically
Native GGUF, GPTQ, AWQ, AutoRound export	One quantization run, four deployment targets
vLLM & Transformers integration (v4.51.3+)	Drop-in inference, no custom kernels to maintain
~10 min for 7B on single GPU	CI/CD friendly — quantize on every release branch
VLM support via `auto-round-mllm`	Quantize Qwen2-VL, Gemma-3, LLaVA in one command

The library hits a sweet spot: research-grade accuracy with engineering-grade ergonomics. Well, not exactly — there are rough edges (random seeding on some models, ChatGLM v1 unsupported). But the trajectory is clear.

Prerequisites & Hardware Reality Check

Before you pip install, audit your iron:

Model Size	FP16 VRAM	4-bit VRAM	2-bit VRAM	Minimum GPU
7B	14 GB	3.5 GB	~2 GB	RTX 3060 12GB
32B	64 GB	16 GB	8 GB	RTX 4090 / A6000
70B	140 GB	35 GB	17.5 GB	2× RTX 4090 or A100 80GB
235B (MoE)	470 GB	~120 GB	~60 GB	4× A100 / H100

Rule of thumb: 4-bit needs ~0.5 bytes/param + KV cache overhead. Group size 128 is the default sweet spot; drop to 64/32 if OOM strikes. AutoRound’s --low_gpu_mem_usage flag trades ~30% slower tuning for ~20 GB VRAM savings — use it on 24 GB cards .

Calibration data: defaults to NeelNanda/pile-10k (10k samples, 2048 tokens). For code models, swap in mbpp. For chat models, enable --dataset ...:apply_chat_template. Custom JSON/JSONL works too — see step-by-step docs.

AutoRound-Quantized Models on Hugging Face (Curated)

Model	Base	Quantization	Size	Author
Gemma-4-Gemsicle-31B-W4A16-AutoRound	Gemma-4-Gemsicle-31B	W4A16 (4-bit weights, FP16 activations)	~19 GB	bf mags
Qwen3.5-27B-heretic-v2-autoround-w4a16	Qwen3.5-27B-heretic-v2	W4A16	~16 GB	groxaxo
autoround-quantized-4bit	Unspecified (likely 7B–14B)	4-bit symmetric	~4–8 GB	jjeccles
Qwen3.6-27B-int4-AutoRound	Qwen3.6-27B	INT4 (group_size=128, sym)	~16 GB	Lorbus
Qwen3.6-27B-INT8-AutoRound	Qwen3.6-27B	INT8 (group_size=128)	~30 GB	Minachist
MiniMax-M2.7-REAP-172B-A10B-AutoRound-W4A16	MiniMax-M2.7-REAP-172B-A10B (MoE)	W4A16	~95 GB	MJPansa

W4A16 = 4-bit weight quantization with 16-bit (FP16/BF16) activations — the sweet spot for vLLM/TGI serving. INT4/INT8 = symmetric integer quantization (group_size=128 default) — runs on llama.cpp, Ollama, Transformers.

Missing a model? Search autoround on HF Hub — 200+ public repos and counting

Installation: CPU, CUDA, Intel GPU, or Gaudi

# CUDA / Intel GPU / CPU (most common)
pip install auto-round

# Gaudi / HPU
pip install auto-round-lib

# Bleeding edge (GGUF export, INT2 extended algorithm)
pip install git+https://github.com/intel/auto-round.git@mainCode language: Bash (bash)

Optional but recommended for CPU inference speed:

pip install intel-extension-for-pytorch  # Intel CPU
# or
pip install intel-extension-for-transformersCode language: Bash (bash)

Verify:

auto-round -h
# Should show: --format options including 'gguf:q4_k_m', 'gguf:q2_k_s', etc.Code language: Bash (bash)

Core Concepts: Bits, Group Size, and Recipes

Bits (2, 3, 4) — Lower = smaller, faster, less accurate. AutoRound shines at 3-bit and 4-bit; 2-bit needs --enable_alg_ext (experimental) or auto-round-best recipe .

Group size (32, 64, 128, 256) — How many weights share one scale/zero-point. Smaller = more precise, larger model file. Default 128 works for most LLMs; VLMs often need 32.

Symmetry (--sym / --asym) — Symmetric quantization (no zero-point) is faster on CUDA kernels. Asymmetric can recover accuracy at 2-bit.

Three recipes (pick one):

Recipe	Command Prefix	Speed	Accuracy	Use Case
`auto-round`	`auto-round`	1×	Balanced	Default for 4-bit
`auto-round-best`	`auto-round-best`	3× slower	Best (esp. 2-bit)	Production 2-bit, quality-critical
`auto-round-light`	`auto-round-light`	2–3× faster	Slight drop at 4-bit, larger at 2-bit	Rapid iteration, CI smoke tests

RTN mode (--iters 0) — Round-to-nearest, zero calibration. Near-instant, usable for 4-bit+; avoid for 2–3 bit unless you’re desperate .

CLI Quantization Walkthrough

Basic 4-bit GGUF Export (Recommended Starting Point)

auto-round \
  --model Qwen/Qwen3-8B \
  --bits 4 \
  --group_size 128 \
  --format "gguf:q4_k_m" \
  --output_dir ./qwen3-8b-q4km-autoroundCode language: Bash (bash)

--format "gguf:q4_k_m" — GGUF with k-quant Q4_K_M (recommended default for llama.cpp/Ollama)
Output: ./qwen3-8b-q4km-autoround/ containing .gguf file(s) + config + tokenizer

Multi-Format Export (CI/CD Friendly)

auto-round \
  --model Qwen/Qwen3-8B \
  --bits 4 \
  --group_size 128 \
  --format "auto_round,auto_gptq,auto_awq,gguf:q4_k_m" \
  --output_dir ./qwen3-8b-multiCode language: Bash (bash)

One run, four artifacts. vLLM reads auto_round; llama.cpp reads gguf; GPTQ/AWQ for legacy pipelines.

2-Bit Production Recipe (Quality-Critical)

auto-round-best \
  --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
  --bits 2 \
  --group_size 128 \
  --low_gpu_mem_usage \
  --format "gguf:q2_k_s" \
  --output_dir ./deepseek-r1-2bitCode language: Bash (bash)

--low_gpu_mem_usage saves ~20 GB VRAM at ~30% time cost. Expect 4–6 hours on A100 80GB for 70B .

VLM Quantization (Experimental)

auto-round-mllm \
  --model Qwen/Qwen2-VL-2B-Instruct \
  --bits 4 \
  --group_size 32 \
  --format "gguf:q4_k_m" \
  --output_dir ./qwen2vl-2b-q4kmCode language: Bash (bash)

VLMs quantize text tower by default. Add --quant_nontext_module for vision encoder (limited support) .

Evaluation During Quantization

auto-round \
  --model Qwen/Qwen3-8B \
  --bits 4 \
  --group_size 128 \
  --format "auto_round,gguf:q4_k_m" \
  --tasks mmlu,gsm8k \
  --eval_bs 16Code language: Bash (bash)

Runs lm-eval-harness on the last format exported. Saves manual eval step.

API Quantization for Pipelines

When quantization lives inside a training/export pipeline, use the Python API:

from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound

model_name = "Qwen/Qwen3-8B"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Balanced recipe (default)
autoround = AutoRound(
    model,
    tokenizer,
    bits=4,
    group_size=128,
    sym=True,
    iters=200,           # default
    nsamples=128,        # calibration samples
    seqlen=2048,
    batch_size=8,
    dataset="NeelNanda/pile-10k",  # or custom list/path
)

# Quantize and save multiple formats
output_dir = "./qwen3-8b-api"
autoround.quantize_and_save(output_dir, format=["auto_round", "gguf:q4_k_m", "auto_gptq"])Code language: Python (python)

VLM API (requires processor):

from auto_round import AutoRoundMLLM
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer

model_name = "Qwen/Qwen2-VL-2B-Instruct"
model = Qwen2VLForConditionalGeneration.from_pretrained(model_name, trust_remote_code=True, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)

autoround = AutoRoundMLLM(model, tokenizer, processor, bits=4, group_size=32, sym=True)
autoround.quantize()
autoround.save_quantized("./qwen2vl-api", format="gguf:q4_k_m", inplace=True)Code language: Python (python)

Pro tip: Set low_gpu_mem_usage=True in AutoRound(...) constructor for 20 GB VRAM savings. Set enable_quanted_input=True (default) for block-wise quantized-input tuning — it’s the secret sauce .

Exporting GGUF — The Format That Runs Everywhere

GGUF is the lingua franca of local inference. llama.cpp, Ollama, LM Studio, kobold.cpp, Jan — they all speak GGUF. AutoRound writes it natively since v0.6.0 (July 2025) .

GGUF Quantization Types (Pick One)

Specifier	Description	Size (7B)	Quality	Speed
`gguf:q2_k_s`	2-bit k-quant small	~2.8 GB	Low	Fastest
`gguf:q3_k_m`	3-bit k-quant medium	~3.8 GB	Good	Fast
`gguf:q4_k_m`	4-bit k-quant medium (default rec)	~4.7 GB	Excellent	Fast
`gguf:q5_k_m`	5-bit k-quant medium	~5.6 GB	Near-FP16	Medium
`gguf:q6_k`	6-bit k-quant	~6.5 GB	Indistinguishable	Slower
`gguf:q8_0`	8-bit legacy	~8.2 GB	Overkill	Slowest

Recommendation: start with q4_k_m. Drop to q3_k_m if VRAM tight. q2_k_s only for extreme compression — expect coherence loss on complex reasoning.

Command Template

auto-round \
  --model <HF_MODEL_ID_OR_LOCAL_PATH> \
  --bits 4 \
  --group_size 128 \
  --format "gguf:q4_k_m" \
  --output_dir ./my-model-ggufCode language: Bash (bash)

Output structure:

my-model-gguf/
├── model-q4_k_m.gguf          # The GGUF file (single file, ~4.7 GB for 7B)
├── config.json                # AutoRound quantization config
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
└── README.md                  # Auto-generated with command usedCode language: Bash (bash)

Single-file guarantee: GGUF bundles tokenizer, config, and tensors. Copy model-q4_k_m.gguf to any llama.cpp-compatible runner — it just works.

Pushing Custom GGUF Models to Hugging Face Hub

Hugging Face Hub treats GGUF as a first-class citizen — but files >10 MB require Git LFS. AutoRound outputs are always >10 MB. Here’s the battle-tested workflow:

1. Install Git LFS & HF CLI

# Linux
sudo apt install git-lfs
# macOS
brew install git-lfs
# Windows: download from git-lfs.github.com

git lfs installCode language: Bash (bash)

# HF CLI (optional but handy)
pip install -U huggingface_hubCode language: Bash (bash)

2. Create Repository (Web or CLI)

# Via CLI (requires HF token with write scope)
hf repo create your-username/your-model-gguf --type model --private
# or public: remove --privateCode language: Bash (bash)

3. Clone & Enable Large Files

git clone https://huggingface.co/your-username/your-model-gguf
cd your-model-gguf

# CRITICAL: enables >5 GB file support
hf lfs-enable-largefiles ./Code language: PHP (php)

4. Track GGUF Files with LFS

git lfs track "*.gguf"
git add .gitattributes
git commit -m "Enable Git LFS for GGUF models"Code language: Bash (bash)

5. Copy Artifacts & Push

# Copy your quantized GGUF + tokenizer files
cp /path/to/autoround/output/model-q4_k_m.gguf ./
cp /path/to/autoround/output/tokenizer*.json ./
cp /path/to/autoround/output/config.json ./
cp /path/to/autoround/output/README.md ./

# Optional: create a clean model card
cat > README.md << 'EOF'
---
license: apache-2.0
tags:
  - gguf
  - autoround
  - quantization
  - qwen3
---

# Qwen3-8B-Q4_K_M-AutoRound

Quantized with Intel AutoRound (v0.6.0) using:
```bash
auto-round --model Qwen/Qwen3-8B --bits 4 --group_size 128 --format "gguf:q4_k_m"
```

**Quantization config:** 4-bit, group_size=128, symmetric, k-quant Q4_K_M
**Calibration:** NeelNanda/pile-10k (default)
**Accuracy:** ~99% of FP16 on MMLU (internal eval)

## Usage

### llama.cpp / Ollama
```bash
ollama create qwen3-8b-q4km -f ./Modelfile
# Modelfile: FROM ./model-q4_k_m.gguf
```

### Transformers (CPU/GPU)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("your-username/your-model-gguf", gguf_file="model-q4_k_m.gguf")
```

EOFCode language: Bash (bash)

git add .
git commit -m "Add Q4_K_M GGUF quantized model"
git pushCode language: Bash (bash)

Watch for: Uploading LFS objects: 100% (X/X), Y GB — that’s your model uploading.

6. Verify & Tag

<code>git lfs ls-files
<em># Should show: model-q4_k_m.gguf</em>

<em># Tag for versioning</em>
git tag -a v1.0-q4km -m "AutoRound Q4_K_M quantization"
git push origin v1.0-q4km</code>Code language: HTML, XML (xml)

Your model is now at https://huggingface.co/your-username/your-model-gguf — discoverable, downloadable, runnable via ollama pull hf.co/your-username/your-model-gguf.

Deploy on Private API endpoint in few minutes with Regolo Custom Models

1. Paste the Hugging Face URL

2. Choose the GPU

3. Use the Endpoint

You’ll be able to inference your private endpoint in few minutes:

curl -X POST \
https://api.regolo.ai/custom-model/v1/chat/completions/ \
-H "Authorization: Bearer YOUR-REGOLO-API-KEY" \
-H "Content-Type: application/json" \
-d '{
  "model": "YOUR_CUSTOM_MODEL_NAME",
  "messages": [{"role": "user", "content": "Hello!"}]
}'Code language: Bash (bash)

Local Inference: vLLM, Transformers, llama.cpp

vLLM (High-Throughput Serving)

from vllm import LLM, SamplingParams

# AutoRound format (native)
llm = LLM(model="your-username/your-model-gguf")  # or local path
# GGUF format (requires vLLM 0.6.3+)
# llm = LLM(model="./model-q4_k_m.gguf")

prompts = ["The future of AI is", "Write a haiku about quantization:"]
params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=128)
outputs = llm.generate(prompts, params)

for o in outputs:
    print(f"Prompt: {o.prompt!r}")
    print(f"Generated: {o.outputs[0].text!r}\n")Code language: Python (python)

Note: vLLM reads AutoRound format natively (since v0.85.post1). For GGUF, use vLLM 0.6.3+ with --quantization gguf.

Transformers (Flexible, CPU/GPU)

from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRoundConfig  # MUST import

# AutoRound format
model = AutoModelForCausalLM.from_pretrained(
    "./qwen3-8b-multi",  # or HF repo
    device_map="auto",
    torch_dtype="auto",
    quantization_config=AutoRoundConfig(backend="auto")  # cuda/cpu/hpu
)
tokenizer = AutoTokenizer.from_pretrained("./qwen3-8b-multi")

# GGUF format (Transformers 4.42+)
model = AutoModelForCausalLM.from_pretrained(
    "your-username/your-model-gguf",
    gguf_file="model-q4_k_m.gguf",
    device_map="auto",
    torch_dtype="auto"
)

text = "Quantization makes models"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))Code language: Python (python)

Critical: Never .to('cpu') or .cuda() on a quantized model — let device_map="auto" handle placement. Manual device moves break quantization kernels .

llama.cpp / Ollama (Maximum Compatibility)

# Direct llama.cpp
./llama-cli -m model-q4_k_m.gguf -p "The meaning of life is" -n 128

# Ollama (create Modelfile first)
cat > Modelfile << 'EOF'
FROM ./model-q4_k_m.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER temperature 0.7
EOF
ollama create my-qwen3-q4km -f Modelfile
ollama run my-qwen3-q4km "Explain AutoRound in one sentence"Code language: Bash (bash)

Advanced: Mixed-Bit, VLM, and Calibration Datasets

Mixed-Bit Quantization (Surgical VRAM Control)

Assign different bits per layer/module via --layer_config or AutoScheme:

# CLI: scheme-based (avg bits target, per-layer options)
auto-round \
  --model Qwen/Qwen3-8B \
  --scheme '{"avg_bits": 3.5, "options": ["GGUF:Q2_K_S", "GGUF:Q4_K_S"]}' \
  --layer_config '{"lm_head": "GGUF:Q6_K"}' \
  --iters 0 \
  --format "gguf:q4_k_m" \
  --output_dir ./mixed-bitCode language: Bash (bash)

# API: granular control
from auto_round import AutoScheme, AutoRound

scheme = AutoScheme(
    avg_bits=3.5,
    options=("GGUF:Q2_K_S", "GGUF:Q4_K_S"),
    ignore_scale_zp_bits=True
)
layer_config = {"lm_head": "GGUF:Q6_K"}  # keep output head higher precision

autoround = AutoRound(model, tokenizer, scheme=scheme, layer_config=layer_config, iters=0)
autoround.quantize_and_save("./mixed-api", format="gguf:q4_k_m")Code language: Python (python)

Why it works: attention layers are quantization-sensitive; FFN layers tolerate 2-bit. Mixed-bit recovers 1–2% accuracy at same model size.

Qwen3-8B-GGUF-Q2KS-AS-AutoRound

Below the link to download the model quantized and the code to use in your local machine.

Download from HF

Custom Calibration Data

# Local JSONL (one text field per line)
auto-round --model ... --dataset ./my_corpus.jsonl

# Multiple datasets with params
auto-round --model ... \
  --dataset "NeelNanda/pile-10k:split=train,num=512,mbpp:split=train+val,num=256,apply_chat_template=True"

# Code models
auto-round --model Salesforce/codegen25-7b-multi --bits 4 --dataset "mbpp" --seqlen 128Code language: Bash (bash)

Pro tip: for instruct/chat models, always use apply_chat_template=True. Calibration distribution must match inference distribution.

VLM Quantization Details

# Text tower only (default, faster)
auto-round-mllm --model Qwen/Qwen2-VL-7B-Instruct --bits 4 --group_size 32 --format "gguf:q4_k_m"

# Full model (experimental, slower, vision encoder quantized)
auto-round-mllm --model Qwen/Qwen2-VL-7B-Instruct --bits 4 --group_size 32 --quant_nontext_module --format "gguf:q4_k_m"Code language: Bash (bash)

Vision encoder quantization is brittle — test thoroughly. Text-only is production-ready .

Troubleshooting & Known Issues

Symptom	Cause	Fix
OOM on 24 GB GPU	70B 4-bit needs ~35 GB VRAM	`--low_gpu_mem_usage` + `--train_bs 1 --gradient_accumulate_steps 8` + `--group_size 64`
Random results across runs	Non-deterministic seeding on some archs	Set `torch.manual_seed(42)` before `AutoRound()`; use `--iters 0` (RTN) for determinism
ChatGLM v1 fails	Unsupported architecture	Use `auto-gptq` or `awq` instead
GGUF not loading in Ollama	Missing tokenizer files	Copy `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` alongside `.gguf`
vLLM “quant_method not supported”	vLLM version too old	Upgrade: `pip install -U vllm>=0.85.post1`
Transformers “AutoRoundConfig not found”	Missing import	Add `from auto_round import AutoRoundConfig` BEFORE `from_pretrained`
Accuracy tanked at 2-bit	Default recipe insufficient	Use `auto-round-best` + `--enable_alg_ext` (v0.6.1+) + `--low_gpu_mem_usage`
Slow quantization	Large seqlen, large batch	Reduce `--seqlen 512 --train_bs 4` (accuracy trade-off)

Known limitations (v0.6.0):

ChatGLM v1 unsupported
Random quantization variance on some models (set seed)
VLM full-model quantization experimental
Gaudi support limited

FAQ

Can I quantize a fine-tuned LoRA/PEFT model?

Yes. Merge LoRA first: model.merge_and_unload(), then quantize the merged FP16 model. AutoRound doesn’t quantize adapters directly.

Does AutoRound support AWQ/GPTQ export for existing pipelines?

Yes. --format "auto_awq,auto_gptq,gguf:q4_k_m" exports all three. AWQ/GPTQ require symmetric quantization (--sym) for Marlin kernel compatibility .

How do I quantize for CPU-only deployment?

Use --format "auto_round" and install intel-extension-for-pytorch. AutoRound CPU kernels are optimized for x86 (AVX2/AVX-512/AMX). GGUF also runs on CPU via llama.cpp — often faster than PyTorch on non-Intel CPUs.

What’s the difference between `q4_k_m` and `q4_0`?

q4_k_m = k-quant (mixed 4/6/8-bit per block, better accuracy). q4_0 = legacy 4-bit uniform. Always prefer k-quant variants (q2_k_s, q3_k_m, q4_k_m, q5_k_m, q6_k).

Can I re-quantize an already quantized model?

No. Quantization is destructive. Always quantize from FP16/BF16 source. If you only have a GGUF, dequantize to FP16 first (llama.cpp convert-hf-to-gguf.py reverse) — but expect generation degradation.

How do I benchmark my quantized model?

bash# lm-eval-harness (built-in)
auto-round --model ./quantized --eval --tasks mmlu,gsm8k,hellaswag --eval_bs 16

# Custom: perplexity on your domain data
python -m auto_round.eval --model ./quantized --dataset ./my_test.jsonl --metric ppl

Is AutoRound better than GPTQ/AWQ?

At 4-bit: comparable. At 3-bit and 2-bit: AutoRound wins consistently due to sign-gradient optimization. GPTQ/AWQ are one-shot; AutoRound iteratively refines rounding .

Can I use AutoRound in a CI/CD pipeline?

Absolutely. auto-round-light --model $MODEL --bits 4 --format "gguf:q4_k_m" --output_dir ./artifacts completes in ~3 min for 7B on A10G. Add --disble_eval to skip eval. Publish artifacts as pipeline output.

Start your free 30-day trial at regolo.ai and deploy LLMs with complete privacy by design.

👉 Talk with our Engineers or Start your 30 days free →

Discord – Share your thoughts
GitHub Repo – Code of blog articles ready to start
Follow Us on X @regolo_ai
Open discussion on our Subreddit Community
Full list of model available: Models

Built with ❤️ by the Regolo team. Questions? regolo.ai/contact or chat with us on Discord

Table of Contents