What Is Model Quantization and Which Format Should You Use for Local LLMs in 2026?

A common question in AI communities like r/LocalLLaMA and r/ollama keeps surfacing: you want to run large language models locally, but you keep seeing strange file extensions like .gguf, .gptq, and .awq. What do these mean? Which one should you download? And why does the same 70B model come in sizes ranging from 40GB to 140GB?

The answer is quantization—and choosing the wrong format can mean the difference between a model that runs smoothly on your hardware and one that either crashes your system or produces gibberish. After testing hundreds of quantized models across different hardware configurations, the differences between formats become stark.

What Is Model Quantization?

Quantization is a compression technique that reduces AI model precision from 16-bit floating point numbers to lower bit widths—typically 4-bit integers. This shrinks model sizes by roughly 75% while maintaining 92-98% of the original quality.

Consider the math: a 70B parameter model in FP16 (16-bit floating point) requires approximately 140GB of VRAM. Quantize that same model to 4-bit, and it drops to around 40GB—small enough to run on a consumer RTX 4090 with 24GB VRAM using smart offloading strategies.

The trade-off is approximation error. Every quantization method introduces some rounding error when converting from high-precision floats to low-precision integers. The art of quantization lies in controlling that error through sophisticated rounding strategies, calibration datasets, and weight protection mechanisms.

Bit Depth Quick Reference

• FP16 (16-bit): Original quality, 100% size baseline
• 8-bit (Q8): ~50% smaller, safe default for sensitive workloads
• 6-bit (Q6): ~62% smaller, balanced speed and quality
• 4-bit (Q4): ~75% smaller, standard for local AI deployment
• 3-bit (Q3): ~81% smaller, experimental, quality drops noticeably

Most local AI practitioners stick with 4-bit quantization as the sweet spot between compression and quality. The three dominant 4-bit formats in 2026 are GGUF, GPTQ, and AWQ—each engineered for different hardware and use cases.

GGUF: The Universal Format

GGUF, developed by the llama.cpp team, emerged as the successor to GGML in late 2023 and has become the default format for cross-platform compatibility. It stores weights in a flexible block-based format with rich metadata, enabling efficient loading on everything from Raspberry Pi devices to NVIDIA H100 data center GPUs.

How GGUF Works

GGUF divides model weights into blocks—typically 32 or 128 elements—and quantizes each block independently using k-quantile methods. This approach handles outliers better than simple rounding, preserving model quality at aggressive compression ratios. The format also supports tensor metadata, allowing applications to inspect model properties without loading the entire file.

The latest GGUF v2.1 specification, released in early 2026, added native support for NVIDIA H100 GPUs through optimized CUDA kernels, significantly closing the performance gap with GPU-native formats.

When to Choose GGUF

• Apple Silicon (M1/M2/M3/M4): GGUF's Metal backend delivers optimal performance on macOS
• CPU-only inference: Efficient AVX2 and AVX-512 implementations extract maximum performance from x86 processors
• Cross-platform deployment: Single file runs on Windows, Linux, macOS, and mobile devices
• Mixed GPU/CPU offloading: Layer-by-layer distribution between GPU and system RAM

A concrete example: Qwen2.5-32B-Instruct in GGUF Q4_K_M format runs comfortably on a MacBook Pro M3 with 36GB unified memory, delivering 22 tokens per second—entirely usable for document analysis and coding assistance.

GGUF Quality Metrics

Benchmark data from 2026 evaluations show GGUF Q4_K_M achieves approximately 92% perplexity retention compared to FP16 baselines. On the MMLU benchmark (massive multitask language understanding), Llama 3.1 8B scores 85.9 with GGUF Q4_K_M versus 87.5 in full precision—a mere 1.6 point drop.

GPTQ: Throughput Champion

GPTQ takes a fundamentally different approach. Instead of block-wise quantization, it employs a one-shot weight quantization method based on approximate second-order information, minimizing the mean squared error for each layer. This produces highly optimized weights for GPU inference—at the cost of requiring calibration during the quantization process.

How GPTQ Works

GPTQ quantizes weights sequentially, layer by layer, using information from the Hessian matrix (second derivatives) to determine which weights can be safely compressed with minimal impact on model output. During inference, weights are dynamically dequantized back to FP16 for computation, then re-quantized—a process that sounds wasteful but actually balances memory bandwidth and compute efficiency effectively on modern GPUs.

The format supports aggressive quantization levels—8, 4, 3, and even 2-bit variants—though 4-bit (INT4) represents the practical standard for quality preservation.

When to Choose GPTQ

• NVIDIA GPU inference: Tensor core utilization delivers 20-30% higher throughput than GGUF
• High-throughput applications: APIs serving multiple concurrent requests benefit from GPTQ's efficiency
• Cloud deployment: Standard format for vLLM and TensorRT-LLM inference engines
• Large models on high-end hardware: The Qwen3-235B model achieves 54 tokens per second on H100 with GPTQ INT4

GPTQ v2.3, released in March 2026, expanded support to AMD Instinct MI300X GPUs, making it viable for heterogeneous data center environments beyond NVIDIA's ecosystem.

GPTQ Quality Metrics

GPTQ 4-bit typically achieves 90% perplexity retention. On the same Llama 3.1 8B model, MMLU scores drop to 84.7—a 2.8 point decrease from baseline. The quality loss is slightly higher than GGUF, but the throughput gains often justify the trade-off for latency-sensitive applications.

AWQ: Quality-First Quantization

Activation-Aware Weight Quantization (AWQ) represents the most sophisticated approach of the three. Rather than treating all weights equally, AWQ analyzes activation distributions during calibration to identify "salient" weights—those that disproportionately affect model outputs—and protects them with higher precision.

How AWQ Works

AWQ observes which weights produce large activation values during forward passes through representative input data. These high-impact weights receive special treatment—either less aggressive quantization or scaled precision—while low-impact weights get compressed more heavily. The result is better preservation of model capabilities, particularly for instruction-following and creative tasks.

AWQ v1.8 added FP8 (8-bit floating point) quantization support in 2026, enabling even higher quality on NVIDIA H100 and AMD Instinct MI300X GPUs that support native FP8 tensor operations.

When to Choose AWQ

• Instruction-tuned models: Better preservation of fine-tuned behaviors and prompt following
• Creative writing tasks: Reduced degradation in narrative coherence and style consistency
• Code generation: Maintains syntactic accuracy better than other 4-bit methods
• Multimodal models: Vision-language models show less performance degradation with AWQ

For developers building coding assistants or creative writing tools, AWQ typically produces the most reliable outputs at 4-bit precision. The trade-off is slightly slower conversion times during the quantization process and marginally higher inference latency compared to GPTQ.

AWQ Quality Metrics

AWQ leads the pack in quality preservation with approximately 95% perplexity retention. On Llama 3.1 8B, MMLU scores hit 86.8—only a 0.7 point drop from the FP16 baseline. This near-lossless compression makes AWQ the choice when quality cannot be compromised.

Head-to-Head Performance Comparison

Benchmark data from 2026 evaluations across multiple model architectures reveals consistent patterns:

Throughputs on comparable hardware (RTX 4090) show GPTQ achieving 50-60 tokens per second, AWQ at 45-55, and GGUF at 35-45 when running GPU-accelerated inference. CPU-only GGUF performance varies widely based on AVX capabilities, ranging from 5-25 tokens per second depending on the processor.

The Decision Framework

Choosing the right quantization format depends on three factors: your hardware, your quality requirements, and your deployment environment.

Use GGUF If:

• You need universal compatibility across operating systems
• You're running on Apple Silicon (M-series chips)
• CPU inference is your only option
• You want simple deployment with tools like Ollama or LM Studio
• You need to offload layers between GPU and system RAM

Use GPTQ If:

• You have NVIDIA GPUs and want maximum throughput
• You're deploying in cloud environments with vLLM or TensorRT-LLM
• Latency is critical (API endpoints, real-time applications)
• You're serving multiple concurrent users

Use AWQ If:

• Quality is your top priority and you can accept slightly slower inference
• You're running instruction-tuned models for specific tasks
• Creative writing or code generation quality matters most
• You have the compute budget for the quantization process

Practical Recommendations for 2026

For most users starting with local LLMs, GGUF Q4_K_M offers the best balance of compatibility and quality. Download through Ollama or directly from Hugging Face, and the model will work across your devices without ecosystem lock-in.

If you've invested in high-end NVIDIA hardware and care about throughput, GPTQ delivers measurably faster inference. The calibration requirement adds complexity, but tools like AutoGPTQ have streamlined the process considerably.

For production applications where output quality directly impacts user experience—coding assistants, content generation tools, specialized domain models—AWQ's activation-aware approach preserves capabilities that other quantization methods degrade.

The quantization landscape continues evolving. FP8 support is expanding beyond H100 to consumer GPUs. New methods like QLoRA and QA-LoRA enable on-device fine-tuning of quantized models. And context windows keep growing—Claude now handles 200K tokens, Gemini reaches 1M—making raw context shoving a viable alternative to retrieval for smaller knowledge bases.

But for running 70B-class intelligence on consumer hardware, quantization remains essential. Understanding these three formats—and choosing the right one for your constraints—separates smooth deployments from frustrating weekends debugging VRAM errors.

Sources

1. Local AI Master — GGUF vs GPTQ vs AWQ: Best AI Quantization in 2026
2. DasRoot — GGUF vs GPTQ vs AWQ: LLM Quantization Methods Compared (January 2026)
3. Premai.io — LLM Quantization Guide: GGUF vs AWQ vs GPTQ vs bitsandbytes Compared (2026)
4. Index.dev — AWQ vs GGUF vs GPTQ: Quantization Methods Compared for AI 2026