LLM Quantization Guide: GGUF vs AWQ vs MLX in 2026

Quantization Is the Reason a 30B Model Fits on Your Desk

Every time we cover a mini PC that runs a 30-billion-parameter model, or a Raspberry Pi doing useful local inference, the thing making it possible is quantization. It is the single highest-leverage concept in self-hosted AI, and it is also where most newcomers lose a weekend to confusing filenames like Q4_K_M, AWQ-INT4, and 4bit-mlx.

This guide explains what those actually mean, how much memory each option needs, what you give up, and which format to pick for the machine you already own.

• Quantization shrinks a model by storing its weights at lower numeric precision, typically 4-bit or 8-bit instead of 16-bit
• A rough memory rule: parameters × bits-per-weight ÷ 8 = gigabytes, plus 1–3 GB of overhead for context and runtime
• GGUF is the format for CPU and mixed CPU/GPU use; AWQ and GPTQ target GPU serving; MLX targets Apple Silicon
• 4-bit quantization typically costs only a small amount of quality on most tasks, and the loss grows sharply below 4-bit

Quick Picks

• A laptop, mini PC, or single-board computer: GGUF at Q4_K_M, run through llama.cpp or Ollama
• An NVIDIA GPU serving multiple users: AWQ 4-bit through vLLM
• A Mac with Apple Silicon: MLX 4-bit
• You have plenty of VRAM and want maximum fidelity: 8-bit (Q8_0 or FP8)
• You are on a very tight memory budget: Q3_K_M, and step down to a smaller model before going below 3-bit

What Is Quantization, Exactly?

A trained model stores its weights as numbers — usually 16-bit floating point. Quantization rewrites those numbers at lower precision, most often 4 or 8 bits each, with a small amount of extra bookkeeping so the original values can be approximately reconstructed during inference.

The effect is close to linear on memory. A 14-billion-parameter model at 16-bit needs roughly 28 GB just for weights. The same model at 4-bit needs roughly 7 GB. That difference decides whether a model runs on the hardware you own or does not run at all.

The cost is precision. Rounding weights introduces small errors, and those errors accumulate through the network. The good news, established repeatedly across the open-weight ecosystem, is that modern 4-bit methods are remarkably well-behaved — the quality drop on general tasks is usually modest. The bad news is that the curve is not gentle forever: below 4 bits, degradation accelerates quickly, especially on reasoning, code, and long-context work.

The Memory Math You Actually Need

Here is the calculation worth memorizing:

GB needed ≈ (parameters in billions × bits per weight ÷ 8) + overhead

Overhead covers the KV cache for your context window, the runtime itself, and activations — budget 1–3 GB for typical use, more if you are running very long contexts.

Worked examples at 4-bit:

• 8B model: 8 × 4 ÷ 8 = 4 GB + overhead ≈ 5–7 GB
• 14B model: 14 × 4 ÷ 8 = 7 GB + overhead ≈ 8–10 GB
• 30B model: 30 × 4 ÷ 8 = 15 GB + overhead ≈ 17–20 GB
• 70B model: 70 × 4 ÷ 8 = 35 GB + overhead ≈ 38–44 GB

This is why 16 GB is the practical entry point for comfortable local AI, why 32 GB opens up 30B-class models, and why the 128 GB unified-memory machines we cover in our best mini PC for local LLMs guide can host 70B models that would otherwise require multiple discrete GPUs.

Comparing the Formats

| Format | Best hardware | Typical bit depths | Runtimes | Strengths | Trade-offs |

|---|---|---|---|---|---|

| GGUF | CPU, mixed CPU+GPU, SBCs | 2–8 bit (K-quants) | llama.cpp, Ollama, LM Studio | Runs almost anywhere; layer offloading; huge model selection | Lower peak throughput than GPU-native formats |

| AWQ | NVIDIA GPUs | 4-bit | vLLM, TGI, SGLang | Strong quality retention at 4-bit; fast batched serving | GPU-only; fewer ready-made conversions |

| GPTQ | NVIDIA GPUs | 3–8 bit | vLLM, TGI, AutoGPTQ | Mature, widely available, well understood | Generally edged out by AWQ at equal size |

| MLX | Apple Silicon | 4–8 bit | mlx-lm, LM Studio | Native to unified memory; excellent Mac efficiency | Apple hardware only |

| FP8 / INT8 | Modern data-center GPUs | 8-bit | vLLM, TensorRT-LLM | Near-lossless quality | Roughly double the memory of 4-bit |

What Do the GGUF Suffixes Mean?

GGUF filenames carry a code like Q4_K_M, and it is simpler than it looks:

• Q4 — the nominal bit depth, here 4-bit
• _K — K-quant, the modern scheme that varies precision across different parts of the model instead of applying one setting uniformly
• _S / _M / _L — small, medium, or large variant within that bit depth, trading file size against quality

In practice, `Q4_K_M` is the default worth starting with. It is the best-balanced option for most models and most hardware. Move up to Q5_K_M or Q6_K if you have memory to spare and want more headroom on precision-sensitive work; move down to Q3_K_M only when the alternative is not running the model at all.

Which Format Should You Choose?

If You Run on CPU or a Small Board

Use GGUF. It is the only ecosystem designed from the start for CPU inference and partial GPU offload, which means you can push as many layers onto a GPU as fit and let the CPU handle the rest. This is what makes local AI viable on the kind of hardware we cover in our edge AI dev board guide — and it is why nearly every accessible local-AI tool ships GGUF support first.

If You Serve a GPU to Multiple Users

Use AWQ through vLLM. Activation-aware quantization protects the weights that matter most for output quality, and it consistently holds up well at 4-bit. Paired with vLLM's batching, it is the standard answer for a small team sharing one GPU.

If You Are on a Mac

Use MLX. Apple Silicon's unified memory means the GPU and CPU share one pool, and MLX is built to exploit that directly. On an M-series machine with 32 GB or more, MLX 4-bit models are frequently faster and more memory-efficient than the GGUF equivalent — though GGUF via llama.cpp remains an excellent fallback with broader model coverage.

How Much Quality Do You Actually Lose?

For everyday use — summarizing, drafting, answering questions, general chat — the difference between a 16-bit model and a good 4-bit quantization of the same model is usually hard to notice. Where the gap shows up is in the demanding cases: multi-step reasoning, code generation, precise instruction following, and very long contexts, where small errors compound.

The most useful heuristic in local AI is this: a larger model at 4-bit almost always beats a smaller model at 8-bit for the same memory budget. A 14B model at Q4_K_M will generally outperform an 8B model at Q8_0, and both occupy roughly 8–10 GB. Spend your memory on parameters first, precision second — until you drop below 4-bit, where that rule stops holding.

Quantization-aware training changes the calculus further. When a model is trained with quantization in mind rather than compressed afterward, the losses shrink dramatically — the approach behind releases like Gemma 4 QAT, which cut local memory use by around 72%. If a QAT build of your chosen model exists, prefer it.

Frequently Asked Questions

Does quantization make inference faster?

Usually yes, though not for the reason people expect. Local inference is typically bound by memory bandwidth, not arithmetic — so moving fewer bytes per token means more tokens per second. The speedup is real but rarely proportional to the size reduction.

Can I quantize a model myself?

Yes. llama.cpp includes conversion and quantization tools for GGUF, and AutoAWQ and AutoGPTQ handle their respective formats. That said, for popular open-weight models the community usually publishes quantizations within days of release, so converting yourself is mostly worthwhile for fine-tunes and niche models.

Is 2-bit or 3-bit quantization worth using?

3-bit is a legitimate option when it is the difference between running a much larger model and not running it. 2-bit is generally a last resort — quality degrades sharply, and you are usually better served by a smaller model at 4-bit.

Does quantization affect context length?

Not directly, but the KV cache scales with context and is stored separately from the weights. Long contexts can consume several gigabytes on their own, which is why the overhead figure in the memory formula grows with your context window. Most runtimes now support quantizing the KV cache too, which helps considerably at 32K tokens and beyond.

The Short Version

Start with GGUF at Q4_K_M. If you are on a Mac, start with MLX 4-bit. If you are serving a GPU to a team, start with AWQ. Size your model so that parameters × 4 ÷ 8, plus a couple of gigabytes, fits inside your available memory — then spend anything left over on a bigger model rather than a higher bit depth. That single decision framework covers the overwhelming majority of local AI setups.

For hardware to run it on, see our mini computers coverage and the Raspberry Pi storage guide — storage throughput matters more than most people expect when you are loading multi-gigabyte weight files.

Sources: Hugging Face — Quantization docs; llama.cpp; Apple MLX; vLLM — Quantization.