This site contains affiliate links. We may earn a commission at no extra cost to you. This helps us keep the site running and continue providing free guides and comparisons.
The Bottom Line#
Quantization shrinks a language model by storing each of its numbers with fewer bits, usually 4 instead of 16. The file gets about 70% smaller, the model fits into the memory of a normal laptop or graphics card, and the measured quality loss at 4 bits is small. That is the whole reason a 27-billion-parameter model can run at home.
If you only remember one thing: pick Q4_K_M unless you have a reason not to. Go up to Q5 or Q6 if you have memory to spare, and treat anything below 3 bits as a last resort.
A concrete example runs through this guide. Qwen3.8-27B is 54.66 GB unquantized and 16.56 GB in Unsloth's Q4_K_M file. Same model, same 27 billion parameters, less than a third of the size.
File sizes and tags were read from Hugging Face and the Ollama library on 13 November 2026. The quality figures come from the sources named next to them. My own measurement is limited to one model on one Mac, and it is marked as such.
What is LLM quantization?#
LLM quantization means rounding the weights of a model to a coarser number format so that each weight takes less space. The weights, also called parameters, are the billions of numbers a model learned in training. A weight stored with 16 bits can take 65,536 different values. With 4 bits there are 16. The model keeps all its weights, but each one is stored less precisely.
Saving a photo as a JPEG instead of a RAW file is a fair comparison. The file is far smaller, and at a sensible setting most people cannot tell the difference. Push the compression too far and you start to see artifacts. Quantization behaves the same way, and the rest of this guide is about where "too far" begins.
Size matters because of how local models run. The llama.cpp documentation states that models are fully loaded into memory, so you need as much RAM as the file is large, before the context is added on top. The context is the text the model keeps in view while it works: your prompt, attached documents and the conversation so far. The documentation's own example: Llama 3.1 8B is 32.1 GB in its original form, which works out to 32 bits per weight, and 4.9 GB as Q4_K_M. How that translates into machines is covered in How much RAM do you need for a local LLM?
What is GGUF?#
GGUF is the file format in which most quantized models are distributed. Hugging Face describes it as "a binary format that is optimized for quick loading and saving of models", and it holds the weights and the metadata the runtime needs in a single file. It comes from the llama.cpp project and is read by llama.cpp, Ollama, LM Studio and GPT4All.
GGUF and quantization are two different things. GGUF is the container, quantization is the precision of what is inside. A GGUF file can hold an unquantized 16-bit model or a 2-bit one. That is why one model appears on Hugging Face as a long list of .gguf files with different suffixes.
GGUF is not the only format. On Macs with Apple Silicon, Ollama and LM Studio also run builds for MLX, Apple's own machine-learning framework. In the Ollama library these carry tags such as nvfp4 (4-bit) and mxfp8 (8-bit) instead of the Q names. The idea is the same: fewer bits per weight.
What do names like Q4_K_M mean?#
The suffix tells you how many bits a weight gets and which method was used. Read it in three parts:
| Part | Meaning |
|---|---|
Q4 | Quantized, about 4 bits per weight. Q8 is 8 bits, Q2 is 2 bits. |
K | A "k-quant". This family was added to llama.cpp in June 2023 by Iwan Kawrakow and stores weights in blocks that share a scale. |
S, M, L, XL | Small, medium, large. The mix decides how many weight groups, called tensors, are stored with extra bits. |
Q4_K_M is a good example of why the letters matter. According to the original pull request, it stores most tensors at 4 bits but uses 6 bits for half of two weight groups, named attention.wv and feed_forward.w2 there. The result is about 4.9 bits per weight on average, not 4.0.
Other labels you will meet:
IQ4_XS,IQ3_XXSand similar. "I-quants" that use an importance matrix to decide which weights need precision. They are smaller than the K file with the same number: Hugging Face listsIQ4_XSat 4.25 bits per weight andQ4_Kat 4.5.Q4_0,Q8_0. Older round-to-nearest methods. Hugging Face labels them "legacy".Q8_0is still offered for most models as the 8-bit option, and Google's QAT files for Gemma 4, explained below, are listed asQ4_0in the Ollama library.F16,BF16. Not quantized. This is the 16-bit original.UD-. A prefix used by Unsloth for its "dynamic" quants. It names a provider's recipe, not a llama.cpp type.- imatrix. Short for importance matrix, a calibration step during quantization. The llama.cpp documentation says a suitable imatrix file can reduce the accuracy loss.
How much quality do you lose with quantization?#
Down to about 4 bits the measured loss is small, and below 3 bits it climbs quickly. Three independent sources point the same way.
The original k-quant measurements. Kawrakow published perplexity figures for a 7B model with the pull request. Perplexity measures how surprised a model is by a test text, so lower is better. The last column is my calculation from his numbers:
| Quantization | File size | Perplexity | Change vs 16-bit |
|---|---|---|---|
| F16 (original) | 13.0 GB | 5.9066 | |
| Q6_K | 5.15 GB | 5.9110 | +0.07% |
| Q5_K_M | 4.45 GB | 5.9208 | +0.24% |
| Q4_K_M | 3.80 GB | 5.9601 | +0.91% |
| Q3_K_M | 3.06 GB | 6.1503 | +4.1% |
| Q2_K | 2.67 GB | 6.7764 | +14.7% |
These figures are from 2023 and an older model generation. Treat them as the shape of the curve, not as exact values for today's models: flat until 4 bits, then steep.
A large evaluation on newer models. In October 2024 a team at Neural Magic, now part of Red Hat, ran more than half a million evaluations on quantized Llama 3.1 models with 8B, 70B and 405B parameters. On the coding benchmark HumanEval, 8-bit models recovered 99.9% and 4-bit models 98.9% of the original score. The 8B model showed more variation than the larger ones. The study used server formats for vLLM, not GGUF, so it confirms the principle rather than a specific file.
An academic study. Jin et al., published in the Findings of ACL 2024, tested 8-, 4-, 3- and 2-bit models on ten benchmarks. They found that 4-bit models stay close to their unquantized versions and that larger quantized models often beat smaller ones.
That last point is the most useful rule in practice. Kawrakow's tables point the same way: the 13B model at Q4_K_M, 7.32 GB, reached a perplexity of 5.3002, clearly better than the 7B model at full 16-bit precision with 13.0 GB and 5.9066. Even the 13B model at Q2_K, 5.13 GB, stayed slightly ahead at 5.8545. This is one pair of models from 2023 and perplexity only, so read it as a tendency: a bigger model at 4 bits tends to beat a smaller model at higher precision.
One limit applies to all three sources. None of them tested long agent runs with many tool calls on low-bit files. If that is your workload, test the quantization you plan to use before you rely on it.
Does quantization make a model faster?#
Usually yes, because the computer has to move less data for every token, the word fragments a model reads and writes. The llama.cpp documentation lists generation speeds for Llama 3.1 8B: 29 tokens per second at F16, 51 at Q8_0, 59 at Q6_K and 72 at Q4_K_M. The page does not name the hardware, so compare the ratios, not the absolute numbers.
The bigger effect is whether the model fits into fast memory at all. A model that no longer fits completely into your graphics card or unified memory and spills over to the CPU slows down noticeably. How to Run an LLM Locally shows how to check that with ollama ps.
One model, 25 quantized files: Qwen3.8-27B#
To see what the choice looks like in practice, here is a real repository. unsloth/Qwen3.8-27B-GGUF offers 25 quantized versions of the same model next to the unquantized original. A selection, with bits per weight calculated from the file size and the 27.3 billion parameters in the file metadata:
| File | Size | Bits per weight | Share of original |
|---|---|---|---|
| BF16 (original) | 54.66 GB | 16.0 | 100% |
| Q8_0 | 29.05 GB | 8.51 | 53% |
| UD-Q6_K | 21.98 GB | 6.44 | 40% |
| UD-Q5_K_M | 19.87 GB | 5.82 | 36% |
| UD-Q4_K_XL | 17.66 GB | 5.17 | 32% |
| UD-Q4_K_M | 16.56 GB | 4.85 | 30% |
| UD-Q4_K_S | 15.36 GB | 4.50 | 28% |
| UD-IQ4_XS | 14.35 GB | 4.20 | 26% |
| UD-Q3_K_XL | 13.15 GB | 3.85 | 24% |
| UD-Q2_K_XL | 9.95 GB | 2.91 | 18% |
| UD-IQ2_XXS | 7.21 GB | 2.11 | 13% |
Two details are easy to miss. First, "4-bit" alone covers files from 14.35 to 17.66 GB in this one repository, so always look at the full suffix. Second, Qwen3.8-27B reads images, and that needs a separate projector file of 0.93 GB. Ollama bundles both: its qwen3.8:27b tag is the Q4_K_M model at 17 GB plus the projector, 18 GB in total.
Unsloth's own guidance for total memory is 12 to 14 GB for 3-bit, 16 to 19 GB for 4-bit, 23 to 26 GB for 6-bit and 31 GB for 8-bit files.
What I measured on my own Mac#
I run the MLX build qwen3.8:27b-mlx through Ollama on a MacBook Pro with an M4 Max and 36 GB of unified memory, the memory a Mac shares between processor and graphics chip. ollama show reports its quantization as nvfp4, a 4-bit format, and the download is 18 GB.
On 13 November 2026 with Ollama 0.40.14, three short runs of 400 generated tokens each gave this: the model loaded in 17.6 seconds, generated 35.1 to 37.8 tokens per second and occupied 19 GB at the default 32K context, fully on the GPU according to ollama ps.
This is one machine and one 4-bit build. It is not a comparison between quantization levels, and I did not measure answer quality. What it does show: a 4-bit 27B model is comfortable on 36 GB, and on a 24 GB Mac the same 19 GB would leave about 5 GB for macOS and everything else.
Which quantization should you pick?#
Start at Q4_K_M and move only if your memory tells you to. In order:
- Default: Q4_K_M. It is what Ollama's standard tags ship for many models, and Hugging Face uses it by default when you load a GGUF into Ollama. LM Studio gives the same advice in its documentation: "Choose a 4-bit option or higher if your machine is capable enough for running it."
- Memory to spare: Q5_K_M or Q6_K. You buy a small quality margin. Beyond Q6 the returns shrink fast, since Q6_K was already within 0.1% of the 16-bit original in Kawrakow's measurements.
- Too tight: go down in steps. Try Q3 first: Q3_K_M cost 4.1% in the measurements above. Below 3 bits the loss gets large, so at that point compare with the next smaller model at Q4 instead of assuming the bigger one still wins.
- Leave room for the context. The file size is the minimum, not the total. Context length adds memory on top, and long documents or agents need a lot of it.
- If a QAT version exists, look at it first for 4 bits. See the next section.
For Qwen3.8-27B that works out roughly as follows. This is arithmetic from the file sizes above, not something I tested on each machine:
- 16 GB graphics card: UD-Q3_K_XL (13.15 GB) with a short context. The 4-bit files need 16 to 19 GB by Unsloth's guidance.
- 24 GB graphics card or 32 GB Mac: UD-Q4_K_M (16.56 GB) to UD-Q5_K_M (19.87 GB).
- 48 GB Mac or more: Q8_0 (29.05 GB) if you want the near-lossless version.
- 12 GB or less: not this model. Take a 12B model instead.
What is QAT?#
QAT stands for quantization-aware training. It is one of the two main types of quantization. The usual kind, post-training quantization, compresses a finished model. With QAT the model already learns during training to cope with the rounding.
Google released QAT versions of all Gemma 4 sizes on 5 June 2026. In its announcement the company writes: "By simulating quantization during training, QAT minimizes quality loss when the model is compressed." Google reports higher overall quality than standard post-training quantization, though the article gives no figures in its text.
The sizes in the Ollama library show what you get. For Gemma 4 12B:
| Tag | Precision | Download |
|---|---|---|
gemma4:12b-it-bf16 | 16-bit original | 24 GB |
gemma4:12b-it-q8_0 | 8-bit | 13 GB |
gemma4:12b-it-q4_K_M | 4-bit, post-training | 8.0 GB |
gemma4:12b-it-qat | 4-bit, QAT | 7.2 GB |
The QAT file is the smallest of the four and the only one the vendor trained for this precision. A smaller file proves nothing about quality, and I found no independent comparison, so the advantage rests on Google's statement. A QAT file has to be created during training, in practice by the model's maker, which is why it exists for few model families. Where it exists, it is my first choice at 4 bits for that reason.
Which local models are worth running in November 2026?#
These are picks by memory class, each with the exact Ollama tag and the size the library listed on 13 November 2026. They follow the size classes in our local LLM guide. Memory means the VRAM of the graphics card on a PC and unified memory on a Mac. The picks are chosen by fit, not by quality: the download leaves room for the context in that memory class, and the tag exists ready-made in the Ollama library. I have not benchmarked them against each other.
| Your memory | Model | Ollama tag | Download | Precision |
|---|---|---|---|---|
| 8 GB | Qwen3.5 4B | qwen3.5:4b-q4_K_M | 3.3 GB | Q4_K_M |
| 8 GB | Gemma 4 E2B | gemma4:e2b-it-qat | 4.3 GB | 4-bit QAT |
| 16 GB | Gemma 4 12B | gemma4:12b-it-qat | 7.2 GB | 4-bit QAT |
| 16 GB | Qwen3.5 9B | qwen3.5:9b-q4_K_M | 6.6 GB | Q4_K_M |
| 24 GB | gpt-oss 20B | gpt-oss:20b | 14 GB | MXFP4 |
| 24 GB | Devstral Small 2 | devstral-small-2:24b-instruct-2512-q4_K_M | 15 GB | Q4_K_M |
| 32 GB | Qwen3.8 27B | qwen3.8:27b | 18 GB | Q4_K_M |
| 32 GB | Gemma 4 31B | gemma4:31b-it-qat | 19 GB | 4-bit QAT |
| 48 GB and up | Qwen3.8 27B | qwen3.8:27b-q8_0 | 30 GB | Q8_0 |
| 48 GB and up | Qwen3.6 35B-A3B | qwen3.6:35b-a3b | 23 GB | Q4_K_M |
Qwen3.8-27B is my pick for 32 GB. Alibaba's Qwen team published it in August 2026 under the Apache 2.0 license. It is a dense 27B model, so all parameters work on every token, and it reads images and video, with a native context of 262,144 tokens and a thinking mode that is on by default. On the model card, Alibaba reports 73.0 on Terminal Bench 2.1 and 61.7 on SWE-bench Pro, against 63.4 and 53.5 for its predecessor Qwen3.6-27B. Those are the vendor's own figures for the model, not measurements of the 4-bit file you download.
If you mainly write code, Best Local LLM for Coding goes into context sizes and agent setups. If no machine you own fits the model you want, renting a GPU by the hour is the alternative, which our RunPod guide covers.
How do you get a specific quantization?#
In Ollama, the quantization is part of the tag:
ollama run qwen3.8:27b # Q4_K_M, 18 GB
ollama run qwen3.8:27b-q8_0 # Q8_0, 30 GB
ollama show qwen3.8:27b # prints the quantization of an installed model
The tag list of each model, for example qwen3.8/tags, shows every available precision with its size.
Ollama can also load GGUF files straight from Hugging Face. The format is ollama run hf.co/{username}/{repository}:{quantization}, for example:
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
In LM Studio you choose among the download options of a model, which carry the same kind of names. For the choice between the two apps, see Ollama vs LM Studio.
Common Mistakes#
- Counting only the file size. The context comes on top. A file that fills your memory to the last gigabyte will not run well.
- Choosing Q8 of a small model over Q4 of a larger one. The measurements above point the other way.
- Dropping to 2 bits without comparing. In Kawrakow's table, Q2_K was 14.7% worse in perplexity, against 0.91% for Q4_K_M. Test the next smaller model at Q4 against it.
- Quantizing an already quantized file. The llama.cpp documentation warns that this "can severely reduce quality compared to quantizing from 16bit or 32bit".
- Reading "Q4" as one fixed thing. Method, mix and provider all change the file, as the Qwen3.8 table shows.
What This Guide Does Not Cover#
- No quality benchmark of my own. The quality figures are from the llama.cpp project, Red Hat and Jin et al. My measurement covers speed and memory for one 4-bit build on one Mac.
- Server formats. FP8, AWQ and other formats for vLLM on data-center GPUs are a separate topic. This guide is about files you run on your own machine.
- How to quantize a model yourself. Ready-made files exist for almost every open model, so beginners rarely need to.
- A quality ranking of models. The table sorts by memory class from documented sizes on one date. Model names change faster than the rules for picking a quantization.
FAQ#
What is quantization in AI?#
Quantization stores the weights of an AI model with fewer bits, typically 4 instead of 16. The model becomes much smaller and needs less memory, at the cost of a small loss in precision. It is the reason large language models can run on ordinary computers.
What does Q4_K_M mean?#
Q4 stands for about 4 bits per weight, K for the k-quant method in llama.cpp, and M for the medium mix, which gives extra bits to the layers that are most sensitive to rounding. On average a Q4_K_M file uses about 4.9 bits per weight.
Is Q4_K_M good enough?#
For most uses, yes. In the original llama.cpp measurements, Q4_K_M was 0.91% worse in perplexity than the 16-bit model, and a Red Hat evaluation found 4-bit models at 98.9% of the original score on a coding benchmark. It is also what Ollama ships for many models and what Hugging Face picks by default.
Q4_K_M vs Q5_K_M: which should I use?#
Take Q5_K_M if it fits into your memory with room left for the context, otherwise Q4_K_M. In Unsloth's files for Qwen3.8-27B the difference is 19.87 GB against 16.56 GB. The quality gap between the two is small, so never give up context or a larger model for it.
What are the two main types of quantization?#
Post-training quantization and quantization-aware training. Post-training quantization compresses a finished model and is what almost all GGUF files use. Quantization-aware training, or QAT, simulates the rounding during training so the model adapts to it, as Google does for Gemma 4.
Is GGUF the same as quantization?#
No. GGUF is a file format, quantization is the precision of the weights inside the file. A GGUF file can contain a 16-bit model or a 2-bit one, which is why the same model is offered as many GGUF files.
How much RAM does Qwen3.8-27B need?#
Plan for 32 GB for the 4-bit version. Unsloth's Q4_K_M file is 16.56 GB, Ollama's bundle of model and image projector is 18 GB, and on my 36 GB Mac the 4-bit MLX build occupied 19 GB at a 32K context. The 8-bit version at 30 GB needs a machine with 48 GB or more.
Sources#
- llama.cpp: quantize documentation and pull request 1684, k-quants by Iwan Kawrakow, June 2023
- Hugging Face: GGUF documentation and Ollama with GGUF models
- Qwen3.8-27B: model card, unsloth/Qwen3.8-27B-GGUF (file sizes via the Hugging Face API, 13 November 2026) and Unsloth documentation
- Ollama library tags: qwen3.8, gemma4, qwen3.5, qwen3.6, gpt-oss, devstral-small-2
- Google: Gemma 4 QAT models, 5 June 2026
- Red Hat Developer: We ran over half a million evaluations on quantized LLMs, 17 October 2024
- Jin et al.: A Comprehensive Evaluation of Quantization Strategies for Large Language Models, Findings of ACL 2024
- LM Studio: Download an LLM
