Navigating the Jungle of LLM Quantization Formats, GGUF, GPTQ, AWQ and EXL2 - Which One To Pick?

Quantizing LLMs reduces calculation precision and thus the required GPU resources, but it can sometimes be a real jungle trying to find your way among all the existing formats.

GGUF, GPTQ, AWQ, EXL2... Which quantization format to choose and why?

There are so many standards and tools because quantization is not just about rounding values. There are in fact more advanced strategies and different philosophies. It's a bit like images with JPEG, PNG, SVG, WEBP formats...

Take GGUF, the format popularized by llama.cpp, its goal is to reduce precision while optimizing calculations from a CPU perspective, with a particular focus on Apple hardware. It remains possible to offload some of the weights to the GPU for more speed. To run an LLM locally, it's therefore a good candidate, especially if you have a Mac.

GPTQ is oriented towards GPU inference performance. It quantizes the weights by minimizing error thanks to a calibration dataset. This allows comparing the outputs of the full precision and quantized model. At inference, GPTQ dynamically "dequantizes" the weights in float16 to limit loss of precision, all with low impact on GPU memory.

AWQ is based on the observation that ~1% of weights contribute significantly to model performance; these must therefore be treated with special care by applying per-channel scaling. AWQ uses a dataset to analyze activation distributions during inference and identify critical weights. AWQ is faster at inference than GPTQ and also seems to have better perplexity but requires slightly more VRAM.

EXL2 uses the GPTQ philosophy but allows mixing weight precisions within the same model. Some critical weights thus retain high precision, with the rest being more quantized to optimize performance. EXL2 probably offers the fastest inference, at the cost of slightly higher VRAM consumption than AWQ.

To finetune a quantized LLM with QLoRA, you’ll only be able to do it with GPTQ (or with bitsandbytes for on-the-fly quantization).

If you want to avoid quantizing an LLM yourself, the user TheBloke on Huggingface has close to 4000 models ready to use in GPTQ, GGUF or AWQ formats.

But things are moving so fast, it’s hard to stay abreast of the latest advances and it wouldn't be surprising to see new standards emerge in the coming months.