The GGUF quantization war intensifies around Qwen3

A benchmark compares several GGUF quantizations of Qwen3-30B-A3B and Qwen3-235B-A22B. Verdict: differences exist, but no format dominates everywhere.

The Qwen3 GGUF quants are the subject of a close comparison pitting bartowski, unsloth, ikawrakow, and ubergarm against each other on the Qwen3-30B-A3B and Qwen3-235B-A22B models. According to the published test, results vary depending on hardware, context, and quantification method, complicating the choice of the "best" file to download. At this stage, the conclusion is simple: they are all quite good, but not interchangeable. And yes, the quant war now has its camps, charts, and little sensitivities.

Why choosing a GGUF is no longer trivial

The article first recalls that the landscape has changed since TheBloke's last publications on Hugging Face. New model architectures, better-optimized inference engines, and more advanced quantification methods have shifted the lines. In this context, a GGUF is no longer just a compact file: it is a compromise between quality, speed, and available memory.

The text also emphasizes the rise of several names that have become essential in the community, including bartowski and unsloth. So-called "static" recipes, where all layers are uniformly quantized, have been disrupted by more dynamic approaches based on importance matrices and finer calibrations. The result: two quants of the same size can now behave differently depending on how they were produced.

Real gaps, but no universal winner

On Qwen3-30B-A3B, measurements show that 4-bit quants remain within a fairly tight range. bartowski's Q4_K_M shows a wiki.test.raw perplexity of 15.194468, versus 15.281833 for unsloth's UD-Q4_K_XL and 15.182811 for ubergarm's IQ4_KS. Differences exist, but they remain modest at this compression level.

At higher compression, the gap widens further. On the same model, bartowski's Q2_K_L drops to 9.7 GB, while unsloth's UD-Q2_K_XL reaches 10.1 GB and ubergarm's IQ2_M 9.7 GB. Perplexity and KL divergence scores then vary more noticeably, confirming that a very compact quant can maintain correct behavior without fully matching the reference model's outputs everywhere.

The comparison on Qwen3-235B-A22B goes in the same direction. ubergarm's Q8_0 serves here as the baseline, with a size of 233 GB. Other variants, such as the 107 GB mix-IQ3_K, the 104 GB Q3_K_L, and the 97 GB UD-Q3_K_XL, greatly reduce memory footprint, but distance metrics from the original model mechanically increase. This is the price to pay to fit a very large model on more reasonable hardware.

Speed matters as much as accuracy

The test does not stop at quality. Speed measurements were performed with llama-sweep-bench, noting however the probable absence of warmup, which penalizes the first measurement point. On Qwen3-30B-A3B, bartowski's Q4_K_M reaches around 2,746 tokens/s in prefill and 140.37 tokens/s in zero-context generation, while unsloth's UD-Q4_K_XL climbs to 2,771.13 tokens/s and 143.07 tokens/s under the same conditions.

Differences remain small, but they show that choosing a quant is not just about theoretical quality. On a local server, the user may favor a faster model, especially if GPU or CPU memory limits options. In this game, the best version is often the one that fits on the machine without turning the computer into a decorative radiator.

A method designed to compare hard-to-compare things

The publication details its methodology, with perplexity, KL divergence, and Δp tests conducted on ik_llama.cpp, as well as speed benchmarks performed with distinct versions of llama.cpp and its fork. Parameters used are specified, with contexts of 512 or 32,768 tokens depending on cases, and thread and GPU settings adapted to the tested models.

The interesting point, for developers and teams deploying LLMs locally, is less to find an absolute "winner" than to understand where each format excels. Some quants favor compactness, others the stability of probability distributions, others still speed on CPU or hybrid configurations. This diversity explains why discussions around GGUF sometimes become very technical, and sometimes a bit tribally enthusiastic.

Key points

  • Qwen3-30B-A3B and Qwen3-235B-A22B were compared.
  • bartowski, unsloth, and ubergarm publish competing quants.
  • 235B's Q8_0 serves as the KLD baseline.
  • Speed tests likely exclude warmup.
  • The article cites ik_llama.cpp@9ba36270 for perplexity.
  • bartowski's Q4_K_M model remains close to others on 30B.

By the numbers

  • 233 GB — size of Qwen3-235B-A22B-Q8_0, base of the KLD comparison.
  • 107 GB — size of Qwen3-235B-A22B-mix-IQ3_K.
  • 15.194468 — wiki.test.raw perplexity of Qwen3-30B-A3B-Q4_K_M.
  • 2,771.13 tokens/s — measured PP speed for UD-Q4_K_XL in zero context.
  • 9.7 GB — size of Qwen3-30B-A3B-IQ2_M by ubergarm.

To read