4-bit training gains ground in large model training

Thinking Machines details end-to-end 4-bit model training on GB300. The challenge is not just speed: quantization error must be contained without breaking learning.

Thinking Machines Lab explains how it trained Inkling 2 Small, a language model, using 4-bit computation at the core on NVIDIA GB300 GPUs. The goal is simple to state but harder to achieve: speed up training and reduce memory usage without degrading model quality. According to the note published in November 2026, the method provides up to 8% pretraining time gain and 20% reinforcement gain compared to full MXFP8 training.

Pourquoi le 4 bits intéresse autant les laboratoires

Reducing calculation precision decreases data size and speeds up matrix multiplications but increases rounding errors. In large language models, these errors accumulate across layers and optimization steps. This is where the topic becomes concrete for AI players: fewer bits mean less memory, less bandwidth, and sometimes higher throughput. But also a higher risk of breaking learning if quantization is too aggressive.

In this approach, the team distinguishes three formats: BF16, MXFP8, and NVFP4. BF16 serves as a more precise reference; MXFP8 is intermediate; NVFP4, a 4-bit format, is the fastest on GB300 but also the trickiest to handle. The note states NVFP4 stores 3.6 times fewer bits than BF16 and can be up to 6 times faster on GB300. Speed thus comes at a price. As often, computation does not lie, it just charges later.

Le piège des zéros qui arrivent en groupe

The core problem lies in block quantization. NVFP4 uses blocks of 16 values, with a scaling factor that allows reconstructing numbers larger than what 4 bits directly allow. This system works well as long as values in a block remain close. When an outlier appears, it can pull the scale up and drop small values to zero. On Mixture of Experts layers, very costly in computation, this effect can quickly impact quality.

Thinking Machines says it tested wider NVFP4 quantization on 16×16 tiles, but the effect on compute efficiency was deemed too penalizing. The team thus chose a subtler compromise: use NVFP4 for some forward calculations, then MXFP8 for backward pass, with re-quantification and stochastic rounding. The latter process probabilistically rounds up or down to preserve the average value. Practically, it lets algebra have some leeway when it starts bouncing.

Un entraînement séparé entre aller, retour et communications

The strategy is not uniform throughout the pipeline. For MoE forward pass layers, Inkling 2 Small uses NVFP4. For backward pass, the note describes a return in MXFP8 with stochastic re-quantification of saved weights and activations. The idea is to avoid quantization bias, more problematic than simple random noise when propagated over multiple steps.

The third often underestimated point concerns GPU-to-GPU communications. Once experts are spread over multiple cards, tokens circulate via NVLink. NVFP4 computation can become faster than the interconnect network, creating a bottleneck. To reduce it, the team also quantifies some communications, then applies a random Hadamard transform before compressing data in S2F6, a fixed 8-bit format. Announced result: no measurable quality degradation during training on this specific component.

Ce que cela dit du marché des GPU et des LLM

Ultimately, this text shows a clear market evolution: the battle is no longer only about model size but about training pipeline finesse. Labs that can reduce precision without losing too much quality save machine time, energy, and bandwidth. GPU manufacturers see their advantage shift toward hardware capable of executing these rare formats, with dedicated instructions and specialized kernels.

The note also highlights a fundamental trend: 8 bits is no longer always the psychological boundary, and 4 bits enters production uses, at least for some steps. One caution remains: the method strongly depends on hardware, targeted layers, and type of inter-GPU communication. In other words, 4 bits is not a magic button. It is rather a tightrope act well paid in FLOPs.

Key points

  • Inkling 2 Small was trained on GB300 NVL72.
  • NVFP4 provides up to 6 times more speed on GB300.
  • Pretraining gains 8% in time, according to the note.
  • Reinforcement gains 20% compared to full MXFP8.
  • The model was pretrained in 68 hours on GB300 NVL72.
  • MoE experts concentrate a large part of the targeted gains.

By the numbers

  • 4 bits — precision used for some forward calculations, source Thinking Machines Lab, November 2026.
  • 16 values — size of an NVFP4 block for local scaling, source Thinking Machines Lab.
  • 8 % — pretraining time gain announced for the largest jobs.
  • 20 % — RL time gain compared to full MXFP8 training.
  • 68 hours — duration of Inkling 2 Small pretraining on GB300 NVL72.

To read