Nvidia breakthrough gives 4-bit pretraining technique the accuracy of FP8
Nvidia breakthrough gives 4-bit pretraining technique the accuracy of FP8
arxiv.org
Pretraining Large Language Models with NVFP4
Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as show...
NVIDIA just trained a 12B-parameter language model on 10 trillion tokens entirely in 4-bit precision.
Here’s why this matters:
- NVFP4 delivers 2–3× faster math throughput and 50% less memory vs FP8
- Accuracy? Practically identical. (MMLU-Pro: FP8 = 62.62%, NVFP4 = 62.58%)
- Stability issues have been solved using Random Hadamard transforms, stochastic rounding, and 2D scaling
This is the first successful demonstration of large-scale 4-bit pretraining without losing accuracy.
The next generation of frontier models will be faster, cheaper, without compromise.