Thinking Machines Trains Inkling 2 Small With 4-Bit Math End to End

Thinking Machines Lab has published details of how it trained Inkling 2 Small using very low-precision numbers. The lab calls it the largest model it knows of trained "end-to-end with a native 4-bit forward pass".

What "4-bit training" means here

Neural networks are trained with numbers stored at some precision. Fewer bits per number means less memory and faster maths, at the risk of less accurate training.

Thinking Machines used NVIDIA's NVFP4 format (about 4.5 bits per value) for the forward pass of the model's mixture-of-experts layers. It kept the backward pass, where gradients are computed, in the 8-bit MXFP8 format. The same setup ran from pretraining through the final reinforcement-learning stage.

Reported results

• Pretraining cost about 2% in compute efficiency compared with full 8-bit training, while running about 8% faster in wall-clock time on the lab's largest jobs.
• Reinforcement learning ran about 20% faster, with "no discernible difference in reward".
• Pretraining Inkling 2 Small took 68 hours on NVIDIA GB300 NVL72 racks. The lab says a similar-sized model previously took several weeks.
• Running the backward pass in 4-bit as well caused "very large hits" to efficiency, so the team didn't use it.

The kernels were developed in-house. The post doesn't say whether code or recipes will be released.

Why it matters for builders

Cheaper, faster training tends to show up later as cheaper and faster models. Low-precision formats are also how many open models are served today. This result suggests 4-bit numbers can be used much earlier in a model's life than many teams assumed.