An experiment published on the Hugging Face blog indicates that training very small language models on enormous amounts of data may lead to declining rather than improving performance. In an experiment involving a model with 0.9 million parameters, the INT score peaked when the model was trained on 20 billion tokens—approximately 22,000 tokens per parameter—then gradually declined until reaching 3.31 at 180 billion tokens, a decrease of 27.3% from the peak.
The experiment was conducted by Banaxi of Banaxi-Tech and published as a community article on Hugging Face on August 12, 2026. The article focuses on a practical question facing developers of small models: how far can the token-to-parameter ratio be increased before additional training turns into harmful overtraining?
An experiment with an unprecedented training ratio
The experiment used an extremely small model consisting of six layers, a hidden dimension of 96, and a medium-sized SwiGLU unit of 380, in addition to GQA configured as 6Q/2KV, a vocabulary containing 384 tokens, and a context length of 8,000. The Muon optimizer was used for two-dimensional parameters with a maximum learning rate of 7e-2, while AdamW was used for the remaining parameters with a maximum learning rate of 4e-3.
The training processed 200 billion tokens from the FineWeb-HQ and Cosmopedia v2 datasets, equivalent to approximately 222,000 tokens per parameter, or about 220 times the ratio associated with the Chinchilla rule, which is approximately 20 tokens per parameter. The rationale behind the experiment was that the small vocabulary of 384 tokens carries less information in each token than a traditional vocabulary of 32,000 tokens, while also leaving a relatively larger number of parameters for the transformer layers.
Checkpoints were evaluated using the standard Open SLM suite, which includes ARC-Easy, ARC-Challenge, HellaSwag, and PIQA, along with ArithMark-2/3. The results were then combined into the INT Index.
Peak at 20 billion tokens followed by a continuous decline
The INT Index reached its highest value, 4.55, at the 20-billion-token checkpoint. It then declined as training continued, reaching 3.31 at 180 billion tokens. According to the article, the decline was continuous with some noise, and no later checkpoint recovered the peak level.
The individual test results reveal the same pattern. Comparing the 20-billion-token checkpoint with the 180-billion-token checkpoint, ARC-Easy increased from 26.98 to 28.32, but the scores declined on the other three tests: PIQA fell from 53.54 to 52.07, ARC-Challenge from 22.27 to 21.25, and HellaSwag from 29.01 to 28.06. The average fell from 32.95 to 32.43.
The experiment’s author did not explain the decline as a direct result of reducing the learning rate during the final 10% of training. A checkpoint at approximately 160 billion tokens, or 80% of training, was evaluated and recorded an average of 32.68, which was closer to the final result at 180 billion tokens than to the performance peak at 40 billion tokens, whose average was 33.44. Accordingly, the source argues that most of the damage had occurred before the learning-rate reduction stage, and that shortening the cosine decay schedule would not have solved the problem.
Comparison with the traditional ratio
The same experiment used a control checkpoint in which the model was trained on 18 million tokens, equivalent to a Chinchilla ratio of approximately 20 tokens per parameter. At that point, the model achieved an INT score of 1.53, with performance close to chance level on the tests: 26.64 on ARC-Easy, 49.78 on PIQA, 26.54 on ARC-Challenge, and 24.88 on HellaSwag.
This comparison provides a progression of results in the experiment:
- 20 tokens per parameter: 18 million tokens and an INT score of 1.53, indicating undertraining.
- 22,000 tokens per parameter: 20 billion tokens and an INT score of 4.55, the peak point.
- 200,000 tokens per parameter: 180 billion tokens and an INT score of 3.31, approximately 27% below the peak.
The proposed practical range
The article does not conclude that all high training ratios harm small models. It indicates that ratios of approximately 7,000, 15,000, and 22,000 tokens per parameter performed well in other community models, including TinyStories and small models on leaderboards with fewer than 3 million parameters. In this experiment, the failure point appeared only after going far beyond that range.
The results show a rapid upward path from 20 tokens per parameter to the peak, with the score increasing by approximately 0.05 per billion tokens between 10 billion and 20 billion tokens. By contrast, the decline was slower, at approximately 0.008 per billion tokens, but continued without interruption after the peak.
Based on this training run, the source estimates that the practical computational optimum for a Pico-class model, with approximately one million parameters, lies between 22,000 and 30,000 tokens per parameter. The practical recommendation is to begin training with a small budget within this range, then evaluate the model before deciding whether to extend training. Blindly increasing the ratio to 200,000 tokens per parameter could consume many hours of GPU operation while producing a model worse than the one achieved at approximately 20,000 tokens per parameter.