An experiment published on the Hugging Face blog indicates that training very small language models on massive amounts of data may lead to declining rather than improved performance. In an experiment involving a model with 0.9 million parameters, the INT score peaked when the model was trained on 20 billion tokens, equivalent to approximately 22,000 tokens per parameter, then gradually fell to 3.31 at 180 billion tokens, a decline of 27.3% from the peak.
The experiment was conducted by Banaxi of Banaxi-Tech and published as a community article on Hugging Face on August 12, 2026. The material focuses on a practical question facing developers of small models: how far can the token-to-parameter ratio be increased before additional training turns into harmful overtraining?
An Experiment with an Unprecedented Training Ratio
The experiment used an extremely small model consisting of six layers, a hidden dimension of 96, and a SwiGLU intermediate module with a size of 380, in addition to GQA configured at 6Q/2KV, a vocabulary of 384 tokens, and a context length of 8,000. The Muon optimizer was used for the two-dimensional parameters with a maximum learning rate of 7e-2, while AdamW was used for the remaining parameters with a maximum learning rate of 4e-3.
Training processed 200 billion tokens from the FineWeb-HQ and Cosmopedia v2 datasets, equivalent to approximately 222,000 tokens per parameter, or nearly 220 times the ratio associated with the Chinchilla rule, which is approximately 20 tokens per parameter. The rationale for the experiment was that the small vocabulary of 384 tokens carries less information in each token than a conventional vocabulary of 32,000 tokens, while also leaving a relatively larger number of parameters for the transformer layers.
Checkpoints were evaluated using the standard Open SLM suite, which includes ARC-Easy, ARC-Challenge, HellaSwag, and PIQA, along with ArithMark-2/3; the results were then combined into the INT Index.
Peak at 20 Billion Tokens, Followed by a Continuous Decline
The INT Index reached its highest value, 4.55, at the 20-billion-token point. It then declined as training continued, reaching 3.31 at 180 billion tokens. According to the material, the decline was continuous with some noise, and no later checkpoint regained the peak level.
The individual test results reveal the same pattern. Comparing the 20-billion-token point with the 180-billion-token point, the ARC-Easy score increased from 26.98 to 28.32, but the results declined on the other three tests: PIQA fell from 53.54 to 52.07, ARC-Challenge from 22.27 to 21.25, and HellaSwag from 29.01 to 28.06. The average fell from 32.95 to 32.43.
The experiment’s author did not attribute the decline directly to the learning-rate reduction during the final 10% of training. A checkpoint at approximately 160 billion tokens, or 80% of training, was evaluated and recorded an average of 32.68, which was closer to the final result at 180 billion tokens than to the performance peak at 40 billion tokens, whose average was 33.44. Accordingly, the source believes that most of the damage had occurred before the learning-rate reduction stage and that shortening the cosine decay schedule would not have solved the problem.
Comparison with the Conventional Ratio
The same experiment used a control point at which the model was trained on 18 million tokens, equivalent to a Chinchilla ratio of approximately 20 tokens per parameter. At that point, the model achieved an INT score of 1.53, with performance close to chance level on the tests: 26.64 on ARC-Easy, 49.78 on PIQA, 26.54 on ARC-Challenge, and 24.88 on HellaSwag.
This comparison provides a gradual range of results in the experiment:
- 20 tokens per parameter: 18 million tokens and an INT score of 1.53, indicating undertraining.
- 22,000 tokens per parameter: 20 billion tokens and an INT score of 4.55, the peak point.
- 200,000 tokens per parameter: 180 billion tokens and an INT score of 3.31, approximately 27% below the peak.
Proposed Practical Range
The material does not conclude that all high training ratios harm small models. It notes that ratios of approximately 7,000, 15,000, and 22,000 tokens per parameter performed well in other community models, including TinyStories and small models on leaderboards with fewer than 3 million parameters. In this experiment, the failure point appeared only after greatly exceeding that range.
The results show a rapid rise from 20 tokens per parameter to the peak, with the score increasing by approximately 0.05 per billion tokens between 10 billion and 20 billion tokens. By contrast, the decline was slower, at approximately 0.008 per billion tokens, but continued without interruption after the peak.
Based on this training run, the source estimates that the practical compute-optimal range for a Pico-class model with approximately one million parameters lies between 22,000 and 30,000 tokens per parameter. The practical recommendation is to begin training with a small budget within this range, then evaluate the model before deciding whether to extend training. Blindly increasing the ratio to 200,000 tokens per parameter may consume many hours of GPU operation to produce a model worse than the one achieved at approximately 20,000 tokens per parameter.