Das Training eines Sprachmodells mit zusätzlichen Daten über den optimalen Punkt hinaus, was in diesem Experiment zu einer Verschlechterung der Leistung führt.
An experiment published on Hugging Face shows that a model with 0.9 million parameters reached its peak performance when trained on 20 billion tokens, before gradually declining as training continued to 180 billion tokens. The results suggest a practical range of 22,000 to 30,000 tokens per parameter for very small models, with evaluation required before increasing the budget.