AdamW

Extreme overtraining of small language models weakens performance after 20 billion tokens

An experiment published on Hugging Face shows that a model with 0.9 million parameters reached its peak performance when trained on 20 billion tokens, before gradually declining as training continued to 180 billion tokens. The results suggest a practical range of 22,000 to 30,000 tokens per parameter for very small models, with evaluation required before increasing the budget.