Follow the latest coverage, related explainers and connected technology stories.
An experiment published on Hugging Face shows that a model with 0.9 million parameters reached peak performance when trained on 20 billion tokens, before gradually declining as training continued to 180 billion tokens. The results suggest a practical range of 22,000 to 30,000 tokens per parameter for very small models, with evaluation necessary before increasing the budget.