For years, the AI industry operated on a simple faith: make the model bigger, train it on more data, and it will get smarter. OpenAI's 2020 scaling laws paper gave that faith a mathematical backbone. Now, with diminishing returns creeping in, researchers are asking what comes after scale.

The paper was called “Scaling Laws for Neural Language Models.” It was published in January 2020 by a team at OpenAI. Its central finding sounds almost banal: model performance improves predictably as you increase model size, dataset size, and compute. Loss follows a power law. The relationship is smooth, reliable, and — crucially — it doesn't seem to flatten.

The implications were enormous. If scaling reliably improved performance, then the path to better AI was clear: spend more, compute more, scale more. No algorithmic breakthroughs required. No new architectures. Just more. The paper didn't just describe AI development — it became a strategic document that shaped billions of dollars of investment decisions.

What the Laws Actually Said

The 2020 paper found that performance — measured as loss on held-out text — scaled as a power law in three quantities: the number of model parameters (N), the size of the training dataset (D), and the total compute budget (C). Double any one of them, and you get a predictable improvement. The laws held across multiple orders of magnitude.

Crucially, the paper found that these three quantities could be traded off against each other — but not equally. For a fixed compute budget, there was an optimal allocation between model size and data size. The original 2020 paper suggested that model size was the dominant factor. This turned out to be wrong.

10×
Improvement in performance per decade of compute — a remarkably consistent empirical finding across model families and architectures
70B
Parameters in Chinchilla — DeepMind's 2022 model that outperformed GPT-3 (175B) by training on 4× more data with the same compute budget
~10T
Tokens of high-quality text estimated to exist on the internet — a finite number that may represent a hard ceiling on data scaling
Partner · Affiliate

Your product name here

One or two sentences on what this product does and why it fits a reader of this journal. Keep the tone of the surrounding writing — it reads better and converts better.

Learn more →

The Chinchilla Correction

In 2022, DeepMind published “Training Compute-Optimal Large Language Models” — the paper that introduced Chinchilla. Its finding was striking: GPT-3, with 175 billion parameters, had been severely undertrained. The optimal strategy for a given compute budget was to use a smaller model trained on dramatically more data.

Chinchilla — 70 billion parameters, trained on 1.4 trillion tokens — outperformed GPT-3 on nearly every benchmark while using significantly less compute. The revised scaling laws suggested that the ratio of training tokens to model parameters should be roughly 20:1. Most existing models had been nowhere near this.

Implication The race to build the largest model had been based on a miscalibration. The field had been spending enormous resources in the wrong direction. Chinchilla forced a recalibration — and raised an uncomfortable question: if data matters this much, what happens when we run out of high-quality data?

We may be approaching the point where the internet has been consumed. After that, the scaling laws become a different kind of problem entirely.

— Ilya Sutskever, at an industry conference, 2024

The Data Wall

The most pressing near-term limit on scaling is data. Estimates suggest there are roughly 10 trillion tokens of high-quality text available on the internet. At the training rates of current frontier models, the field could exhaust this supply within a few years. After that, what?

Several responses are in play. Synthetic data — using AI to generate training data for AI — has moved from a fringe idea to a mainstream strategy. Models like GPT-4 and Claude are now being used to generate vast quantities of synthetic training examples, reasoning chains, and curated Q&A pairs. The question is whether synthetic data introduces subtle distribution shifts that limit generalization.

Data curation and quality filtering has become a research area in itself. FineWeb, a recent dataset from Hugging Face, demonstrated that careful filtering of Common Crawl data could dramatically outperform larger, less curated datasets. Quality, it turns out, matters as much as quantity — perhaps more.

What Comes After Scale

The most interesting research in 2025 and 2026 has been focused on what the field calls “post-training” — the suite of techniques applied after the initial pretraining phase. Reinforcement learning from human feedback, constitutional AI, synthetic reasoning chains, test-time compute — these approaches have produced dramatic improvements without scaling the base model at all.

Direction 01
Test-Time Compute
Rather than training larger models, allocate more compute at inference time — allowing models to “think” longer before answering. OpenAI's o1 and o3 models demonstrated that extended chain-of-thought reasoning could unlock capabilities far beyond what raw model size would predict.
Direction 02
Architecture Innovation
The transformer is not the final word. Mamba, state space models, mixture-of-experts, and hybrid architectures are all exploring whether different computational structures can achieve better performance-to-compute ratios. The field that declared “attention is all you need” is quietly hedging its bets.

The era of naive scaling — bigger model, more data, better results — is over. What's replacing it is more complex, more diverse, and arguably more interesting. The constraints that scaling imposed forced the field to get smarter about what it was actually trying to do.

The scaling laws gave us a roadmap. The limits of those laws are giving us something more valuable: a reason to think harder about the destination.