What are parameters in an AI model?

Parameters are the model's learned settings — the numbers (weights) inside its layers that determine how it turns input into output. A model with 70 billion parameters has 70 billion of these numbers, adjusted during training. Broadly, more parameters mean more capacity to store patterns, though not always more usefulness.

What does 'billions of parameters' mean?

It is a rough measure of model size and capacity. Each parameter is one number the model tuned while learning. Counts grew from millions (early models) to hundreds of billions, because — up to a point — bigger models trained on more data get more capable. But size is only useful when matched with enough training data.

What are scaling laws?

Scaling laws are empirical rules describing how model performance improves predictably as you increase parameters, training data and compute together. They let researchers forecast how good a model will be before training it, and they showed that many models were undertrained for their size.

What is the Chinchilla result?

A 2022 DeepMind study found that, for a fixed compute budget, most large models had too many parameters and too little data. It recommended scaling parameters and training tokens roughly in proportion — about 20 tokens per parameter — producing smaller but better-trained ('compute-optimal') models.

What is a Mixture of Experts (MoE)?

Mixture of Experts is a design where a model has many specialised sub-networks ('experts') but only activates a few per token. This lets total parameter count grow huge while keeping the compute per token modest — a popular way in 2026 to get big-model quality at lower running cost.

AI Model Parameters & Scaling Laws Explained (2026)

Parameters & scaling laws: what “billions of parameters” means

Why did models get so big, and does bigger always mean better? A clear look at parameters, the scaling laws that predict quality, and the Mixture-of-Experts trick behind 2026 models.

You constantly hear models described by size — “8 billion parameters,” “405 billion parameters.” But what is a parameter, and why did the numbers explode? This piece explains model scale without the hype, and introduces the scaling laws that turned model-building from guesswork into forecasting. It builds on how the transformer works.

The model-size ladder (rough guide)

A parameter is a learned number

Inside a transformer are large matrices of numbers called weights (plus a few biases). Each of these numbers is a parameter. Training is the process of nudging all of them until the model predicts well. So “70 billion parameters” literally means 70 billion tuned numbers working together. More parameters give the model more room to store patterns and nuance.

Why size kept growing

Researchers found that performance improved smoothly and predictably as they increased three things together — parameters, training data, and compute. These scaling laws meant you could estimate a model's quality before spending millions to train it. The rough shape:

bigger model + more data + more compute → lower loss (better predictions)

Bigger is not the whole story: Chinchilla

In 2022, DeepMind's Chinchilla study delivered a twist: for a given compute budget, most models were too big and undertrained. The compute-optimal recipe was to balance parameters and data — roughly 20 training tokens per parameter. A smaller, better-fed model beat a larger, hungrier one. This is why 2026 has excellent “small” models, not just giant ones.

Mixture of Experts: big brain, small bill

To keep growing capacity without growing cost, many 2026 models use Mixture of Experts: dozens of expert sub-networks, but only a couple activated per token. Total parameters can be enormous while the compute spent on any single token stays modest — a clever way to get large-model quality at more affordable inference cost.

What it means in practice

Parameter count is a rough guide, not a quality guarantee. A well-trained, well-aligned smaller model often beats a larger, sloppier one on real tasks — which is exactly why evaluation matters more than headline size. How those parameters actually get their values is the subject of how LLMs are trained.

The Chinchilla rule, in numbers

DeepMind's compute-optimal finding is easy to remember: train on roughly 20 tokens per parameter. A 70-billion-parameter model therefore “wants” about 1.4 trillion tokens of training data to be well-fed. Frontier models now train on many trillions of tokens — effectively much of the public web, plus books and code — which is as much a data race as a parameter race.

Why smaller models won 2026

Two forces made compact models punch far above their size: Chinchilla-style better-trained models, and distillation from larger teachers. The result is that an 8–30B model in 2026 can match a 175B model from a couple of years earlier — which is why headline parameter count is a weak proxy for quality, and real evaluation matters more.