How are large language models trained?

LLMs are trained by pretraining: the model is shown enormous amounts of text and repeatedly asked to predict the next token. Each wrong guess produces an error (loss), and the model adjusts its parameters slightly to be less wrong next time. Repeat this billions of times and it learns the patterns of language.

What is the training objective of an LLM?

Next-token prediction. Given a sequence of tokens, the model outputs a probability for every possible next token, and is scored on how much probability it gave to the actual next token. Minimising that error across a vast corpus is the entire pretraining objective.

What is a loss function?

A loss function measures how wrong the model's prediction is. LLMs use cross-entropy loss, which is large when the model assigned low probability to the correct token and near zero when it was confident and right. Training is the process of making this loss as small as possible.

What is gradient descent?

Gradient descent is the optimisation method that improves the model. It computes the gradient — the direction in which each parameter should move to reduce the loss — and takes a small step that way. Backpropagation is how those gradients are calculated efficiently across all layers.

How long does it take to train an LLM?

Pretraining a frontier model takes weeks to months on thousands of specialised GPUs or TPUs, costing millions. This is why very few organisations pretrain from scratch; most instead adapt an existing base model through fine-tuning, which is far cheaper.

How LLMs Are Trained: Loss, Gradients & Backprop (2026)

How LLMs are trained: loss, gradients & backprop

Underneath the magic is a surprisingly simple loop: predict the next token, measure the error, nudge the weights. Here is how that loop actually works — with just enough maths.

How does a pile of parameters become something that can write, reason and code? Through training — and the core idea is simpler than it sounds. This is the most mathematical of our fundamentals, but we will keep the maths to what genuinely helps.

The pretraining loop

The whole objective: predict the next token

Pretraining sets the model one repetitive task: given some text, predict the next token. It reads a colossal corpus and, at every position, outputs a probability for each possible next token. That is all. Everything the model “knows” is a side-effect of getting very good at this one game.

Measuring wrongness: cross-entropy loss

To improve, the model needs a score for how wrong it was. That score is the loss, and LLMs use cross-entropy:

loss = − log p(correct token)

If the model gave the correct token a high probability, −log p is near zero (good). If it gave it a tiny probability, the loss is large (bad). Averaged over billions of predictions, this single number tells training exactly how it is doing.

Improving: gradient descent

Now the model must adjust its parameters to lower the loss. It computes the gradient — for each parameter, which direction and how much to change it to reduce error — and takes a small step in that direction:

new weight = old weight − (learning rate × gradient)

The learning rate controls step size: too big and training is unstable, too small and it crawls. Repeat this step across the whole dataset many times and the loss steadily falls.

Backpropagation: gradients, efficiently

Backpropagation is the algorithm that computes all those gradients in one clever backward pass through the network, applying the chain rule from calculus. It is what makes training deep models practical rather than impossibly slow.

Why almost nobody pretrains from scratch

This loop, run on thousands of GPUs for weeks, costs millions — so only a handful of labs pretrain frontier models. Everyone else starts from a released base model and adapts it cheaply, which is exactly what fine-tuning, distillation and RLHF are for.

The scale behind “pretraining”

The numbers are staggering. A frontier model is trained on trillions of tokens, its compute measured in floating-point operations (FLOPs) that run for weeks on thousands of GPUs, at a cost commonly reported in the tens of millions of dollars. This is why only a handful of labs pretrain from scratch — and why the sensible path for almost every business is to adapt an existing base model rather than build one.

Reading a loss curve

Training is monitored by watching loss fall over time. Teams track it on held-out validation data too: if training loss keeps dropping while validation loss rises, the model is overfitting — memorising rather than generalising. A healthy run shows both curves falling together and levelling off, which is the signal that more compute would yield diminishing returns.