What is inference in AI?

Inference is when a trained model is actually used — it takes your prompt and generates an answer. Unlike training, which adjusts the model's weights, inference leaves them fixed. Every time you chat with an AI, you are running inference, and it is where most real-world AI cost is spent.

How does an LLM generate text?

One token at a time. The model predicts a probability distribution over the next token, picks one, appends it, and repeats — feeding its own output back in — until it produces a stop signal. This step-by-step, autoregressive process is why longer answers take longer and cost more.

What is temperature in AI?

Temperature controls randomness. At low temperature the model almost always picks the most likely next token, giving focused, repeatable answers. At higher temperature it samples more adventurously, giving varied, creative output. It scales the probabilities before the model chooses.

What is top-p (nucleus) sampling?

Top-p sampling limits the model's choice to the smallest set of most-likely tokens whose probabilities add up to p (say 0.9), then samples from just those. It keeps outputs coherent by cutting off the long tail of unlikely tokens while still allowing variety.

Why is AI inference expensive?

Because the model does substantial computation for every token it reads and writes, and it generates answers one token at a time. Serving millions of requests means running huge models constantly. Techniques like the KV cache, quantization and Mixture-of-Experts exist largely to make inference cheaper and faster.

AI Inference & Sampling Explained (2026)

AI inference & sampling: how a model generates answers

Training builds the model; inference is where it earns its keep. How an LLM writes token by token, what temperature and top-p really do, and why inference is where the cost lives.

Once a model has been trained, using it is called inference — and it is where nearly all real-world AI cost is spent, because training happens once but inference happens on every single request. Understanding it demystifies the settings (temperature, top-p) you have probably seen but maybe not understood.

How a model generates an answer

Generation is one token at a time

An LLM writes autoregressively: it predicts a probability for every possible next token, picks one, adds it to the text, and feeds the whole thing back in to predict the next — over and over until it emits a stop token. This is why a long answer takes longer and costs more: each token is its own forward pass through the network.

Temperature: the randomness dial

Before choosing a token, the model's raw scores pass through softmax to become probabilities, and temperature scales them first:

low temperature → sharp, deterministic | high temperature → flat, creative

At temperature near 0 the model almost always takes the single most-likely token — great for factual, repeatable tasks. Turn it up and it explores less-likely tokens — better for brainstorming and varied writing.

Top-p (nucleus) sampling

Pure randomness would let the model occasionally pick nonsense. Top-p sampling fixes this by only considering the smallest group of top tokens whose probabilities sum to p (e.g. 0.9), then sampling within that group. It keeps output coherent while still allowing variety — which is why top-p plus a modest temperature is a common default.

Making inference affordable

Serving big models cheaply is an engineering art. The KV cache avoids recomputing earlier tokens on every step; quantization stores weights at lower precision (say 8-bit or 4-bit) to shrink memory and speed things up; and Mixture-of-Experts activates only part of the model per token. Together these are why 2026 AI is fast and cheap enough to build products on.

Why this matters for your bill

Because you pay per token generated, output length and sampling settings directly affect cost and latency. A team that understands inference writes tighter prompts, caps output length, and picks the right temperature per task — turning an expensive prototype into an affordable product. That is the bridge from theory to the systems we teach in AI engineering.

Prefill vs decode: why the first token is slow

Inference has two phases. Prefill processes your entire prompt in one parallel pass — fast per token, but it is why a long prompt takes a moment before anything appears. Decode then generates the answer one token at a time, which is why replies stream. Understanding this split explains a lot of AI UX: a big document in the prompt delays the first word, while a long answer simply takes longer to stream out.

Quantization, in numbers

Models are trained in 16-bit precision, but for serving they are often quantized to 8-bit or even 4-bit. Each step roughly halves the memory and speeds things up, so a model that needed an expensive multi-GPU server can sometimes run on a single card — with only a small quality trade-off. Combined with the Mixture-of-Experts trick, this is much of why 2026 AI is cheap enough to build products on.