What is a transformer in AI?

A transformer is the neural-network architecture behind virtually all modern language models, introduced by Google in 2017. Its key idea is self-attention, which lets the model weigh how much every word in a sentence should influence every other word — all in parallel, which is why transformers scale so well.

What is attention in a transformer?

Attention is the mechanism that lets a model decide which other tokens are relevant to the one it is currently processing. For each token it computes a query, and compares it against the keys of all tokens, producing weights that blend their values together. In effect, each word gathers context from the words that matter to it.

What are query, key and value (Q, K, V)?

They are three learned projections of each token. The query asks 'what am I looking for?', the key advertises 'what do I contain?', and the value is 'what I pass on if selected'. The model matches queries to keys to decide how much of each value to mix in.

Why did transformers replace RNNs?

Earlier models (RNNs, LSTMs) processed text one step at a time, which was slow and struggled with long-range connections. Transformers process all tokens at once and let any token attend directly to any other, so they train far faster on modern hardware and capture long-distance relationships better.

What is positional encoding?

Because attention has no built-in sense of word order, transformers add positional information to each token so the model knows sequence — that 'dog bites man' differs from 'man bites dog'. This can be done with fixed sinusoidal patterns or learned position vectors.

Transformers & Attention Explained (2026)

Transformers & attention: how LLMs process language

One 2017 idea — self-attention — is the engine inside GPT, Claude and Gemini. Here is what attention does, in plain language with just enough of the maths to make it click.

Almost every model you have heard of — GPT, Claude, Gemini, Llama — is a transformer. The architecture was introduced in a 2017 paper with the memorable title “Attention Is All You Need,” and its central mechanism, self-attention, is worth understanding because it explains both why these models are so capable and why they behave the way they do. Once text is turned into tokens, this is what happens next.

How self-attention processes a token

The problem attention solves

Meaning depends on context. In “the bank of the river,” the word “bank” means something only because of “river.” Older models read left-to-right, one token at a time, and struggled to connect distant words. Attention lets every token look at every other token directly and decide, for itself, which ones matter.

Query, key, value

For each token the model produces three vectors: a query (what I am looking for), a key (what I offer), and a value (what I contribute). It compares one token's query against all keys to get attention weights, then blends the values accordingly. The canonical formula looks intimidating but says exactly that:

Attention(Q, K, V) = softmax( Q·Kᵀ / √d ) · V

The dot product Q·Kᵀ measures similarity between what a token wants and what others offer; dividing by √d keeps the numbers stable; softmax turns them into weights that sum to 1; and multiplying by V mixes the selected information. That is the whole trick.

Many heads, many layers

Models run attention many times in parallel — multi-head attention — so different heads can specialise (one tracks grammar, another tracks who-did-what). Stack dozens of these layers and the model builds up increasingly abstract representations of the text.

Order matters: positional encoding

Attention alone is order-blind — it would treat “dog bites man” and “man bites dog” the same. So transformers add positional encoding to each token, giving the model a sense of sequence. It is a small addition that makes language possible.

Why it changed everything

Because attention is parallel, transformers train efficiently on GPUs, and because any token can reach any other, they handle long-range meaning far better than their predecessors. That combination is what made today's scale possible — a scale we unpack in parameters and scaling laws.

The quadratic cost of attention

Attention's power has a price: every token is compared with every other token, so the compute grows with the square of the sequence length (n²). Double the context and you roughly quadruple the attention work — the reason long context was historically slow and expensive. Advances like FlashAttention and sparse or linear-attention variants exist largely to tame this cost, which is what made the 100K-plus token windows of 2026 practical.

Why modern LLMs are “decoder-only”

The original 2017 transformer had two halves — an encoder (reads input) and a decoder (writes output). Today's generative models like GPT and Claude keep only the decoder stack, trained purely to predict the next token. That single simplification, scaled up with more parameters and data, is most of what separates a 2017 research model from a 2026 assistant.