Transformers explained through one ambiguous sentence

Start with tokens and position

A tokenizer maps text into token IDs. A token may be a word, part of a word, punctuation, or another text fragment; the split depends on the tokenizer. Each ID maps to a learned vector. Those vectors are not the same as the document embeddings you will use for retrieval later, even though both are numerical representations.

Order also matters. “Customer refunded seller” and “Seller refunded customer” contain similar words but different relationships. Transformer designs include positional information through mechanisms such as positional embeddings or rotary position methods. The original paper's design is an important starting point, not a complete description of every current model.

Make queries, keys, and values less mysterious

For each position, learned projections produce a query, a key, and a value. Query–key comparisons give scores describing how strongly that position should combine information from other positions. Scaling and softmax turn scores into weights; a weighted sum of values produces the attention output.

The familiar expression is softmax(QKᵀ / √dₖ)V. Q, K, and V are matrices; dₖ is the key dimension. The arithmetic is precise, but the verbal analogy is only an aid: a query is not literally a natural-language question. Multiple attention heads allow different learned combinations before their results are mixed.

Walk through a tiny weighted sum

Suppose a demonstration head assigns weights 0.1, 0.7, and 0.2 to three value vectors [1,0], [0,1], and [1,1]. The combined output is [0.3,0.9]. This is invented arithmetic for illustration; it is not a measurement of attention in a real model.

weights = [0.1, 0.7, 0.2]
values = [(1, 0), (0, 1), (1, 1)]
result = [sum(w * v[j] for w, v in zip(weights, values))
          for j in range(2)]
print([round(x, 2) for x in result])  # [0.3, 0.9]

A transformer block also includes feed-forward transformations, residual connections, and normalization. Attention is central, but it is not the whole model. Nor should an attention visualization be treated as a complete causal explanation of an answer.

Understand the generation constraint

An encoder can use context on both sides of a token when its objective permits it. A causal decoder masks future positions so a token prediction cannot peek at later target tokens. An encoder-decoder architecture combines an encoded input with a generating decoder.

Training can process many known sequence positions in parallel under the appropriate mask. Autoregressive generation still produces new tokens sequentially, usually reusing cached state. This distinction corrects the common claim that transformers generate an entire answer in parallel. It also helps explain why long outputs can be slow even when reading the prompt is relatively efficient.

References and further reading

Use these primary references for deeper study and current API details. Examples in this lesson use fictional Northstar data.

• Research paper — 1706.03762