Positional Encoding and Context

Why position matters

The words dog bites man and man bites dog contain the same words but express different events. A model needs information about order. Position mechanisms complement attention rather than replacing it.

Common approaches

Sinusoidal encodings use fixed sine/cosine patterns; learned absolute encodings use learned position vectors. Relative methods express offsets or biases. Rotary positional embeddings (RoPE) rotate query/key components; they are not simply vectors added to token embeddings.

Sinusoidal formula

PE(pos,2i) = sin(pos / 10000^(2i/d)); PE(pos,2i+1) = cos(pos / 10000^(2i/d)). Here pos is the token position and d is the encoding width. Different frequency pairs provide distinguishable patterns.

Managing long contexts

Count input and output against the model and serving configuration. Over-limit requests may fail or be truncated by the application; older text is not always silently ignored. Long-context capacity does not guarantee reliable retrieval of every detail. Retrieval, summaries, and chunking can help.

Worked example

The example creates four-dimensional sinusoidal vectors for three positions, then computes a small token budget.

Download lesson 07 Python example

from math import sin, cos

def position_vector(pos, d=4):
    result = []
    for i in range(d//2):
        angle = pos / (10000 ** (2*i/d))
        result.extend([round(sin(angle), 3), round(cos(angle), 3)])
    return result

for pos in range(3):
    print(pos, position_vector(pos))
print("Remaining:", 4096 - 256 - 2000 - 512)

Expected output

0 [0.0, 1.0, 0.0, 1.0]
1 [0.841, 0.54, 0.01, 1.0]
2 [0.909, -0.416, 0.02, 1.0]
Remaining: 1328

Position 0 has alternating 0 and 1. Later positions change each frequency pair differently. After reserving instructions, retrieved evidence, and output, 1,328 tokens remain for other input.

Practice and self-check

Common mistake

FlashAttention improves attention execution and memory use; it does not by itself create unlimited usable context or remove all long-context quality problems.

Student tasks

1. Generate positions 3 and 4 and compare both frequency pairs.
2. Explain why RoPE should not be described as merely adding a position vector.
3. Plan a 2,048-token request with a 400-token output reserve and list your allocations.