Educational Cards
Learn from video content, text, and interactive tasks
Filters
Memorization, privacy, and copyright pressure points
Large models can memorize rare training sequences, including private or licensed text....
Bridge to transformer mechanisms
Language models before transformers often used RNNs : a hidden state updated serially along time,...
Self-attention mixes tokens via learned compatibilities
Chapter 5 framed language as next-token prediction; this chapter asks how a model can mix...
Positional encodings and length generalization
Attention scores depend on content vectors alone unless you tell the model where each token sits in...
Transformer block: attention + MLP + residuals + norm
A transformer block stacks two sublayers around a residual highway. First, multi-head...
Encoder vs decoder masking
Not every transformer attends the same way. Encoder blocks (BERT-style) use bidirectional...
Training stability at scale
Training billion-parameter transformers is as much engineering as theory. AdamW decouples weight...
Inference systems: KV-cache and batching
Training processes full sequences in parallel; inference generates one token at a time. Recomputing...
Linear projections produce Q, K, V
The previous chapter showed attention as a story; this one implements it as matrix multiplies....
Score matrix and softmax along keys
Stack all query-key scores into a matrix S in mathbbR^n times n with S_ij = Q_i cdot K_j / sqrtd_k....
Differentiable key-value lookup view
Attention can be read as a soft dictionary lookup . Keys index rows of a memory table; values store...
Tensor programs: batch × heads × sequence × dim
Production attention is a stack of GEMMs (general matrix multiplies) on GPU. Typical shapes: batch...