Educational Cards
Learn from video content, text, and interactive tasks
Filters
Debugging gradients in practice
A wrong backward implementation can train for hours before NaNs appear. Standard debugging starts...
Handoff to the calculus-heavy walkthrough
This chapter kept the graph picture intuitive. The next chapter slows down and writes partial...
Bookkeeping partials on a small lattice of variables
Backprop at scale is still the chain rule; the hard part is bookkeeping . Assign a symbol to every...
Worked tiny graphs before code-size graphs
Before trusting a million-line framework, trace a three-node or five-node graph by hand. The...
JVPs and VJPs without_full Jacobians
A full Jacobian for a layer mapping mathbbR^4096tomathbbR^4096 has roughly sixteen million entries....
Log-space and stabilization patterns
Softmax exponentials can overflow when logits are large and underflow when logits are very...
Higher-order vs first-order in deep learning
SGD and Adam use first derivatives only. Newton methods incorporate curvature via the Hessian,...
Transition to scaled autoregressive modeling
The calculus hygiene on small graphs parallels the massive tensor programs inside language models:...
Autoregressive core loop: predict the next token
Large language models treat text as a sequence of discrete tokens (subwords or bytes). At each...
Scale: data, compute, and emergent capability patterns
Empirically, validation loss often improves predictably as model size, data, and compute grow....
Alignment after pretraining
Pre-training imitates internet-scale text, including toxicity, bias, and unsafe instructions....
Hallucination and overconfidence
Fluent prose can assert false specifics with high apparent confidence. Models optimize plausibility...