Educational Cards
Learn from video content, text, and interactive tasks
Filters
Masking bugs and leakage
Two mask families appear in almost every codebase. Padding masks zero out attention to padded...
Efficient attention approximations (concept map)
Full softmax attention costs O(n^2) per layer in sequence length. When n reaches tens of thousands,...
Parametric memory is distributed, not a tidy file cabinet
When a model answers "Paris is the capital of France," where did that fact live? Not in a single...
MLP neurons as key-value-ish nonlinear transforms
Transformer MLP blocks widen activations, apply a smooth gate (GeLU, SiLU), then project back to...
Catastrophic forgetting and continual learning pain
Fine-tuning updates weights to fit new data. Those same weights encoded prior abilities;...
Sparse autoencoders and feature directions
Sparse autoencoders (SAEs) train on internal activations to reconstruct them from a larger sparse...
Retrieval-augmented generation and tool use
Parametric weights cannot stay current with the live web. Retrieval-augmented generation (RAG)...
Knowledge editing and limits of surgical updates
Can you patch one fact without retraining? Methods like ROME and MEMIT attempt localized weight...
Diffusion intuition: destroy, then learn to undo
Diffusion models run a forward process that gradually adds noise until data looks almost Gaussian,...
Video is harder than still images (temporal coherence)
Independent per-frame generation produces flicker : object identity, lighting, and texture jump...
Engineering trade-offs: steps, guidance, distillation
Interactive systems cannot run 1000 denoising steps per click. Knowledge distillation trains a...
Safety stacks beyond the loss function
Minimizing denoising loss does not prevent misuse. Production stacks layer refusal policies ,...