Educational Cards
Learn from video content, text, and interactive tasks
Filters
Classification heads and probabilistic outputs
The output layer presents ten competing scores, one per digit. For probabilistic training we often...
Pedagogical edge cases: saturation, dead ReLUs, initialization
The closing interview contrasts historical sigmoid activations, motivated by biological on/off...
The cost function as a high-dimensional surface
Chapter 1 showed how a forward pass turns pixels into digit guesses; this chapter asks how those...
Learning rate as step size: fragile knob
The learning rate eta controls how far each descent step moves in weight space. Too small and...
Stochasticity: minibatches approximate the full-data gradient
Exact gradient descent recomputes nabla C using every training example each step. For MNIST that is...
Local minima, saddles, and plateaus
Zero gradient marks a critical point , but not every critical point is a desirable resting place. A...
Momentum and adaptive methods (conceptual)
Plain SGD uses only the current batch gradient. Momentum maintains a velocity vector mathbfv that...
Bridge to automatic differentiation next
Gradient descent is useless without efficient partial derivatives. A naive finite-difference...
Backprop as structured chain rule on a DAG
Training needs partial C/partial w_i for every weight. Backpropagation organizes that bookkeeping...
Memoization: forward activations feed backward formulas
Automatic differentiation frameworks tape the forward computation: each op records inputs and...
Forward-mode vs reverse-mode: when each wins
Automatic differentiation has two standard modes. Forward mode seeds a perturbation on one input...
Structured graphs: weight sharing and modular layers
Production networks are not bare MLP chains. Convolutions reuse the same kernel at every spatial...