Entropy as average information per symbol

Entropy is the average information per symbol for a distribution. For probabilities $p_i$,

$$H = \sum_i p_i (-\log_2 p_i).$$

Each term is probability times information; visualize bars of width $p_i$ and height $-\log_2 p_i$: entropy is their total area .

For the robot, plugging instruction probabilities into $H$ reproduces the ~1.75 bit average from the prefix-code story. The basic formula assumes i.i.d. symbols; English needs conditional models and entropy rates for dependent sequences .

Part 1 stops before the full cross-entropy training loss, but the ingredients are in place: predicted probabilities, realized bits, and optimizing cross-entropy as expected code length under your model .