Gradient Descent on a Loss Surface

Watch a ball roll down a 3D loss surface while the same trajectory zig-zags across the contour map beside it — and see how the learning rate decides whether it settles, crawls, bounces, or diverges.

This interactive gradient descent visualization shows how an optimizer finds the minimum of a loss function one step at a time. On the left, a ball rolls down a 3D loss surface; on the right, the same path is drawn across a top-down contour map, where the classic zig-zag of a poorly-tuned step size is easy to see.

At each iteration the update rule is $\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$: evaluate the gradient (the direction of steepest increase), take a step in the opposite direction scaled by the learning rate $\eta$, and repeat. The green arrow marks exactly where the next step will land, and it always meets the contour lines at right angles.

Use the animation to build intuition for the single most important hyperparameter in machine learning. A learning rate that is too small crawls; too large and the ball overshoots the valley and bounces, or diverges entirely. Switch between a convex bowl, a saddle between two pits, a narrow ravine, and a noisy surface to see how the landscape shape changes the story — including how a saddle point can stall training by looking "converged" when the gradient briefly vanishes without reaching a true minimum.

Play it, step through it iteration by iteration, or click either view to drop a new starting point and watch a different trajectory unfold.