Optimizers and Their Differences: SGD, Momentum, Nesterov, RMSProp, Adam, Prodigy

Six gradient-based optimizers show up in almost every training script: SGD, Momentum, Nesterov, RMSProp, Adam, and Prodigy. They all chase downhill on a loss surface, but they get there at different speeds and sometimes by different routes. The update rules themselves look almost like abstract math, so here is the concrete version: watch them actually run on real loss surfaces, right here in your browser.

SGD (stochastic gradient descent) takes fixed steps straight down the steepest slope. Momentum learns to coast on that same slope, building up speed so it doesn't get stuck oscillating across narrow valleys. Nesterov is momentum with a lookahead, checking the slope a bit ahead before updating. RMSProp scales each step by the recent history of gradients in that direction, so parameters with larger gradients don't hog all the movement. Adam combines momentum with RMSProp's per-parameter scaling. Prodigy estimates its own step size as it goes, learning how big a step makes sense for the problem at hand, without needing a learning rate tuned in advance.

Below are two loss surfaces: first a simple bowl, then a rougher landscape with multiple wells. Click anywhere on either surface to watch all six optimizers race from that starting point. The differences you'll see are real, and they matter.

Watch what happens to plain SGD (black) on this tilted bowl: it zigzags back and forth across the slope, never quite going straight down. That's because the bowl is stretched, much steeper in one direction than the other, and a fixed step size overshoots the narrow direction while the wide direction still wants more. The adaptive methods barely react to the shape at all. They watch the local gradient history and scale each step to each direction separately, so they straighten out the path almost immediately. The cost is that they need more steps to settle: you'll see SGD reach the bottom first, then Momentum and Nesterov about halfway through, then RMSProp and Adam take longer still.

Each optimizer gets a fixed budget: SGD and Prodigy take 400 steps, Momentum and Nesterov take 200, and RMSProp and Adam take 150. These rates are tuned so that faster optimizers (in terms of raw iteration count) finish sooner, the slower ones take longer, and you can see both effects at once. The races run at 9 milliseconds per step, so when you see a path freeze mid-slide, that optimizer has burned through its budget while others are still rolling.

Adam is worth watching closely, especially the moment where it suddenly shifts its approach. Adam combines momentum (building speed) with per-parameter scaling (speeding up in directions with consistent gradients, slowing down in noisy ones). That shift from zigzagging to straight acceleration is Adam noticing the shape and adapting to it.

Prodigy stands apart: it estimates its own step size from how far it has moved since the start, rather than taking a fixed learning rate. That's why it looks hesitant at first, creeping along while others are already sprinting. It's still working out the scale of the problem. Once it figures out what a reasonable step size is, though, it moves decisively.

A rougher hill

The tilted bowl is kind. Real loss surfaces aren't. This one has several wells of different depths, some with steep walls, others so shallow that a rolling ball barely notices them. Click anywhere and watch where each optimizer lands. Different starting points lead to different wells, and different optimizers from the same starting point can land in entirely different places. One of these wells is the deepest, the true minimum, but not all optimizers find it from every starting point.

A gradient only tells you which way is down from where you're standing, never whether the ground gets lower somewhere else. Momentum can carry an optimizer straight through a shallow pit that would otherwise stop it, which is one reason it's almost always left on in practice. Sometimes that's an advantage: you overshoot a bad local minimum and find something better. Sometimes it's a trap: you overshoot the real bottom and have to climb back out. The rough surface makes all of this visible.

Watch where the deepest well actually is on this landscape, versus where most starting points end up. The true minimum isn't always the easiest one to reach. That deep well near the top right corner? From the bottom left, some optimizers miss it entirely and settle into a shallow pit instead, satisfied that they can't find a lower gradient direction from there.

The timing model here is different from Fig. 3. All six paths are drawn as a fraction of their own length over a shared 4-second window, so every optimizer finishes at exactly the same wall-clock moment. This reveals the real difference in work: Prodigy needs 6,000 steps to warm up its step size estimate on this flatter surface, compared to 400 for SGD, 200 for Momentum and Nesterov, and 150 each for RMSProp and Adam. That early stillness while others move is real, it's not a stall, it's Prodigy slowly building enough signal to trust its step-size estimate. Thousands of tiny exploratory moves, versus hundreds for the rest, all compressed into one animation window.

For more on what optimizers are and why they matter, see the regularization post, which covers related territory from a different angle.