Nothing learns on a flat line.

Every layer in a network picks its activation function from a short list: tanh, sigmoid, ReLU, SiLU. But the thing backpropagation actually consumes at each layer isn't the function itself — it's the function's derivative. So the two panels below plot value and slope on the same x axis, deliberately, because reading them apart is how this topic turns into a table of shapes instead of an explanation.

The chain rule multiplies exactly one f′(x) into the gradient for every layer it passes through on the way back to layer 1. That reframes the question. It isn't "what shape is this activation" so much as "what does this activation do to a number that has to survive N multiplications."

The four, each in one sentence: tanh(x) = (e^x − e^−x)/(e^x + e^−x), output (−1, 1), zero-centered. sigmoid σ(x) = 1/(1 + e^−x), output (0, 1), never negative. ReLU(x) = max(0, x), output [0, ∞), a kink rather than a curve. SiLU(x) = x·σ(x), also called swish, output [−0.2785, ∞), the only one of the four that dips below zero before it rises.

Fig. 6's top panel is f(x), the bottom is f′(x), and they share an x axis pixel-for-pixel. Drag across either one, or use the slider underneath, and watch both panels update together. It starts parked at x = −3 — that's not an arbitrary default.

The value panel is where each function's output range shows up as a cost, not just a shape. Sigmoid's outputs never go negative, which means every gradient flowing backward into a given weight vector from a sigmoid unit tends to share a sign — the weight updates end up correlated instead of independent, and descent zigzags more than it needs to. That's the smaller of sigmoid's two problems. ReLU and SiLU are both unbounded above, so activations can grow without limit as they pass through layers, which is the practical reason both are usually paired with normalization. SiLU's floor sits at −0.2785, reached at x = −1.2785, and nowhere lower.

The slope panel is where the real differences are. tanh peaks at exactly 1, at x = 0 — the best any of these four does at preserving a gradient unchanged. Sigmoid peaks at 0.25, also at x = 0, which means even at its single best input it throws away three quarters of whatever arrived. ReLU's slope is exactly 1 to the right of zero and exactly 0 to the left, with nothing in between. SiLU peaks above 1, near 1.0998 at x ≈ 2.4, and dips to about −0.0998 at x ≈ −2.4.

Now look at x = −3, where the figure starts. The four slopes read 0.0099 (tanh), 0.0452 (sigmoid), exactly 0 (ReLU), and −0.0881 (SiLU). Drag to x = −6 and they read 0.0000246, 0.00247, exactly 0, and −0.0123. Two different things are happening in that one comparison: tanh and sigmoid are decaying toward zero, while ReLU simply is zero — a flatter, more permanent kind of nothing. SiLU is the only one of the four still carrying a gradient large enough to matter, and its sign is negative, which turns out to matter later.

Ten layers of multiplication

Sigmoid's best case — 0.25, at x = 0 — raised to the tenth power is 9.54 × 10⁻⁷. tanh does worse than its own best case would suggest: at a perfectly ordinary pre-activation of x = 2, its slope is 0.0707, and 0.0707¹⁰ is 3.1 × 10⁻¹². That is not a pathological input. It's the arithmetic of stacking ten unremarkable layers, and it is closer to the real reason ReLU displaced sigmoid-family activations around 2011–2012 than any story about biological plausibility. Fig. 7 shows the other edge of that same arithmetic: SiLU's slope at x = 2 is 1.0908, so ten layers of it don't shrink the gradient at all — they multiply it by 2.38.

The half that is exactly zero

ReLU's left side isn't a small gradient, it's an exact one: 0. A unit whose pre-activation lands negative for every example in the training set receives precisely zero gradient on its incoming weights and its bias, on every step, forever — nothing in ordinary SGD can move it, because there's nothing there to follow. This is usually caused by an update large enough to push the bias well into negative territory, or a bias that drifts there gradually over training. Worth noting separately: ReLU also isn't differentiable at x = 0 itself, and frameworks simply pick a convention for what to return there (PyTorch returns 0, matching the left side).

SiLU's answer to the same problem is structural rather than incremental. It's smooth everywhere, and it's non-monotonic — it dips before it rises — which means its derivative is zero at exactly one point, x ≈ −1.2785, instead of across an entire half-line. A unit sitting at a negative pre-activation still gets a small, negative gradient there, and a negative gradient on a negative input pushes the unit's parameters back toward the region where it's active again. That's the dead-unit failure removed by construction, not just made less likely. The costs are real: an exponential per element instead of a single comparison, and a derivative that climbs past 1 over a band on the positive side, which means SiLU can amplify a gradient as readily as it preserves one.

None of this makes SiLU strictly better in practice. ReLU is still the cheap default, and it works fine at ordinary depths, because residual connections and normalization attack the vanishing-gradient problem from a completely different direction and mostly win regardless of which activation sits inside the block. The durable takeaway is narrower than "use this one": when you're comparing activation functions, the bottom panel is the one that tells you anything.