A layer without a bend does nothing. One bend changes everything.

ML notes for admitted.dk

A single neuron with no hidden layer computes one thing: a weighted sum of its inputs, plus a bias, squashed through a sigmoid into a number between 0 and 1. Read as a probability, that number crosses 0.5 exactly where the weighted sum crosses zero — and the set of points where a weighted sum of two inputs equals zero is a straight line. However you tune the weights, that neuron's decision boundary is always some straight line through the input space. Nothing you feed it changes that.

The obvious fix looks like adding depth: put a hidden layer of units between the inputs and the output. But if those hidden units are themselves just weighted sums with no squashing function of their own, depth buys nothing. A weighted sum of weighted sums is still just a weighted sum — two matrix multiplications in a row collapse algebraically into one. A network with a hundred such "linear" hidden layers computes exactly the same family of functions as a network with none, and lands on exactly the same boundary.

What actually changes the picture is putting a nonlinearity — here, tanh — inside that hidden layer, squashing each hidden unit's sum before it's combined into the next one. That single nonlinear step is what stops the composition from collapsing back into an affine map, and it's what lets the same architecture bend its boundary into a curve, then another curve, until it can wrap around a shape as convoluted as the data demands.

The figure below trains a real network — 2 inputs, 8 hidden units, 1 output — live, by actual gradient descent, on a dataset a straight line cannot solve: points scattered inside a small disk against points scattered in a ring around it. Switch the architecture and press Play to watch training unfold on the network diagram (edge thickness is |weight|, blue is positive, orange is negative) and on the decision boundary next to it.

epoch 0 loss — accuracy —
Network
Decision boundary
positive weight negative weight inner disk, label 0 outer ring, label 1
Fig. 5 — a tiny network training live, right here, on a ring dataset. Same seed and weight initialization every time you reset, so the story is reproducible.
Try it: switch to "Hidden layer, linear" and press Play. It converges in well under a hundred epochs to a loss of 0.6888 and 57.9% accuracy — and switching to "No hidden layer" and pressing Play lands on that exact same loss, to four decimal places. Eight extra hidden units, zero extra expressive power.

For related territory — why some optimizers reach a minimum faster than others, and why L1 regularization can zero out a weight that L2 never quite can — see the optimizers post and the regularization post.