▶Linear composition
Composing affine layers without activation is still only one affine map.
Nonlinear Activation Functions
The source of neural-network expressiveness
Without nonlinear activations, stacked layers collapse into one linear transformation. Sigmoid, tanh, and ReLU break that collapse in different ways.
watchone idea
→manipulateone example
→leave withone intuition
The essential idea: depth adds expressive power only when nonlinear transformations appear between linear layers.
One Concept · One Example
What to notice
- Sigmoid is smooth but saturates.
- Tanh is zero-centered but can still saturate.
- ReLU is efficient and piecewise linear.
Break the linear collapse
ReLU
A hinge at zero allows many neurons to assemble piecewise-linear curves.
Compare activation shapes
Switch activation functions and inspect both the output and derivative across the input range.
Nonlinear activations prevent linear collapse and balance expressiveness with trainable gradients.