Activation functions: Sigmoid, Tanh, ReLU, GeLU & Swish

Drag the input across all four at once. Step jumps instantly between 0 and 1 — no in-between. Sigmoid and tanh curve smoothly through the middle. ReLU — short for Rectified Linear Unit — does something different again: it's flat at exactly 0 for every negative input, then rises like a plain diagonal line.

An activation function turns a neuron's raw weighted sum into its output. Four of the most common:

• $x$ — the neuron's raw weighted sum, before activation.
• $\text{step}(x)$ — the perceptron's hard yes/no rule: 0 below zero, 1 at or above it.
• $\sigma(x)$ — the sigmoid, a smooth curve ranging over $(0,1)$.
• $\tanh(x)$ — a smooth curve ranging over $(-1,1)$.
• $\text{ReLU}(x)$ — zero for negative inputs, and equal to $x$ itself for positive ones.

1. Step has no usable slope

   Step is what the perceptron used — but it's flat everywhere except one point, so its slope is useless for learning by gradient.

2. Sigmoid and tanh are smooth substitutes

   Both curve smoothly through the middle, giving a real, usable slope everywhere.

3. ReLU trades smoothness for simplicity

   It throws out smoothness for something else: dead simple to compute, and its slope never shrinks to nothing on the positive side.

Watch what happens right around $x=0$. Step is discontinuous there. Sigmoid and tanh are at their steepest — right where they're most useful for learning. ReLU has a kink: flat on the left, a clean 45° line on the right.

Tanh isn't a fourth independent shape — it's sigmoid, rescaled: $\tanh(x) = 2\sigma(2x)-1$.

1. Evaluate sigmoid at 2x

   At $x=1$: $\sigma(2) = 1/(1+e^{-2}) \approx 0.881$.

2. Rescale it into tanh's range

   $2(0.881)-1 = 0.762$ — exactly $\tanh(1)$. Sigmoid ranges over $(0,1)$; tanh stretches and shifts that same curve to range over $(-1,1)$.