Activation Function

Types of Activation Functions

1. Sigmoid: The sigmoid function squashes any input into a value between 0 and 1, which reads naturally as a probability in binary classification. Its weakness is the vanishing gradient problem: the curve flattens for large positive or negative inputs, gradients shrink toward zero, and deep networks learn slowly or stop learning.

   Formula: \( \sigma(x) = \frac{1}{1 + e^{-x}} \)

2. ReLU (Rectified Linear Unit): Passes positive inputs through unchanged and outputs zero for everything else. It is cheap to compute, and its gradient doesn’t shrink for positive inputs, which is why it became the default choice for deep networks.

   Formula: \( f(x) = \max(0, x) \)

3. Tanh (Hyperbolic Tangent): Squashes inputs into a range between -1 and 1. Because its output is centered on zero, it often trains faster than sigmoid, but it flattens at the extremes the same way, so it shares the vanishing gradient problem.

   Formula: \( \tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} \)

4. Softmax: Used in the output layer when the network must pick one class among several. It converts a vector of raw scores into probabilities that sum to 1, so the network’s answer reads as “72% cat, 20% dog, 8% fox.”

   Formula: \( \text{Softmax}(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}} \)

5. Leaky ReLU: A ReLU that lets a small gradient through for negative inputs instead of zeroing them out. This prevents “dead neurons” – neurons stuck at zero that never recover during training.

   Formula: \( f(x) = \max(\alpha x, x) \), where \( \alpha \) is a small constant such as 0.01.

Choosing an Activation Function

For hidden layers, start with ReLU; switch to Leaky ReLU if dead neurons show up. For the output layer, the task decides: sigmoid for a yes/no answer, softmax for picking one class among many, and no activation at all when the network predicts a raw number.

Example of an Activation Function

Take a network deciding whether an email is spam. Each neuron in the hidden layers applies ReLU, so the network can combine signals like “contains the word FREE” and “sender is unknown” in non-linear ways – one signal alone means little, both together mean a lot.

The final neuron applies sigmoid, turning the network’s raw score into a number like 0.93: a 93% probability the email is spam.

Related AI terms: Neural Network · Sigmoid Function · Softmax · Vanishing Gradient · Deep Learning

Did you like the Activation Function gist?

Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.