Batch Normalization
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A neural network technique that normalizes a layer's activations using the mini-batch's mean and variance, which stabilizes and speeds up training.
What is Batch Normalization?
Batch normalization is a neural network technique that normalizes a layer’s activations using the mean and variance of the current mini-batch, then rescales them with two learned parameters. Introduced by Sergey Ioffe and Christian Szegedy in 2015, it let networks train faster, tolerate higher learning rates, and grow much deeper.
The name states the mechanism: statistics are computed across the batch. Each feature is normalized using the values that feature takes over all examples in the batch.
How Batch Normalization Works
For each feature, the layer computes the batch mean \( \mu_B \) and variance \( \sigma_B^2 \), then normalizes:
\( \hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \), followed by \( y = \gamma \hat{x} + \beta \)
The learned scale \( \gamma \) and shift \( \beta \) let the network undo the normalization where that helps, so no representational power is lost. The original paper framed the benefit as reducing “internal covariate shift”; later research argues the real gain is a smoother loss landscape. Either way, the empirical effect holds.
There is a catch at inference time: a single input has no batch to take statistics from. The layer therefore keeps running averages of mean and variance during training and switches to those when the model serves predictions, so the network behaves differently in training and deployment.
Batch Normalization vs Layer Normalization
The practical difference is the axis of normalization: batch normalization averages each feature across the examples in a batch, while layer normalization averages across the features of each example independently.
That makes batch normalization batch-size dependent. With batches of 4 or 8 the statistics turn noisy and performance drops, and variable-length sequences make batch statistics awkward. Layer normalization has neither problem, which is why transformers use it instead.
| Criterion | Batch Normalization | Layer Normalization |
|---|---|---|
| Normalizes across | The batch: one feature over all examples | The features of one example |
| Batch-size dependence | Degrades with small batches | None |
| Training vs inference | Different (running statistics at inference) | Identical |
| Typical home | CNNs, vision models | Transformers, RNNs |
Example of Batch Normalization
Before 2015, very deep networks were hard to train: gradients shrank or blew up as they passed through dozens of layers. The ResNet architecture, published that year, placed batch normalization after every convolution.
The result was a 152-layer network that trained stably and won the ImageNet competition. Batch normalization kept every layer’s activations in a well-behaved range, so the vanishing gradient problem no longer set the depth limit.
Related AI terms: Layer Normalization · Mini-Batch · Vanishing Gradient · Convolutional Neural Network · Learning Rate
Did you like the Batch Normalization gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About