Layer Normalization
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A technique that normalizes a neural network layer's activations across the features of each example, independent of batch size; standard in transformers.
What is Layer Normalization?
Layer normalization is a neural network technique that normalizes a layer’s activations across the features of each individual example, using that example’s own mean and variance. Proposed by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton in 2016, it is a standard component of every transformer, including BERT and GPT.
Because the statistics come from a single example, the technique works identically for a batch of one and a batch of a thousand.
How Layer Normalization Works
For each example, the layer computes the mean \( \mu \) and variance \( \sigma^2 \) of its activations across all features, then normalizes:
\( \hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \), followed by \( y = \gamma \hat{x} + \beta \)
The learned scale \( \gamma \) and shift \( \beta \) restore the layer’s freedom to represent any distribution. Normalized activations keep gradients in a healthy range, so deep stacks of layers train stably at higher learning rates.
Nothing in the computation references other examples. Training and inference run the exact same arithmetic, with no running statistics to maintain.
Layer Normalization vs Batch Normalization
The practical difference is the axis of normalization: layer normalization averages across the features of each example independently, while batch normalization averages each feature across all examples in the mini-batch.
Batch statistics degrade when batches are small and sit awkwardly on variable-length sequences, where positions across examples don’t align. Layer normalization avoids both issues, which is why it displaced batch normalization in transformers and recurrent networks, while batch normalization remains standard in convolutional vision models.
| Criterion | Layer Normalization | Batch Normalization |
|---|---|---|
| Normalizes across | The features of one example | The batch: one feature over all examples |
| Batch-size dependence | None | Degrades with small batches |
| Training vs inference | Identical | Different (running statistics at inference) |
| Typical home | Transformers, RNNs | CNNs, vision models |
Example of Layer Normalization
A chatbot generates its reply one token at a time: a batch of exactly one sequence. Batch normalization would have nothing meaningful to average over, but layer normalization needs nothing beyond the token being processed.
At each transformer layer, the vector representing the current token is normalized using the mean and variance of its own features, then scaled and shifted. This happens around every attention and feed-forward sublayer, hundreds of times per token, and it is one reason models with billions of parameters generate text without their activations drifting out of range.
Related AI terms: Batch Normalization · NLP Transformer · Attention Mechanism · BERT · Neural Network
Did you like the Layer Normalization gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About