Attention Mechanism
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A neural network technique that lets a model weigh which parts of its input matter most for each prediction – the core idea behind the transformer and GPT.
What is an Attention Mechanism?
An attention mechanism lets a neural network weigh which parts of its input matter most for the prediction it is making, rather than treating every part equally.
It is the core of the transformer architecture. The 2017 paper “Attention Is All You Need” (Vaswani et al.) showed that attention alone, with no recurrence, was enough to build state-of-the-art language models – the design behind GPT and BERT.
How the Attention Mechanism Works
For each token in a sequence, the model computes a relevance score against every other token. The scores become weights that sum to one, and the token’s representation is updated as a weighted blend of the others: heavily influenced by relevant words, barely touched by irrelevant ones.
When a sequence attends to itself this way, it is called self-attention. Multi-head attention runs several attention patterns in parallel, so one head is free to track adjacent words while another tracks subject and verb or a pronoun and its referent.
This fixed a real limit of earlier recurrent networks, which squeezed an entire input through a running summary and lost track of the beginning of long sequences. Attention gives every token a direct connection to every other, however far apart.
Example of the Attention Mechanism
Consider: “The animal didn’t cross the street because it was too tired.” What does “it” refer to? Now change “tired” to “wide”, and the answer flips from the animal to the street.
A transformer resolves this with self-attention. Processing “it”, the model scores every other word in the sentence; with “tired” present, the attention weight concentrates on “animal”, and with “wide” it shifts to “street”.
No grammar rule was written for this. The model learned, from text alone, which words to consult when a pronoun is ambiguous – the behavior that makes transformers effective at translation, summarization, and dialogue.
Related AI terms: NLP Transformer · Natural Language Processing · Word Embedding · Sequence-to-Sequence · Recurrent Neural Network
Did you like the Attention Mechanism gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About