Stochastic Gradient Descent
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A variant of gradient descent that updates model parameters after each sample or mini-batch instead of the full dataset, trading noise for speed.
What is Stochastic Gradient Descent?
Stochastic gradient descent (SGD) is an optimization algorithm that updates a model’s parameters after computing the gradient on a single training example or a small mini-batch, rather than the full dataset. It is the workhorse behind almost all deep learning training.
“Stochastic” refers to the randomness: each update is based on a random slice of the data, so it only approximates the true gradient. That approximation is what makes training at scale affordable.
How Stochastic Gradient Descent Works
Each epoch, the training data is shuffled and split into mini-batches. For every batch, the algorithm computes the loss, derives the gradient, and applies the update:
\( \theta_{t+1} = \theta_t - \eta \nabla L(\theta_t; x_i) \)
where \( x_i \) is the current batch and \( \eta \) the learning rate. Because each batch is a random sample, successive gradients disagree, and the parameters jitter on their way downhill. The jitter is not purely a defect: it helps the optimizer escape sharp, poorly generalizing minima.
In practice, “SGD” almost always means this mini-batch version, frequently extended with momentum. Adaptive methods such as the Adam optimizer build on the same idea with per-parameter step sizes.
Stochastic Gradient Descent vs Gradient Descent
The practical difference is how much data feeds each update: SGD steps after one sample or a mini-batch, while batch gradient descent processes the entire training set before moving once.
On a dataset of a million examples, batch gradient descent produces one exact update per pass; SGD with a batch size of 256 produces roughly 3,900 approximate ones. The approximations are noisy, but their sheer number wins.
| Criterion | Stochastic Gradient Descent | Gradient Descent (batch) |
|---|---|---|
| Data per update | One sample or a mini-batch | Entire training set |
| Cost per update | Small and nearly constant | One full pass over the data |
| Path to the minimum | Noisy, fluctuating | Smooth, deterministic |
| Best suited for | Large datasets, deep learning | Small datasets, convex problems |
Example of Stochastic Gradient Descent
Consider training an image classifier on ImageNet, whose training set holds about 1.28 million labeled images. Full-batch gradient descent would grant one parameter update per pass over all of them – a crippling pace.
SGD with a batch size of 256 instead delivers about 5,000 updates per epoch. Each one is computed from a random sliver of the data, yet after a few dozen epochs the network classifies images it has never seen. Every large vision and language model is trained this way.
Related AI terms: Gradient Descent · Mini-Batch · Learning Rate · Epoch · Adam Optimizer
Did you like the Stochastic Gradient Descent gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About