Gradient Descent
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
An optimization algorithm that trains machine learning models by repeatedly adjusting parameters in the direction that reduces the loss, step by step.
What is Gradient Descent?
Gradient descent is an optimization algorithm that trains a machine learning model by repeatedly adjusting its parameters in the direction that most reduces the loss function. Nearly every neural network in production was trained with some variant of it.
The gradient is the vector of partial derivatives of the loss with respect to each parameter. It points toward higher error, so gradient descent steps the opposite way.
How Gradient Descent Works
Picture a hiker descending a valley in thick fog. She cannot see the bottom, but she can feel the slope under her feet, so she steps in the steepest downhill direction, checks the slope again, and repeats until the ground flattens. The valley floor is the minimum of the loss; her position is the model’s current parameters.
Each training step computes the loss on the data, obtains the gradient (in neural networks, via backpropagation), and updates every parameter:
\( \theta_{t+1} = \theta_t - \eta \nabla L(\theta_t) \)
Here \( \theta \) holds the parameters, \( \nabla L \) is the gradient of the loss, and \( \eta \) is the learning rate, which sets the step size. Enough steps and the loss settles into a minimum – ideally a good one, since a loss landscape has many valleys.
Gradient Descent vs Stochastic Gradient Descent
The practical difference is how much data feeds each update: batch gradient descent computes the gradient over the entire training set before taking one step, while stochastic gradient descent updates after a single sample or a mini-batch.
Full-batch steps are exact but expensive; on millions of examples, one update costs a complete pass over the data. Stochastic updates are noisy approximations, but you get thousands of them for the same compute, and in practice the noisy path reaches good minima far sooner.
| Criterion | Gradient Descent (batch) | Stochastic Gradient Descent |
|---|---|---|
| Data per update | Entire training set | One sample or a mini-batch |
| Cost per update | One full pass over the data | Small and nearly constant |
| Path to the minimum | Smooth, deterministic | Noisy, fluctuating |
| Best suited for | Small datasets, convex problems | Large datasets, deep learning |
Example of Gradient Descent
Take a linear regression that predicts apartment prices from floor area, with two parameters: slope and intercept. Start both at zero; the model predicts every price as zero, and the loss is enormous.
Compute the gradient: it says the slope is far too low. Nudge both parameters downhill, recompute the loss, and repeat. After a few hundred iterations the line stops moving – the loss has reached its minimum, and the fitted line is the one that best matches the training data.
Related AI terms: Stochastic Gradient Descent · Learning Rate · Mini-Batch · Backpropagation · Adam Optimizer
Did you like the Gradient Descent gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About