Hyperparameter Tuning
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The search for the hyperparameter values that give a model its best performance, run as an outer loop of training and scoring candidate configurations.
What is Hyperparameter Tuning?
Hyperparameter tuning is the search for the hyperparameter values, such as learning rate or network depth, that give a machine learning model its best performance on data it has not seen.
Training adjusts a model’s weights, but nothing inside training adjusts the settings that govern it. Tuning is the outer loop that does, and tuned settings regularly beat a library’s defaults by margins that matter in production.
How Hyperparameter Tuning Works
Every tuning method runs the same loop. Pick a candidate configuration, train a model with it, and score the result on a validation set or through cross-validation. Repeat with the next candidate and keep the configuration that scores highest.
The score must come from data held out of training. Scoring candidates on the training data rewards memorization, and scoring them on the final test set silently burns the one honest measurement you have.
Methods of Hyperparameter Tuning
- Grid search tries every combination of values from a predefined list. Exhaustive and simple, but expensive as the grid grows.
- Random search samples configurations at random from defined ranges. With the same budget it explores more distinct values per hyperparameter than a grid does.
- Bayesian optimization builds a probabilistic model of which regions of the search space look promising and spends its remaining trials there.
- Neural architecture search extends the search from training settings to the structure of the network itself.
Hyperparameter Tuning vs Model Optimization
The practical difference: model optimization adjusts a model’s weights inside a single training run, while hyperparameter tuning searches across many training runs for the best settings to launch them with.
| Hyperparameter tuning | Model optimization | |
|---|---|---|
| What changes | Settings like learning rate or depth | Weights and biases |
| What changes it | A search loop across training runs | Gradient descent within one run |
| Guided by | Score on validation data | Loss on training data |
| Scale | Tens to hundreds of trials | Thousands to millions of weight updates |
Example of Hyperparameter Tuning
An engineer trains an image classifier and gets 84% validation accuracy with the default learning rate of 0.001. She defines a search space over learning rate, batch size, and dropout, then runs 30 random configurations overnight.
The best trial pairs a learning rate of 0.01 with a smaller batch size and reaches 91%. Nothing about the data or the architecture changed; the settings did.
Related AI terms: Hyperparameter · Grid Search · Neural Architecture Search · Cross-Validation · Optimization
Did you like the Hyperparameter Tuning gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About