Introduction
Ever written code, run it, hit an error, tried to fix it (only God knows how), run it again… only to get another error, and repeat this cycle until it finally works? That’s exactly how Gradient Boosting works, except it uses calculus instead of blind fixes!
Hold up! This isn’t Gradient Boosting for Dummies, expect your brain to do some heavy lifting, just like that last deadly rep at the gym where you question all your life choices!
If you’re new to Gradient Boosting, check out the beginner-friendly resources at the reference, so you can enjoy the deep dive here.
What to expect from this blog?
This blog gives you the mathematical intuition behind Gradient Boosting, not just “it fits trees to residuals, trust me bro!” You’ll understand:
Why residuals? Why train on errors instead of actual labels?
What’s gradient about it? How is this really “gradient descent in function space”?
Why does it work? How adding trees iteratively minimizes loss
How to tune it? Practical hyperparameter tuning strategies that actually work
Blog Structure
The blog follows a gradual deep-dive approach, each section builds on the previous one:
Core Idea → Big picture in plain English
Algorithm → Step-by-step break down
Worked Example with Calculations → Find math hard? Jump here!
Key Concepts → Deep dive into why and how it really works
Practical Considerations → When to use, Practical tuning strategies for real-world use
Loss Functions → How Gradient Boosting works with different loss functions
Recap → Quick Intuition, cheat sheet (& few closing jokes ofc!)
I suggest using a multi-pass reading approach, which I follow when reading research papers or math-heavy blogs. First, read it quickly end-to-end without getting stuck, just to build a rough (≈ 40-50%) understanding of the big picture. Then revisit it more carefully to connect ideas, resolve confusion, and understand the details.
Ready to dive into this iterative error-correction machine? Let’s go!
Core Idea
Build an ensemble of weak learners (shallow decision trees) sequentially, where each new tree doesn’t predict the target y directly. Instead, it learns to predict the residual errors (mistakes) left by all previous trees combined.
The magic: Each tree corrects the mistakes of the previous models so far, and by adding these corrections together, the model gradually improves its predictions.
Algorithm
The algorithm here is specifically for regression with Mean Squared Error (MSE) loss.
Initialise first model F with mean
\(F_0 = \bar{y}\)For each decision tree model m=1 to M:
Calculate the residuals r to measure the error of the previous model (m-1)
\(r_i = y_i - F_{m-1}(x_i) \quad \text{for } i = 1, \ldots, n \text{ samples}\)Train a decision tree model h to learn and predict the current residuals r i.e. mistakes in the previous model, instead of the original labels y.
\(\text{Train decision tree } h_m(x) \text{ on } (x_i, r_i)\)Calculate the prediction value γ for each leaf j in tree m, which is the mean of all residuals r in that leaf region
\(\gamma_{jm} = \frac{1}{|R_{jm}|} \sum_{x_i \in R_{jm}} r_{im}\)where R is the leaf region (the subset of training examples that fall into leaf j of tree m)
The tree's prediction function h(x) returns the leaf value γ for whichever leaf the input x lands in:
\(h_m(x) = \sum_{j=1}^{J_m} \gamma_{jm} \cdot \mathbb{1}(x \in R_{jm})\)In simple terms: pass x through the tree → it lands in some leaf → return that leaf’s γ value.
Update the model by adding scaled correction to prediction of the previous model.
\(F_m = F_{m-1} + \nu \cdot h_m \)Here, learning rate v controls step size, while h(x) = γ for the leaf that x
falls into
Final model prediction will be initial guess (mean) + all scaled tree corrections
\(\hat{y}(x) = F_M(x) = F_0 + \nu \sum_{m=1}^{M} h_m(x)\)
Worked Example with Calculations
Let me make things easier now. let's see the algorithm in action with real numbers.
Substack bravely refuses to support Markdown, so to witness the content in its true, divine form, open The Holy Grail Example using the link!
Key Concepts and Intuitions
Now for the good stuff: understanding the ‘why’ behind algorithm design decisions. It’s time for few ‘aha!!’ moments, let’s dive into the why behind the what!
Initialization with Mean
Starting with F₀ = mean(y) is important:
Provides a simple baseline that minimizes squared error
First tree learns corrections from this baseline
The choice of F₀ doesn’t matter much after many iterations, but a good starting point helps convergence
Training on Residuals Instead of Labels
Key Insight: Each new tree doesn’t try to predict y directly. Instead, it predicts the errors (residuals) that the previous model made.
Tree 1 learns the residuals from F₀ (the mean)
Tree 2 learns the residuals from F₀ + Tree 1
Tree 3 learns the residuals from F₀ + Tree 1 + Tree 2
And so on...
This is called gradient descent in function space - we’re iteratively moving toward the true function by correcting errors.
Connection to Classic Gradient Descent:
Lets take regression using mean square error loss for both algorithm
In classic gradient descent, we update parameters:
Start with initial parameters θ₀ (initialized randomly or with zeros)
Train/fit model with current parameters (use θ to make predictions)
Compute loss and gradient with respect to parameters (∂Loss/∂θ = direction to move each weight)
Update: θ = θ - learning_rate × gradient (subtract gradient to move downhill toward lower loss)
Repeat until convergence
In gradient boosting, we update predictions (functions):
Start with mean as initial prediction function F₀ (baseline model : step 1)
Compute gradient of loss with respect to prediction function F (∂Loss/∂F = -(y - F) = - residuals, so negative gradient = residuals : step 2.a)
Train decision tree on residuals (fit tree h(x) to learn the residuals : step 2.b)
Update: F = F + learning_rate × correction_tree (add correction to move predictions downhill : step 2.e)
Repeat until convergence
Key insight: Both minimize loss by iteratively moving in the direction that reduces error. Instead of updating parameters (like in neural networks), here we update the prediction function itself by adding trees that point toward lower loss.
What’s equivalent?
Parameters θ ↔ Prediction function F (what we’re updating)
Gradient ∂Loss/∂θ ↔ Residuals ∂Loss/∂F (direction to move)
Train model with current θ ↔ Train tree on current residuals (what we fit)
Subtract gradient ↔ Add correction tree (how we update)
Here, for the update, we are adding instead of subtracting because the residual itself is a negative gradient.
Learning rate controls step size in both
Tree Prediction Function h(x)
Tree Prediction Function h(x) or say γ determines how much to correct the error of the previous models.
Understanding how h(x) = γ works:
When we pass an input x through the decision tree, it follows the splits (e.g., “is feature_1 < 5?”) and ends up in one specific leaf
Each leaf j has a prediction value γ, which is the average of all residuals of training samples that fell into that leaf
Why average? For MSE loss, the optimal leaf value is found by setting the gradient to zero: ∂(MSE)/∂γ = 0, which gives γ = mean of residuals in that leaf
Why residuals? We’re fitting the tree to predict errors (residuals), not the original targets. The tree learns “how much correction is needed”
So h(x) = γ means: “the prediction for x is whatever γ value is stored in the leaf where x lands”
Example: If x lands in leaf 3, and leaf 3 contains training samples with residuals [−2.0, −2.5, −3.0], then γ = mean(−2.0, −2.5, −3.0) = −2.5, so h(x) = −2.5.
This means the previous model was underpredicting by 2.5 on average for samples in this region, and our current tree has learned to add this correction.
Learning Rate (v)
The learning rate controls how much each tree contributes to the final model:
Higher v (e.g., v = 1.0):
The model learns faster by making larger corrections
Each tree has more influence on the final prediction
Risks overfitting to the training data
Fewer trees needed but less robust
Lower v (e.g., v = 0.1):
The model learns slowly by making smaller, more conservative corrections
Each tree contributes less, requiring more trees to reach good performance
Generally leads to better generalization on unseen data
Creates a smoother, more stable model
Typical values: 0.01 to 0.3
Trade-off: Lower learning rate with more trees (higher M) often gives best results
Number of Trees (M)
The number of trees controls model complexity:
Too few trees: Underfitting - the model hasn’t learned enough patterns
Too many trees: Overfitting - the model learns noise in training data
Best practice: Use early stopping with a validation set to find optimal M
Relationship with v: Smaller v requires larger M (more trees with smaller steps)
Tree Depth
Decision trees in gradient boosting are typically shallow (depth 2-4), as it’s faster to train and less prone to overfitting.
Additive Nature
Gradient boosting is an additive model, where Final prediction is a weighted sum of all tree outputs:
Each tree adds its contribution to improve predictions
We never modify previous trees, we only add new corrections
Practical Considerations
When to Use
High accuracy on tabular data: Best-in-class performance on structured datasets with complex non-linear relationships
Interpretability matters somewhat: Can examine feature importance, visualize individual trees, and track each tree’s contribution, though full interpretability becomes challenging with 100+ trees
When to NOT to Use
Sparse or high-dimensional data: Trees waste splits on noise when most features are zero or irrelevant; use linear models (handle sparsity via regularization) or deep learning (learn from high dimensions) instead
Online/incremental learning needed: Each new tree depends on residuals from all previous trees, requiring full retraining from scratch when new data arrives
Key Hyperparameters for Fine Tuning
max_depth: How many levels deep each tree can grow (deeper = more complex patterns)
min_samples_leaf: Minimum number of training examples required in each leaf node (higher = more regularization)
subsample: Fraction of data to use for training each tree (< 1.0 adds randomness, prevents overfitting)
learning_rate: How much each tree contributes to the final prediction (lower = slower, more careful learning)
n_estimators: How many sequential trees to build (more trees = more refinement)
Hyperparameters Tuning Guide
Model is overfitting: Decrease max_depth, increase min_samples_leaf, use subsample < 1.0
Model is underfitting: Increase max_depth, increase n_estimators (with lower learning_rate for best results)
Training is too slow: Increase learning_rate, decrease n_estimators, use subsample < 1.0
Want best accuracy: learning_rate = 0.01-0.05, high n_estimators with early stopping, tune max_depth carefully
Have small dataset: max_depth = 3-4, higher min_samples_leaf, use subsample < 1.0
Have large dataset: max_depth = 6-8, lower min_samples_leaf, subsample < 1.0 for speed
Loss Functions
The algorithm above is specifically for regression with Mean Squared Error (MSE) loss. Gradient Boosting can work with different loss functions by fitting trees to the negative gradient of that loss.
MSE is the most intuitive starting point since its gradient equals the residual.
What Changes with Different Loss Functions?
When you change the loss function, two things change:
What the tree fits to (the gradient) - this determines the direction of correction
Leaf values γ (the optimal step size) - this determines how much to correct
Key insight: The tree structure (splits) is built using the gradient, but the final leaf values are computed by minimizing the actual loss function within each leaf region.
Here are the most common ones:
MSE (L2 Loss):
Gradient: y − F (residual)
Leaf value γ: Mean of residuals in the leaf
When to use: Default choice, smooth and differentiable
Downside: Sensitive to outliers (large errors get squared)
MAE (L1 Loss):
Gradient: sign( y − F ) (just direction, not magnitude)
Leaf value γ: Median of residuals in the leaf
When to use: Robust to outliers, when you don’t want a few extreme errors to heavily influence the model as MAE treats all errors equally by magnitude
Downside: Not differentiable at zero; slower convergence
Why different leaf values?
Each loss function has a different optimal statistic: MSE is minimized by the mean and MAE by the median,. The tree splits guide where to correct, but the leaf values determine how much, tailored to each loss function’s geometry.
Recap
Quick Intuition
Gradient Boosting is basically that special friend who won’t let you forget your mistakes. ‘Remember when you predicted 100 but it was actually 50? Yeah, I built an entire tree about that. And that time you were off by 20? Got a tree for that too.’ Each tree corrects what all the previous trees missed, so by the end, you've fixed every single mistake layer by layer.
Why “Gradient”? Because at each step, we move in the direction of the negative gradient (residuals for MSE) to minimize the loss function, just like walking downhill to reach the valley.
Why “Boosting”? Because we boost weak learners (shallow trees) into a strong ensemble by sequentially correcting errors, where each tree adds a small correction to improve the overall prediction.
Key Formula Cheat Sheet
For Regression with MSE Loss
That’s a Wrap!
Congrats! You now understand an algorithm that iteratively corrects its mistakes, which is more self-aware than most people ;)
Resources
New to Gradient Boost? Check these out, they did amazing job!
Visit Rahul’s AI Lab for more interesting blogs, organized by topic and subtopic. It’s more than a blog index, it’s where curiosity turns into code, projects, and learning :)