A comprehensive series explaining gradient boosting from first principles to advanced implementation details. Learn how modern gradient boosting libraries like XGBoost and LightGBM actually work under the hood.
Posts in this series
- What is Gradient Boosting? — Intuition for gradient boosting without heavy math
- Functional Gradient Descent — The mathematical foundation: gradient descent in function space
- Trees and the Split Gain Formula — Why trees are the standard weak learner and how we derive optimal splits
- Histogram-Based Split Finding — How XGBoost and LightGBM achieve O(bins) complexity
- Depth-Wise vs Leaf-Wise Growth — Two strategies for tree construction
- Gradient-Based Sampling (GOSS) — LightGBM’s sampling optimization
- EFB and Categorical Features — Exclusive Feature Bundling and categorical handling
- Regularization in Practice — Hyperparameter tuning for gradient boosting
- XGBoost vs LightGBM — A practical comparison