RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clipping

Understand

Natural gradient has been recently introduced to the field of boosting to enable the generic probabilistic predication capability.

• Natural gradient boosting shows promising performance improvements on small datasets due to better training dynamics, but it suffers from slow training speed overhead especially for large datasets.
• We present a replication study of NGBoost(Duan et al., 2019) training that carefully examines the impacts of key hyper-parameters under the circumstance of best-first decision tree learning.
• We find that with the regularization of leaf number clipping, the performance of NGBoost can be largely improved via a better choice of hyperparameters.