MLWhiz | AI Unwrapped

MLWhiz | AI Unwrapped

Share this post

MLWhiz | AI Unwrapped
MLWhiz | AI Unwrapped
The Simple Math behind 3 Decision Tree Splitting criterions
Copy link
Facebook
Email
Notes
More

The Simple Math behind 3 Decision Tree Splitting criterions

Rahul Agarwal's avatar
Rahul Agarwal
Nov 12, 2019
∙ Paid
1

Share this post

MLWhiz | AI Unwrapped
MLWhiz | AI Unwrapped
The Simple Math behind 3 Decision Tree Splitting criterions
Copy link
Facebook
Email
Notes
More
Share
The Simple Math behind 3 Decision Tree Splitting criterions

Decision Trees are great and are useful for a variety of tasks. They form the backbone of most of the best performing models in the industry like XGboost and Lightgbm.

But how do they work exactly? In fact, this is one of the most asked questions in ML/DS interviews.

We generally know they work in a stepwise manner and have a tree structure where we split a node using some feature on some criterion.

But how do these features get selected and how a particular threshold or value gets chosen for a feature?

In this post, I will talk about three of the main splitting criteria used in Decision trees and why they work. This is something that has been written about repeatedly but never really well enough.

1. Gini Impurity

According to Wikipedia,

Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset.

In simple terms, Gini impurity is the measure of impurity in a node. …

Keep reading with a 7-day free trial

Subscribe to MLWhiz | AI Unwrapped to keep reading this post and get 7 days of free access to the full post archives.

Already a paid subscriber? Sign in
© 2025 Rahul Agarwal
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More