A Decision Tree is a powerful machine learning algorithm used for both classification and regression tasks. It operates by dividing the dataset into smaller parts, creating a tree-like structure. The initial step involves a single node examining the complete dataset. Subsequently, the tree divides the data at each level based on the optimal feature until the subsets are pure (comprising only one class) or meet predefined stopping criteria.

Impurity, which measures the mixture of classes in a dataset or subset, determines whether a dataset is pure (contains only one class) or impure (contains a mix of classes).

Gini Impurity:

Gini Impurity assesses the extent of misclassification and is instrumental in identifying the split that minimizes impurity the most.

• Gini impurity is faster to calculate than entropy as it doesn’t need logarithms.
• The Gini score is intuitive and easier to interpret in decision trees.

Entropy:

Similar to Gini, entropy measures disorder or uncertainty. Lower entropy indicates better splits.

Information Gain:

This metric quantifies the reduction in impurity following a split. Higher information gain signifies a better split.

Misclassification Error:

This metric evaluates the frequency of incorrect class predictions.

Classification Trees:

Classification trees come into play when the target is categorical (e.g., high-potential vs. low-interest customers). Key metrics in Classification Decision Trees include:

• Gini Index: Focuses on optimizing splits by minimizing class impurity.
• Information Gain: Measures the reduction in entropy post dataset split.

Regression Trees:

Regression trees are utilized for continuous target variables (e.g., predicting house prices). Key metrics in Regression Decision Trees are:

• Variance Reduction: Measures the drop in variance achieved after dataset split.
• Mean-Squared Error (MSE): Calculates the average squared difference between predicted and actual values.

Decision trees can handle both categorical and numerical data via techniques like: