ML Decision Tree

ML Decision Tree: Interpretable Rules and Overfitting Control

A Decision Tree predicts outcomes by asking a sequence of yes/no threshold questions on your features—just like a structured flowchart. Because every path from root to leaf translates into human-readable rules, Decision Trees are widely used in credit scoring, triage, and explainable ML.

Unlike linear models, Decision Trees handle non-linear curves, feature interactions, and unscaled numeric columns naturally without normalization.

Training and Inspecting a Decision Tree Classifier

flowchart TD
    A["credit_score <= 640?"] -- "Yes" --> B["High Risk"]
    A -- "No" --> C["income <= 45k?"]
    C -- "Yes" --> D["Review"]
    C -- "No" --> E["Approve"]

Always constrain tree growth with max_depth or min_samples_leaf; an unconstrained tree grows until every single training row sits in its own leaf and overfits badly.

from sklearn.tree import DecisionTreeClassifier, export_text
import numpy as np

# Features: [credit_score, debt_ratio_pct] -> Label: 1=approve, 0=decline
X = np.array([[580, 45], [620, 38], [690, 22], [740, 15], [710, 30]])
y = np.array([0, 0, 1, 1, 1])

tree = DecisionTreeClassifier(max_depth=2, random_state=42)
tree.fit(X, y)

print("Prediction for [700, 20]:", tree.predict([[700, 20]]).tolist())
print(export_text(tree, feature_names=["credit_score", "debt_ratio"]))

How Trees Pick the Best Split

At each node, DecisionTreeClassifier evaluates thresholds using Gini impurity (default) or entropy to pick the feature split that creates the purest child groups.

Common Mistakes to Avoid

• Leaving max_depth=None on noisy datasets: Deep trees memorize noise and lose accuracy on new data; tune max_depth (3–8) or min_samples_leaf (5–20).
• Passing raw string labels like "yes"/"no" as features: Scikit-Learn trees still require numeric feature inputs, so encode text columns first.
• Relying on a single deep tree in production: Single trees have high variance—small data tweaks reshape the whole tree; switch to RandomForestClassifier or GradientBoostingClassifier for higher accuracy.

Frequently Asked Questions

Do Decision Trees need feature scaling (StandardScaler)?

No. Splits test single-column thresholds (credit_score <= 650), so multiplying a column by 1,000 produces the exact same tree structure.

What is the difference between Gini impurity and Entropy?

Both measure class mixing at a node (0 means pure). Gini is slightly faster to compute because it avoids logarithms, and both usually yield nearly identical trees.

How do I inspect which features mattered most in a trained tree?

Read tree.feature_importances_, which sums to 1.0 across all features based on total impurity reduction.