ML Scale

ML Feature Scaling: Standardization vs Normalization

Feature scaling transforms numerical input features into a uniform numeric range, preventing features with large magnitudes (e.g. Income: $50,000$) from dominating features with small magnitudes (e.g. Age: $25$).

Scaling Techniques Comparison

flowchart TD
    A["Raw Numerical Features"] --> B{"Scaling Strategy"}
    B -- "Standardization (Z-Score)" --> C["x' = (x - mean) / std_dev
(Result: mean=0, std=1)"]
    B -- "Min-Max Normalization" --> D["x' = (x - min) / (max - min)
(Result: range [0.0, 1.0])"]

Scaling Methods

1. Standardization (Z-Score): Rescales data to have a mean of $0$ and standard deviation of $1$. Robust against outliers.
2. Min-Max Normalization: Rescales data to a fixed range $[0.0, 1.0]$. Sensitive to outliers.

Practical Code Example

import numpy as np
from typing import Tuple

def standardize_features(X: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
    """Standardize features: Z = (X - mean) / std."""
    mean = np.mean(X, axis=0)
    std = np.std(X, axis=0)
    X_scaled = (X - mean) / std
    return X_scaled, mean, std

def minmax_scale_features(X: np.ndarray) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
    """Min-Max scale features to range [0.0, 1.0]."""
    x_min = np.min(X, axis=0)
    x_max = np.max(X, axis=0)
    X_scaled = (X - x_min) / (x_max - x_min)
    return X_scaled, x_min, x_max

if __name__ == "__main__":
    # Features: [Age (years), Salary ($)]
    raw_data = np.array([
        [25, 45000],
        [30, 60000],
        [35, 75000],
        [50, 120000]
    ], dtype=np.float64)

    z_scaled, mean, std = standardize_features(raw_data)
    minmax_scaled, _, _ = minmax_scale_features(raw_data)

    print("--- Z-Score Standardized Features ---")
    print(np.round(z_scaled, 2))
    print(f"Scaled Mean: {np.round(np.mean(z_scaled, axis=0), 2)}")

    print("
--- Min-Max Scaled Features ---")
    print(np.round(minmax_scaled, 2))

Best Practices & Gotchas

• Fit on Train, Transform on Test: Calculate mean and std ONLY on training data, then apply those exact parameters to transform test data.
• Algorithms Requiring Scaling: Distance-based algorithms (KNN, SVM, K-Means) and Gradient Descent optimizations require feature scaling.
• Tree Models Exemption: Decision Trees and Random Forests are invariant to monotonic feature scaling and do NOT require feature scaling.

Self-Check Challenge

Perform Z-score standardization on a 1D NumPy array np.array([10, 20, 30, 40, 50]) and verify that its mean is $0.0$.