ML Polynomial Regression

ML Polynomial Regression: Non-Linear Curve Fitting

When relationships between features and targets are non-linear, polynomial regression models $y$ as an $n$-th degree polynomial: $y = c_0 + c_1 X + c_2 X^2 + \dots + c_n X^n$.

Linear vs Polynomial Regression

flowchart TD
    A["Data Points (Non-linear Curve)"] --> B{"Fit Model"}
    B -- "Degree 1 (Linear)" --> C["Underfitting (High Bias)"]
    B -- "Degree 2 or 3 (Polynomial)" --> D["Optimal Fit"]
    B -- "Degree 15 (High Degree)" --> E["Overfitting (High Variance)"]

Polynomial Degree Selection

• Degree 1: Simple line ($y = c_0 + c_1 X$).
• Degree 2: Quadratic curve ($y = c_0 + c_1 X + c_2 X^2$).
• Degree 3: Cubic curve ($y = c_0 + c_1 X + c_2 X^2 + c_3 X^3$).

Practical Code Example

import numpy as np
from typing import Tuple

def fit_polynomial_regression(X: np.ndarray, y: np.ndarray, degree: int = 2) -> np.poly1d:
    """Fit a polynomial regression curve of specified degree using NumPy."""
    # polyfit returns polynomial coefficients
    coefficients = np.polyfit(X, y, deg=degree)
    # Convert coefficients to executable polynomial object
    poly_model = np.poly1d(coefficients)
    return poly_model

if __name__ == "__main__":
    # Time of Day (Hours 1 to 12) vs Customer Traffic in Store
    hours = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
    traffic = np.array([10, 15, 25, 45, 80, 100, 95, 70, 40, 20, 15, 10])

    model_deg2 = fit_polynomial_regression(hours, traffic, degree=2)
    
    # Calculate R-squared score manually
    y_pred = model_deg2(hours)
    r2 = 1 - (np.sum((traffic - y_pred)**2) / np.sum((traffic - np.mean(traffic))**2))

    print(f"Polynomial Degree 2 Model:
{model_deg2}")
    print(f"R-Squared Score: {r2:.4f}")
    print(f"Predicted Traffic at Hour 5.5: {model_deg2(5.5):.1f} customers")

Best Practices & Gotchas

• Beware of Overfitting: Using high polynomial degrees (e.g. degree=10) creates models that fit training noise perfectly but fail on new test data.
• Use scikit-learn PolynomialFeatures: In professional pipelines, combine PolynomialFeatures(degree) with LinearRegression() inside a Pipeline.
• Extrapolation Risk: Never extrapolate polynomial models far beyond the min/max range of training data.

Self-Check Challenge

Fit a degree 2 polynomial to $X = [1, 2, 3, 4]$ and $y = [1, 4, 9, 16]$ using np.polyfit() and check if the coefficient for $X^2$ equals $1.0$.