ML Multiple Regression

ML Multiple Regression: Multi-Feature Linear Models

Multiple linear regression models a target variable ($y$) using two or more input feature variables ($x_1, x_2, \dots, x_k$): $y = eta_0 + eta_1 x_1 + eta_2 x_2 + \dots + eta_k x_k$.

Multi-Feature Matrix Formulation

$$\mathbf{y} = \mathbf{X}oldsymbol{eta} + oldsymbol{\epsilon}$$

Where $\mathbf{X}$ is an $N imes (k+1)$ matrix containing a column of ones for the intercept term.

Practical Code Example

import numpy as np

def solve_multiple_regression_ols(X: np.ndarray, y: np.ndarray) -> np.ndarray:
    """Solve multiple linear regression via Normal Equation: beta = (X^T X)^-1 X^T y."""
    # Add column of ones for intercept beta_0
    N = X.shape[0]
    X_b = np.c_[np.ones((N, 1)), X]
    
    # Normal Equation computation
    beta = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
    return beta

if __name__ == "__main__":
    # Features: [Engine Size (L), Weight (1000 lbs)]
    X_features = np.array([
        [1.6, 2.5],
        [2.0, 3.0],
        [2.5, 3.4],
        [3.0, 3.8],
        [3.5, 4.2]
    ])
    # Target: CO2 Emissions (g/km)
    y_emissions = np.array([120, 140, 165, 185, 210])

    coefficients = solve_multiple_regression_ols(X_features, y_emissions)

    print("--- Multiple Regression Coefficients ---")
    print(f" Intercept (Beta 0)   : {coefficients[0]:.2f}")
    print(f" Engine Size (Beta 1) : {coefficients[1]:.2f}")
    print(f" Weight (Beta 2)      : {coefficients[2]:.2f}")

    # Predict emissions for 2.4L Engine weighing 3.2k lbs
    new_car = np.array([1.0, 2.4, 3.2])
    predicted_co2 = new_car.dot(coefficients)
    print(f"
Predicted CO2 for [2.4L, 3.2k lbs]: {predicted_co2:.1f} g/km")

Best Practices & Gotchas

• Multicollinearity Warning: Highly correlated input features cause unstable coefficient estimates. Check Variance Inflation Factor (VIF) or remove redundant features.
• Feature Scaling: Scale input features prior to training when using regularization (Ridge / Lasso).
• Adjusted $R^2$: Use Adjusted $R^2$ instead of standard $R^2$ when comparing models with different numbers of features.

Self-Check Challenge

Write a function that predicts $y$ given new feature vector $[x_1, x_2]$ and learned coefficients $[eta_0, eta_1, eta_2]$.