ML Linear Regression

ML Linear Regression: Ordinary Least Squares & Line Fitting

Linear regression models the relationship between a scalar target ($y$) and one or more explanatory feature variables ($X$) by fitting a linear equation $y = mX + c$.

Linear Regression Architecture

flowchart LR
    X["Feature Input (X)"] --> Model["y_pred = slope * X + intercept"]
    Model --> Error["Residual Errors: (y_actual - y_pred)"]
    Error --> Optimization["Minimize Sum of Squared Errors (OLS)"]

Key Metrics

• Slope ($m$): Change in target per unit feature change.
• Intercept ($c$): Value of $y$ when $X = 0$.
• R-squared ($R^2$): Coefficient of determination ($0.0 \le R^2 \le 1.0$), representing variance explained by the model.

Practical Code Example

import numpy as np
from scipy import stats
from typing import Dict

def train_simple_linear_regression(X: np.ndarray, y: np.ndarray) -> Dict[str, float]:
    # Compute Ordinary Least Squares via SciPy
    slope, intercept, r_value, p_value, std_err = stats.linregress(X, y)
    r_squared = r_value ** 2

    return {
        "slope": float(slope),
        "intercept": float(intercept),
        "r_squared": float(r_squared),
        "p_value": float(p_value)
    }

if __name__ == "__main__":
    # Car Age (years) vs Price ($)
    car_age = np.array([1, 2, 3, 5, 7, 8])
    car_price = np.array([24000, 22000, 19500, 15000, 10500, 8500])

    results = train_simple_linear_regression(car_age, car_price)

    print("--- Linear Regression Model ---")
    print(f" Equation   : Price = ({results['slope']:.2f} * Age) + {results['intercept']:.2f}")
    print(f" R-Squared  : {results['r_squared']:.4f} ({results['r_squared']*100:.1f}% variance explained)")

    # Predict price for 4-year-old car
    predicted_price = (results['slope'] * 4) + results['intercept']
    print(f" Predicted Price (4yo Car): ${predicted_price:,.2f}")

Best Practices & Gotchas

• Check Linearity First: Ensure a linear relationship actually exists between variables before applying linear regression.
• High $R^2$ Does Not Imply Causation: Correlation and high $R^2$ scores indicate statistical association, not physical causality.
• Watch Out for Outliers: OLS minimizes squared errors, making it highly sensitive to extreme outliers.

Self-Check Challenge

Given $X = [1, 2, 3, 4, 5]$ and $y = [2, 4, 5, 4, 5]$, compute the slope and $R^2$ using scipy.stats.linregress().