ML Train and Test Split

ML Train and Test Split: Measuring Real Generalization

Splitting your dataset into training and testing subsets checks whether a model learned genuine patterns or simply memorized training rows. You train the model exclusively on the training set (typically 70%–80% of rows) and score it once on the unseen test set (20%–30%).

Evaluating a model on the same data it trained on creates false confidence—a memorized model can hit 99% training accuracy and fail completely on new customers.

Splitting Data Cleanly with train_test_split

flowchart LR
    A["Full Dataset (100%)"] --> B["Train Set (80%) -> .fit()"]
    A --> C["Test Set (20%) -> .score()"]

Pass random_state for reproducible splits, and pass stratify=y for classification problems so rare classes keep the same percentage in both train and test sets.

from sklearn.model_selection import train_test_split
import numpy as np

X = np.arange(20).reshape(10, 2)
y = np.array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print("Train shape:", X_train.shape, "Test labels:", y_test.tolist())

Time-Series Splitting Exception

Never shuffle time-ordered data like stock prices or daily traffic randomly—that leaks future days into past training. Use TimeSeriesSplit or split strictly by cutoff date.

Common Mistakes to Avoid

• Forgetting stratify=y on imbalanced classification: If only 5% of rows are fraud, a random split might leave almost zero fraud examples in the test set.
• Tuning hyperparameters repeatedly on the test set: Use cross-validation (cross_val_score) or a separate validation split for tuning so the test set stays untouched until final evaluation.
• Scaling or imputing before splitting: Always run train_test_split first, then fit preprocessing steps on X_train only.

Frequently Asked Questions

What train/test ratio should I use?

An 80/20 or 75/25 split works well for medium datasets. On very large datasets (millions of rows), even a 98/2 split gives tens of thousands of test rows.

What is K-Fold Cross-Validation?

K-Fold splits training data into k folds (commonly 5), training and validating k times on rotating folds to give a much steadier accuracy estimate than a single split.

Why set random_state=42?

Setting an integer seed guarantees your split is identical every time you re-run the script, making model comparisons fair.