Data leakage and evaluation pitfalls

A model that looks excellent offline and then underperforms in production almost always has a leakage problem, not a modeling problem. The most common form is subtle: imputing missing values, scaling features, or selecting features using statistics computed over the entire dataset -- including the test set -- lets information from data the model should never have seen at training time leak into training. The fix is procedural discipline: every transformation that learns a statistic (a mean, a scaler, a vocabulary) must be fit on the training split alone, then applied unchanged to validation and test.

Time-series and any naturally-ordered data add a second, easier-to-miss leakage path: a random train/test split lets the model train on data from after the point it's meant to be predicting, which is not a small violation of good practice but a direct contradiction of what the model will face in production, where the future is never available at prediction time. Splitting chronologically instead of randomly is the fix, and it should be treated as non-negotiable for anything with a time axis, not an optional refinement.