← All questions
Python & DataHard· Coding Challenges

Write a complete, production-quality sklearn pipeline —...

What separates a “production” pipeline from a notebook pipeline is that it removes every place a human could accidentally introduce leakage. Numerical and categorical columns need different preprocessing — imputation and scaling for one, imputation and encoding for the other — and ColumnTransformer applies each to the right columns inside a single object rather than juggling separate DataFrames by hand.

That preprocessor and the final estimator both go inside one Pipeline. The reason this matters structurally, not just for tidiness: calling pipeline.fit(X_train, y_train) guarantees every step — imputer, scaler, encoder, model — fits on training data only, and pipeline.predict(X_test) applies those exact learned parameters consistently, so leakage becomes structurally impossible rather than something you have to remember to avoid. Cross-validation runs on the training split to get a trustworthy performance estimate — mean and std, not a single number — before the held-out test set gets touched exactly once, at the end, for a final honest read. Saving the whole pipeline with joblib.dump() — preprocessing bundled with the model — means deployment can never apply the wrong transform to new data, because there’s only one artifact and it does everything.

Code playground (Python)