40. Data Preprocessing for Machine Learning (Missing Values, Scaling, Encoding)
40.1 Why Preprocessing Before Models
Why. Real India tables have blank cells, ₹ and kg on different scales, and city names as text. Models like LogisticRegression and nets expect tidy numbers. Preprocessing is that tidy-up step.
What. Before fit, you often: fill or drop missing values, scale numeric columns, and encode categories as numbers. Do this using training statistics only — never peek at the test set to set the scaler.
How.
Name the three jobs
- Missing values → impute or drop.
- Scaling → put columns on a similar numeric range (e.g. StandardScaler).
- Encoding → turn “Pune” / “Nashik” into columns the model can read.
import pandas as pd
raw = pd.DataFrame({
"city": ["Pune", "Nashik", None, "Pune"],
"litres": [120.0, None, 90.0, 150.0],
"distance_km": [12, 35, 40, 8],
})
print(raw)
print("Problem: NaNs + text city + mixed scales.")
Ravindra Bagale's Tip
Preprocessing = cooking before recipe. Raw sabzi model madhe taku naka.