Guides

How to Handle Missing Values in pandas

By Ravindra Bagale, Data Science & Data Analytics trainer · LinkedIn · Published: 1 October 2026 · Last updated: 3 October 2026

Read this guide in: English · मराठी · हिंदी

Missing values show up as NaN. First detect them, then choose dropna or fillna, then re-check totals. Netflix-style example: watches 100, NaN, 90 → after fillna(0) the sum is 190.

detect → decide → dropna/fillna → verify → document

Language: English · मराठी · हिंदी

What do I need before this guide?

• A DataFrame with at least one messy column.
• Course: Types, missing, duplicates.

Before and after (look at the tables first)

Before: Show B is NaN. After: filled with 0; total watches 190.

Why missing values happen

1. Export left a blank cell.
2. API skipped a field.
3. Someone typed spaces instead of a number.
4. Join/merge created gaps.

Why you care: a chart can plot NaNs as gaps or drop points silently. Managers then argue with ghosts.

Netflix-style sample

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "title": ["Show A", "Show B", "Show C"],
    "watches": [100, np.nan, 90],
})
df.isna().sum()

What you see: watches missing count 1.

Detect (do this every time)

Vertical list:

1. df.isna().sum()
2. df.isna().mean() — share missing (optional).
3. df.info() — non-null counts vs rows.
4. Spot-check with a filter on isna().

Course messy sample context: Messy sample.

Decide: drop or fill?

dropna — when it is honest

1. The whole row is useless without that field.
2. Missingness is rare.
3. You document how many rows you removed.

clean = df.dropna(subset=["watches"])

fillna — when zero (or another value) is honest

1. For counts, 0 often means “no watches recorded”.
2. For categories, a label like "unknown" can be clearer than silence.
3. For some numeric EDA, median fill is a choice — say so in notes.

df["watches"] = df["watches"].fillna(0)
print(df["watches"].sum())  # 190

Shop sales blank cell

Suppose Pune sales blank:

1. If the row is a real sale with unknown amount — do not silently fill 0 without asking the business.
2. If the export uses blank for “no sale line” — maybe the row should be dropped.
3. Cleaning is a business conversation wearing Python clothes.

Rename / strip first (often)

Missing work sits next to messy headers:

1. Strip column names.
2. Standardise "N/A", "na", "" into real NaN.

Course: Rename, strip, standardise.

df["city"] = df["city"].astype(str).str.strip()
df["watches"] = pd.to_numeric(df["watches"], errors="coerce")

errors="coerce" turns junk into NaN — then you handle NaN on purpose.

Duplicates (neighbour topic)

1. Missing and duplicates are different diseases.
2. Same cleaning chapter covers both — treat them as separate decisions.
3. After fillna, check df.duplicated().sum().

What you see after a good clean

1. isna().sum() is zero on columns you claim are complete.
2. Totals match the rule you chose (190 with fill 0 on the sample).
3. A Markdown cell explains the rule in one sentence.

Outliers (do not confuse with NaN)

1. An outlier is a surprising number that exists.
2. NaN is a hole.
3. Course: Outliers — after missing values.