23.5 Filters That Drop Too Much

By Ravindra Bagale Last updated: 23 October 2026

Why. A wrong | / & or a missing parenthesis drops half of Maharashtra. Cancel rate becomes 0% because you filtered Delivered twice.

What.

Combine masks with & / | / ~, each clause in parentheses.
isin for city lists.
Prefer positive filters you can read aloud.
Log before / after counts every time.

How.

Filter without fear

Build named masks: is_pune, is_delivered.
Combine: mask = is_pune & is_delivered.
before, after = len(df), mask.sum().
Avoid and / or (Python scalars) on Series — use & / |.
Watch NaN in comparisons: df["city"] == "Pune" is False for NaN city.

Worked example A — Parentheses.

df = pd.DataFrame({
    "city": ["Pune", "Pune", "Nashik", "Nagpur", None],
    "status": ["Delivered", "Cancelled", "Delivered", "Delivered", "Delivered"],
    "amount": [100, 80, 90, 110, 70],
})
# Wrong idea with Python 'and' — will fail or mislead; use &
mask = (df["city"].isin(["Pune", "Nashik"])) & (df["status"] == "Delivered")
print("kept", df.loc[mask])
print("dropped", len(df) - mask.sum())

Worked example B — Accidental double filter.

delivered = df.loc[df["status"] == "Delivered"].copy()
# Later someone filters Delivered again on a column already gone — OK
# But if status was overwritten to NaN during cleaning, everything drops:
broken = delivered.copy()
broken["status"] = pd.NA
print("rows left", broken.loc[broken["status"] == "Delivered"].shape[0])

Worked example C — isin for six cities.

allowed = [
    "Pune", "Solapur", "Nashik", "Sambhaji Nagar", "Kolhapur", "Nagpur"
]
before = len(df)
df2 = df.loc[df["city"].isin(allowed)].copy()
print(before, "->", len(df2), "excluded", sorted(set(df["city"]) - set(allowed)))

None / NaN cities exit the allow-list — decide fill vs drop explicitly (Chapter 9).

Practice task

Keep Delivered orders in Kolhapur or Solapur with amount ≥ ₹100. Print before/after counts. Use parentheses.