Session 19 · Phase 3: Python

Pandas — Filtering & Cleaning Missing/Duplicate Data

Real data is never clean. This session is the analyst's most-practiced skill: finding missing values, deciding to drop or fill them, removing duplicates, and filtering rows.

⏱ ~2 hrs 📚 Core content 🎯 Highest priority

Learning Objectives

0. The Sample DataFrame

Note the deliberate problems: missing values (NaN) and a duplicate row.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":    ["Aarav", "Priya", "Rahul", "Sneha", "Aarav"],
    "city":    ["Mumbai", "Delhi", None, "Bengaluru", "Mumbai"],
    "balance": [25000, 120000, 8000, None, 25000]
})

1. Detecting Missing Values

df.isnull()          # boolean DataFrame — True where missing
df.isnull().sum()    # count of missing values per column
df.info()            # also shows non-null counts

In Pandas, missing values are represented by NaN (or None).

2. dropna vs fillna — the Key Choice

When you find missing values, you have two main options: remove them or fill them.

dropna — remove rows/columns with missing values

df.dropna()                    # drop rows with ANY missing value
df.dropna(subset=["city"])     # drop only rows missing in "city"
df.dropna(axis=1)              # drop columns with missing values

fillna — replace missing values

df.fillna(0)                   # fill all missing with 0
df["city"].fillna("Unknown")   # fill missing city with "Unknown"
df["balance"].fillna(df["balance"].mean())   # fill with the column mean
⚠️
When to drop vs fill: Drop when missing rows are few and unimportant; fill when you need to keep the row. Filling with the mean distorts the distribution, so understand why data is missing before choosing. Never fill blindly.

3. Removing Duplicates

df.drop_duplicates()                    # remove fully duplicate rows
df.drop_duplicates(subset=["name"])     # dedupe based on "name" only
df.drop_duplicates(keep="last")         # keep the last occurrence instead of first
📝
Note: subset controls which columns define a duplicate. For example, deduping on name alone treats two rows with the same name as duplicates.

4. Boolean Indexing — Filtering Rows

The most common way to filter: build a boolean mask and pass it inside [].

df[df["balance"] > 50000]       # rows with balance over 50,000
df[df["city"] == "Mumbai"]      # rows in Mumbai

Combining conditions

Use & for AND and | for OR — and wrap each condition in parentheses.

# Mumbai AND balance over 20,000
df[(df["city"] == "Mumbai") & (df["balance"] > 20000)]

# Mumbai OR Delhi
df[(df["city"] == "Mumbai") | (df["city"] == "Delhi")]

isin — match a list of values

df[df["city"].isin(["Mumbai", "Delhi"])]
⚠️
Common mistake: using and/or instead of &/|. Python's and/or work on single booleans, not arrays — use &/| (with parentheses) for element-wise filtering.
📋 Stable content — Reviewed: August 2026

5. Interview Questions (with Model Answers)

The data-cleaning and filtering questions interviewers ask. Self-test before revealing.

IQ1. How do you check for missing values in a DataFrame?

Model answer: "I use df.isnull() (or isna()) to get a boolean DataFrame, and df.isnull().sum() to count missing values per column. df.info() also shows non-null counts."

IQ2. What's the difference between dropna and fillna?

Model answer: "dropna() removes rows or columns containing missing values; fillna() replaces them with a specified value. One discards data, the other preserves it."

IQ3. When would you drop vs fill missing data?

Model answer: "I'd drop when missing rows are few and not critical, and fill when I need to keep the row. The method depends on why data is missing — and I'd never fill with the mean blindly, because it distorts the distribution."

IQ4. How do you remove duplicate rows?

Model answer: "With df.drop_duplicates(). I can use subset to define which columns make a row a duplicate, and keep='last' to keep the last occurrence."

IQ5. How do you filter rows using boolean indexing?

Model answer: "I build a boolean mask and pass it inside brackets — like df[df['balance'] > 50000]. It returns only the rows where the condition is True."

IQ6. How do you combine multiple filter conditions?

Model answer: "With & for AND and | for OR, wrapping each condition in parentheses — df[(df['city']=='Mumbai') & (df['balance'] > 20000)]."

IQ7. What's the difference between == and isin for filtering?

Model answer: "== compares against one value; isin() matches against a list of values. So df[df['city'].isin(['Mumbai','Delhi'])] is cleaner than chaining two == conditions with |."

IQ8. How do you fill missing values with the column mean?

Model answer: "df['balance'].fillna(df['balance'].mean()) — I compute the column mean and use it to fill its own missing values."

IQ9. Why is handling missing data important before analysis?

Model answer: "Because missing values distort statistics and can break operations — means get skewed, aggregations silently ignore or error on NaN, and models fail. Cleaning first prevents wrong conclusions."

Hands-On Project: Clean a Messy DataFrame

Using the sample DataFrame (with missing values and a duplicate), complete the following.

Steps

  1. Count the missing values per column.
  2. Remove the duplicate row.
  3. Fill missing city values with "Unknown".
  4. Fill missing balance values with the column mean.
  5. Filter rows where balance is greater than 30,000.
  6. Filter rows in Mumbai or Delhi using isin().
View Solution / Walkthrough 
# 1. Missing counts
df.isnull().sum()

# 2. Remove duplicates
df = df.drop_duplicates()

# 3. Fill missing city
df["city"] = df["city"].fillna("Unknown")

# 4. Fill missing balance with mean
df["balance"] = df["balance"].fillna(df["balance"].mean())

# 5. Filter balance > 30000
high = df[df["balance"] > 30000]

# 6. Filter Mumbai or Delhi
subset = df[df["city"].isin(["Mumbai", "Delhi"])]

Key Takeaways

1

isnull().sum() is your first check for missing data.

2

dropna removes; fillna replaces — choose based on why data is missing.

3

drop_duplicates() with subset controls what counts as a duplicate.

4

Boolean indexing filters rows; use &/| (not and/or).

5

isin() matches a list of values cleanly.

Objective Questions — Test Your Understanding

Q1. How do you check for missing values in a DataFrame?

Q2. Which method removes rows with missing values?

Q3. Which method replaces missing values with a specified value?

Q4. How do you remove duplicate rows?

Q5. Which expression filters rows where balance is greater than 50,000?