Boolean Filtering

Analogy categoryAnalogies

Boolean filtering is the pandas idiom for selecting rows that satisfy a condition: a comparison like df['price'] > 100 produces a Series of True/False values (a 'boolean mask') with the same index as the original DataFrame, and passing that mask back into df[mask] or df.loc[mask] returns only the rows where the mask is True. This single pattern replaces explicit for loops and if statements for row selection, runs at NumPy's vectorized speed rather than Python's interpreted speed, and composes naturally with the rest of the pandas API, making it the standard way to answer questions like 'which orders exceeded $100?' or 'which customers are in California?'

🏏

Cricket analogy: Virat Kohli's scorer flags every ball where runs>4 as True, creating a boolean mask over the entire innings; feeding that mask back filters the scorecard to show only boundary balls, no manual scan through each delivery needed.

Combining Multiple Conditions

Multiple conditions are combined with the bitwise operators & (and), | (or), and ~ (not) — not Python's and/or/not keywords, which are designed for single boolean values and raise a ValueError ('truth value of a Series is ambiguous') when applied to an entire boolean Series. Each individual condition must also be wrapped in parentheses when combined, because Python's operator precedence evaluates & before comparison operators like >, so df['a'] > 1 & df['b'] < 2 parses incorrectly without parentheses around each comparison.

🏏

Cricket analogy: Selecting players who scored runs>50 AND wickets>2 needs both conditions parenthesized and joined with & across the whole squad list, not Python's 'and', which only works comparing two single scores, not two full team columns.

python

import pandas as pd

orders = pd.DataFrame({
    'customer': ['Ada', 'Grace', 'Alan', 'Katherine'],
    'state': ['CA', 'NY', 'CA', 'TX'],
    'total': [250.0, 45.0, 120.0, 310.0],
})

mask = (orders['state'] == 'CA') & (orders['total'] > 100)
print(orders[mask])
#   customer state  total
# 0      Ada    CA  250.0

print(orders[~(orders['state'] == 'CA')])   # negation: everyone NOT in CA
print(orders[orders['state'].isin(['CA', 'TX'])])  # membership test

isin, between, and query for Cleaner Syntax

.isin([...]) tests membership in a list of values in one call, replacing a chain of |-combined equality checks and reading far more clearly for categorical filters like state codes or product categories. .between(low, high) similarly replaces a two-sided range comparison ((s >= low) & (s <= high)) with a single readable call, and is inclusive of both endpoints by default. For complex, multi-condition filters, df.query('state == "CA" and total > 100') lets you write the condition as a string expression, which can be more readable than chained boolean operators and avoids the parenthesization pitfalls of &/|, though it requires column names to be valid Python identifiers.

🏏

Cricket analogy: .isin(['India','Australia','England']) replaces three chained OR checks to filter matches by those teams in one call, .between(20,50) grabs innings scores from 20 to 50 inclusively, and df.query() lets you write "team=='India' and score>50" as readable text.

A boolean mask is just a Series of True/False values aligned to the DataFrame's index — you can inspect it directly (print(mask)) or reuse it across multiple selections, e.g. filtering both a DataFrame and a related Series with the same mask.

Using Python's and/or instead of &/| between two boolean Series raises 'ValueError: The truth value of a Series is ambiguous', because Python's keywords expect a single unambiguous True/False, not an entire array of them — always use the bitwise operators with parentheses around each comparison for element-wise DataFrame/Series filtering.

A boolean mask is a True/False Series aligned to a DataFrame's index, produced by a comparison expression.
Passing a mask into df[mask] or df.loc[mask] returns only the rows where the mask is True.
Combine conditions with &, |, and ~ — never Python's and/or/not — and parenthesize each comparison.
.isin([...]) replaces chained equality-OR checks for membership filtering.
.between(low, high) is an inclusive, readable shortcut for two-sided range filters.
df.query('expr') offers a string-based alternative syntax for complex filters, avoiding operator-precedence pitfalls.

Practice what you learned

1. What type of object does a comparison like `df['price'] > 100` produce?

2. Why does `df[(df.a > 1) and (df.b < 2)]` raise a ValueError?

3. What does `df['state'].isin(['CA', 'TX'])` return?

4. Why must individual comparisons be wrapped in parentheses when combined with & or |, e.g. `(df.a > 1) & (df.b < 2)`?

5. What is a key advantage of df.query('state == "CA" and total > 100') over the equivalent chained boolean-mask syntax?

Was this page helpful?

Indexing with loc and iloc

MultiIndex Basics

Topics covered

#Python#PandasNumPyStudyNotes#DataScience#BooleanFiltering#Boolean#Filtering#Combining#Multiple#StudyNotes#SkillVeris

Boolean Filtering