CSE3144 · Advanced Data Structures · Jul–Nov Semester 2026 · Lecture 28 of 36 · CO CSE3144.5

Bloom Filters and Probabilistic Searching

Every structure so far answered "is x in this set?" with a guaranteed yes or no, at the cost of storing every key. What if we could answer almost as reliably while storing none of the keys at all — accepting, in exchange, a small and precisely tunable chance of being wrong in one specific direction?

Dr. Manu ShrivastavaCourse Instructor · Consultation Fri 2–5 PM, LHC 308F
~110 minutesSession outcome: explain probabilistic membership testing methods
L28 · 00 — Agenda ~110 min

Trading certainty for space — on purpose, and by a known amount

Why probabilistic membership testing?

00–08

What exact structures cost when all you need is a fast, approximate "have I seen this before?"

Bloom filter definition & properties

08–16

A bit array standing in for a set — no false negatives, ever; false positives, sometimes; no deletion, ever.

Structure & operations — concept

16–22

k hash functions, m bits, two operations: INSERT and LOOKUP.

Animation: INSERT

22–34

Three words, one 12-bit filter, every hash computed by hand.

Animation: LOOKUP — including a real false positive

34–50

A true positive, two true negatives (one with a nuance), and a genuine false positive, all on the same filter.

Animation: why deletion breaks everything

50–62

Clearing one word's bits silently corrupts two other words already in the filter.

Sizing: the false-positive-rate math

62–74

The formulas, plus a fully worked numeric example.

Choice of hash functions

74–80

What makes a hash function suitable for this job — and what doesn't matter here that matters elsewhere.

Variants, applications & recap

80–110

Counting Bloom filters, real production systems, and what's next.

CSE3144 — Lecture 28
L28 · 01 — Why probabilistic membership testing? ~8 min

What "is x in this set?" costs when the answer must be exact

A concrete motivating problem

A signup page needs to check whether a username is already taken, out of hundreds of millions of registered usernames, on every keystroke. A linear scan is hopeless. A sorted structure (BST, B-tree) needs O(log n) comparisons AND must store every username somewhere to compare against. A hash table gets O(1) average lookup, but still stores every key (or a reference to it) — memory scales linearly with the number of usernames, forever.

The question this lecture answers

What if the application can tolerate an occasional wrong answer, PROVIDED it's always wrong in the SAME safe direction (only ever a false positive — see the table below), and provided we can dial the error rate as low as we like by spending a little more memory? That trade — a small, controllable, one-directional error rate in exchange for memory that doesn't need to store the keys at all — is exactly what a Bloom filter sells.

Preview of the deal

A Bloom filter can tell you "definitely not in the set" with full certainty, or "probably in the set" with a small, known chance of being wrong. It will never do the reverse — it will never claim something is absent when it was actually inserted. That asymmetry is the entire design.

Possible mistakeFilter saysTruthBloom filter?Cost in the username example
False positive“probably present”never insertedYes, rarely (rate is tunable)One extra database lookup; the final answer is still correct — safe
False negative“absent”was insertedNeverThe database is skipped and two users get the same name — a silent wrong result, unsafe

So every mistake points the same way — towards “present” — and that direction is safe because the application trusts every “no” and double-checks every “maybe” against the real data. A false positive costs only time; a false negative would cost correctness.

CSE3144 — Lecture 28 · Why probabilistic testing
L28 · 02 — Bloom filter definition & properties ~8 min

A bit array standing in for a set

Definition

A Bloom filter is a space-efficient, probabilistic data structure for testing set membership. It never stores the actual keys — only their hash-derived positions in a shared bit array. A fixed-size filter can represent a set with an arbitrarily large number of elements, at the cost of a rising false-positive rate as more elements are added.

The three properties that define it
  • Never a false negative. If an item was actually inserted, LOOKUP always correctly reports it as present.
  • Sometimes a false positive. LOOKUP can occasionally report an item as present when it was never inserted — purely by coincidence of shared bits.
  • No deletion. Clearing one item's bits can silently make OTHER, still-valid items look absent, because bits are shared between items. (Section 06 demonstrates exactly how this breaks, on a real example.)
CSE3144 — Lecture 28 · Definition & properties
L28 · 03 — Structure & operations — concept ~6 min

m bits, k hash functions, two operations

The structure

An empty Bloom filter is a bit array of m bits, all initially 0. It uses k independent hash functions h₁, h₂, …, hₖ, each mapping any input to a position in [0, m−1].

INSERT(x) and LOOKUP(x)
  • INSERT(x): compute h₁(x), h₂(x), …, hₖ(x); set the bit at EACH of these k positions to 1 (some may already be 1 — that's fine, just leave them).
  • LOOKUP(x): compute the same k positions; check each one. If ANY of them is 0, x is DEFINITELY NOT in the set. If ALL of them are 1, x is PROBABLY in the set.

Today's worked example uses m=12 bits and k=3 hash functions, small enough to compute and check every single bit by hand.

CSE3144 — Lecture 28 · Structure & operations
L28 · 04 — Animation: INSERT ~12 min

Insert "cat", "dog", "bat" into a 12-bit filter, k=3

Hash functions (a=1, b=2, …, z=26; value = sum of letter positions): h₁(w) = value mod 12, h₂(w) = (3·value + 1) mod 12, h₃(w) = (7·value + 2) mod 12. Every value below is computed, not asserted.

Interactive — INSERT 3 words
CSE3144 — Lecture 28 · INSERT
L28 · 05 — Animation: LOOKUP ~16 min

Four queries on the same filter: cat, rat, hat, mat

Only "cat" was ever inserted. Watch what happens to the other three — including one genuine false positive, computed from scratch, not asserted.

Interactive — LOOKUP 4 words
CSE3144 — Lecture 28 · LOOKUP
L28 · 06 — Animation: why deletion breaks everything ~12 min

Deleting "dog" silently corrupts "cat" and "bat"

"dog" occupies bits {2, 4, 7}. Bit 2 is shared with "cat"; bit 7 is shared with "bat". Watch what a naive delete (just clear dog's bits) does to both.

Interactive — attempt to delete "dog"
CSE3144 — Lecture 28 · Why deletion breaks
L28 · 07 — Sizing: the false-positive-rate math ~12 min

Three formulas, and one fully worked example

The formulas

Let m = bit-array size, k = number of hash functions, n = number of elements expected to be inserted. The false-positive probability is approximately:

p ≈ (1 − e−kn/m)k

Given a target n and p, the required bit-array size:

m ≈ −(n · ln p) / (ln 2)2

Given m and n, the optimal number of hash functions:

k ≈ (m/n) · ln 2
Worked example

A system expects n = 1000 usernames and wants at most p = 0.01 (1%) false positives.

ln(0.01) ≈ −4.6052
m ≈ −(1000 × −4.6052) / (0.6931)² ≈ 4605.2 / 0.4805 ≈ 9585 bits (≈1.2 KB)
k ≈ (9585/1000) × 0.6931 ≈ 6.64 → 7 hash functions

That's ≈9.6 bits per username — matching the well-known rule of thumb that roughly 10 bits per key gives about a 1% false-positive rate, independent of how long each username actually is.

CSE3144 — Lecture 28 · Sizing math
L28 · 08 — Choice of hash functions ~6 min

Independent, uniform, and fast — cryptographic strength is not the point

What actually matters here
  • The k hash functions should behave independently of each other — two functions that tend to collide together waste one of the k "checks."
  • Outputs should be uniformly distributed across [0, m−1] — a skewed hash sets some bits far more often than others, effectively shrinking m.
  • They should be fast: a Bloom filter's whole appeal is speed at scale, and k hash computations happen on every single INSERT and LOOKUP.
What doesn't matter here (unlike, say, password hashing)

Cryptographic guarantees (collision-resistance against a deliberate attacker, one-way-ness) are unnecessary overhead for this use case. Fast, well-distributed non-cryptographic hash function families are the standard choice in real implementations — the priority is raw speed and good statistical spread, not security.

CSE3144 — Lecture 28 · Hash function choice
L28 · 09 — Variants, applications & recap ~30 min

One fix for the delete problem, and where this runs in production

Counting Bloom filter

Replace each bit with a small counter. INSERT increments each of the k counters; DELETE decrements them. A position only reads as "unset" once its counter reaches 0 — so decrementing one item's counters can't wrongly erase another item's evidence, fixing exactly the Section 06 problem, at the cost of more memory per position than a single bit.

Where the trade-off is worth it

Any system checking "have I seen this before?" at very high volume, where an occasional false positive just means a slightly-more-expensive fallback check (not a correctness failure), and where storing every key is genuinely too expensive. A false positive costs one avoidable disk read or one avoidable database query — not a wrong answer to the end user.

Databases & storage engines

Systems like Cassandra, HBase, and BigTable-style stores keep a Bloom filter per on-disk file; a lookup first checks the filter and only touches disk if it says "probably present," skipping disk I/O entirely for keys that are genuinely absent.

Web & content platforms

Feed-ranking systems use Bloom filters to avoid re-showing content a user has already seen, without storing every user's full viewing history.

Browsers & security

Bloom filters have historically been used to give a fast local "is this URL possibly on a blocklist?" check before making a slower, authoritative lookup.

Lecture 29 — next

Priority Search Trees

CO CSE3144.5 — combining search-tree ordering with priority-queue extraction in one structure.

Homework — bring to Lecture 29
  • Using today's exact hash functions and the SAME 12-bit array (after inserting cat, dog, bat), check whether "pat" (p=16,a=1,t=20) produces a false positive, a true negative, or would need to be freshly inserted. Show every computed value.
  • A system expects n=10,000 elements and wants p=0.001 (0.1%) false positives. Compute m and k using today's formulas.
  • In 3–4 sentences: explain why a Counting Bloom filter's DELETE is safe when a plain Bloom filter's is not, tying your answer to today's dog/cat/bat example.
CSE3144 — Lecture 28

Questions?

Dr. Manu Shrivastava — LHC 308F — Friday 2:00–5:00 PM

Next: Lecture 29 — Priority Search Trees.