module 04

Naive Bayes

free pdf

A Comparison of Event Models for Naive Bayes Text Classification

Andrew McCallum & Kamal Nigam

People used two different models under the name naive Bayes for text: one only asks whether a word appears, the other counts how often. This paper tested them side by side and found counting usually wins, especially with large vocabularies.

the classifier in one line

Bayes' rule says: how likely is this class, given the evidence? It's how common the class is, times how well the evidence fits that class. Working out "how well the whole document fits" is impossible, because every document is unique. So naive Bayes makes one bold, naive assumption: each word (or feature) is independent of the others once you know the class.

c^=arg⁡max⁡c[log⁡P(c)+∑jlog⁡P(xj∣c)]\hat c = \arg\max_c \Big[\log P(c) + \sum_j \log P(x_j \mid c)\Big]

Training is just counting. Predicting is just adding.

two different "naive bayes" for text

People used the same name for two quite different models, often without saying which. McCallum and Nigam put them side by side.

Bernoulli model multinomial model
a document is a list of yes/no: is each vocabulary word present? a bag of word counts
does "chinese" ×3 count more than ×1? no yes
do missing words count? yes, each one adds evidence no

Their finding: the Bernoulli model is fine with small vocabularies but gets worse as the vocabulary grows, because thousands of missing words drown out the few present ones. The multinomial model keeps improving and usually wins. That's why "naive Bayes for text" means the multinomial model today.

for numbers: gaussian naive bayes

For continuous features, model each class as a bell-shaped cloud, with one mean and one spread per feature. The boundary between two clouds is straight if they have the same spread, and curved if they don't:

gaussian naive bayes draws curved boundaries

accuracy 0.960

● filled class · ○ hollow class · darker background = more confident it's the filled class

each class is modelled as a bell-shaped cloud. if both clouds have the same spread, the boundary between them is a straight line. if one class is tight and the other spread out, the boundary curls around the tight one.

how the code works

idea in naive_bayes.py
multinomial: count words per class, add alpha to every count MultinomialNB.fit
Bernoulli: count documents containing each word BernoulliNB.fit
missing words count too (Bernoulli) the (1 - X) term
everything in log space joint_log_likelihood, logsumexp
choosing a vocabulary mutual_information
continuous features GaussianNB

Three details:

  • Add instead of multiply. A 400-word document means multiplying 400 small numbers, which rounds to exactly 0. Adding their logarithms gives a normal number.
  • Smoothing. Without alpha, a word never seen in a class gets probability 0 and vetoes that class outright, however strong the rest of the evidence.
  • Zero spread. A feature that never changes would divide by zero, so a tiny floor is added to every spread.

what the demo checks

Run uv run python examples/04_naive_bayes.py.

  • Same documents, opposite answers. On the classic textbook example, the multinomial model says china (0.69, matching the hand calculation) and the Bernoulli model says japan (0.81). The first counts "chinese" three times; the second ignores repeats and is penalised for missing words.
  • With no smoothing, one unseen word sets a class's probability to exactly 0, overriding five strong words. With smoothing: 0.96.
  • Multiplying 400 probabilities gives 0.0; adding their logs gives −2110.
  • Copy one feature 20 times and the model counts its evidence 20 times: confidence rises from 0.88 to 0.99 while accuracy falls from 0.98 to 0.83.

when it goes wrong

  • Related features: the model double-counts them and becomes overconfident. It often ranks the classes right but gets the probabilities wrong.
  • No smoothing: one unseen word can veto a class.
  • Bernoulli with a big vocabulary: missing words take over.