Chapter 25 · Text Analytics: Bag of Words and Beyond

1. From Words to Numbers

Text is the most common data in the world. But models need numbers, and text is not numbers. The dataset is Restaurant_Reviews.tsv. It holds 1,000 reviews, one per row, each labelled Liked = 1 or 0. The classes are perfectly balanced: 500/500. The question: will a new review praise us or criticise us? This is sentiment classification. It is supervised learning, like Part III. But the features are raw sentences.

The plan of this chapter is the plan of all classical NLP. Turn every review into a fixed-length vector of numbers. Then reuse every model already built in this course. Sections 2–4 build the ideas: the map of the field, and the bag-of-words intuition on a real email. Sections 5–8 then run the pipeline on the real reviews:

2. The Map of NLP — Where This Chapter Lives

Before the code, let us draw one picture. The picture organises the whole field. It also shows what this course will not build. Two circles overlap:

• Natural language processing (left circle): any model that works with human language. No neural network is needed.
• Deep learning (right circle): any neural network, applied to anything: images, numbers, language.
• Deep NLP (the overlap): the intersection of the two. Neural networks that process language. Embeddings, LSTMs and transformers live here.
• Sequence-to-sequence (innermost): the centre of deep NLP. These models read a sequence and write one. Translation engines, chatbots and caption generators.

Why draw the map? It sets honest expectations. The famous NLP applications — machine translation, image captioning, speech recognition, question answering, summarisation, chatbots — are sequence-to-sequence problems. They sit at the frontier of the field. This course builds the classical corner: sentiment classification. A bag of numbers plus a classifier from Chapters 11–15 already delivers real value here. And every concept in this chapter — tokenisation, vocabulary, sparsity, negation — transfers directly to the deep models later.

3. Classical vs Deep — Five Models on the Map

Let us put five real models on the map, one per region. They are all worth knowing. Even the ones we will never build.

One key insight deserves its own paragraph. The bag of words is not classical or deep by itself. It is a representation. Give the vector to logistic regression: you have built classical NLP. Give the same vector to a neural network: you have built deep NLP. The difference is the learner, not the bag. That is why the pipeline in Figure 25.1 ends with an any-classifier box.

3.1 A first example: marking essays

Before the email, let us see the bag-of-words idea on the simplest possible example. Examiners mark essays. Each mark is a pass (1) or a fail (0). Each examiner also leaves a short comment:

The bag of words puts all these words into one bag. Then it counts. How often does "amazing" appear in comments marked 1? How often in comments marked 0? "amazing" goes with 1. "poor" and "harder" go with 0. "work" appears on both sides. It carries no signal.

Now a new comment arrives: "Good job, keep it up." It has no mark. The model pulls each word out of the bag and checks its history. "good" — mostly seen with 1. "job" — mostly with 1. "keep" and "up" — mostly with 1. Most words point to a pass. So the model predicts: pass.

That is the whole idea. No grammar. No word order. Just a count of which words go with which outcome. Everything else in this chapter is this idea, done carefully at scale.

4. The Email Experiment — a 20,000-Dimension Intuition

Let us learn bag of words with a real email. A friend writes: "Hello Cyril, checking if you're back in Oz. Let me know if you're around and keen to sync… I defo could use some of your creative thinking… Cheers, Vee." ("Oz" means Australia. "defo" is short for definitely.) Under the message, the mail app offers three ready replies: "Yes, I'm around" · "I'm back" · "Sorry, I'm not". That is sequence-to-sequence, working on a real phone. We start with a simpler question. Can a model learn to reply yes or no? That small goal opens the door to all of NLP.

4.1 The vector, and why 20,000 slots

The model's whole memory of language will be one long vector of counters. At the start, every counter is zero. How many slots? Let us do the arithmetic with real numbers (commonly cited figures):

Every word gets a fixed position. For example, "if" lives in slot 7. "badminton" and "table" live in other slots. A word keeps its slot forever. Fixed positions are what let us compare vectors, add them, and feed them to any classifier.

4.2 Three reserved slots

Slots 1 and 2 are reserved. They hold the markers sos (start of sentence) and eos (end of sentence). They work like punctuation for sequence models. They will matter in Part VIII, when networks read text in order. The last slot is the unknown bucket. Every word the vector has never met goes there.

4.3 Pouring the email in

One more detail. Punctuation can get a slot too. A comma is not a word. But it is a character in the text. In an uncleaned vector, it gets its own counter. Section 5 deletes punctuation before counting. That is one reason cleaning comes first.

4.4 Teaching the vector to say yes or no

Now comes the supervised step used in every chapter since Chapter 4: gather history. Take real pairs from a (theoretical) inbox — "Have you read about Hinton's capsule networks?" → "No", "Did you like that recipe?" → "Yes", "Coming to dinner tonight?" → "Yes", "Service your car with us again?" → "No", "Coming to Australia in December?" → "Yes". Thousands of such pairs. Each email becomes the same 20,000-slot vector. Every vector has the same length and the same slot meanings. So any classifier in this course can read them:

One honest note. Real inboxes do not hold clean yes/no labels. People write full replies. To build this dataset, someone must read every reply and judge: was that a yes or a no? That is the real groundwork. This inbox is a theoretical example. Nobody would build it for their own mail. But the method is exactly right.

5. Cleaning & the Corpus — with a Bug Hunt

Now we move from intuition to the 1,000 real reviews. The first review in the dataset, exactly as written: "Wow... Loved this place." Cleaning has five steps. We will build them one line at a time:

5.1 The re library — cleaning with patterns

The first import in the cleaning code is import re. The name re is short for regular expression. A regular expression is a pattern. We write the pattern once, and the library uses it to find every matching character in the text.

Why do we need it? Look at the raw reviews. They contain dots, commas, exclamation marks, numbers and other symbols. These characters do not tell us if a review is positive or negative. So we remove them. The re library removes them with one function:

Now read the pattern [^a-zA-Z] piece by piece. Each symbol has one simple meaning:

Here is the pattern working on real text. The outputs below come from running the exact code:

Notice two things in the results.

First: we replace punctuation with a space, not with nothing. If we delete the comma in "Loved,this" completely, the two words join into one: "Lovedthis". That single word has no meaning, so the model cannot use it. A space keeps the words apart.

Second: the extra spaces are not a problem. Step 3 (.split()) breaks the text at any group of spaces. So "Wow    Loved" gives the same two words as "Wow Loved".

One last point. The re library only cleans the text. It does not do stemming and it does not remove stop words. Those two jobs belong to the next library, NLTK.

5.2 Stop words — the words we drop

NLTK gives us two tools here: the stop words and the stemmer. Stop words first. A stop word is a word with no sentiment. It cannot help us decide positive or negative. Examples: the articles "the", "a", "an". The pronouns "I", "you", "he", "she", "we", "they". Small words like "is", "this", "that".

Look at "Wow... Loved this place." The word "this" is a stop word. Remove it. The meaning does not change. "Loved" stays. It carries the sentiment. So the cleaned review keeps "wow love place" and drops only "this".

Why remove stop words? Two reasons. First, they carry no signal. Second, every kept word becomes a column. A few hundred stop words would add a few hundred useless columns. In code, we keep a word only if it is not in the stop-word list: [ps.stem(word) for word in review if not word in set(all_stopwords)]. NLTK ships the list ready-made: stopwords.words('english').

5.3 The bug hunt: the day "not" disappeared

Now print the cleaned corpus and look at it. Review 2 in the raw data reads "Crust is not good.". The printed corpus says:

The fix matters beyond review 2. Review 3 in the raw data reads "Not tasty and the texture was just nasty." Another negative review. With "not" restored, its cleaned form is "not tasti textur nasti" — the negation survives. Without the fix, this review would also look positive to the bag. The same one-line bug corrupts every negated review in the corpus.

5.4 Stemming vs lemmatization

Why clean at all? Here is the reason: every word that survives becomes a column. Stemming merges "love", "loved" and "loving" into one column. Stop-word removal deletes a few hundred columns that can never carry sentiment. The matrix stays as small as the meaning allows.

6. The Bag of Words

The corpus is clean. The next step is vectorisation. Collect the vocabulary. Then represent each review as a row of word counts. Build a small one by hand first. Three toy reviews. The vocabulary is sorted alphabetically:

That is the whole representation: a document-term count matrix. One more name to learn: the step of taking every word from every review and giving each word its own column is called tokenisation. Now the real run on the 1,000 cleaned reviews:

6.1 The 1,566 → 1,500 decision

Here is a good habit. Build the matrix with no vocabulary limit first. Count the columns with len(X[0]). Only then choose the cap. Re-run for real on this exact corpus:

Two words about the code. fit_transform does two jobs. fit collects the words: it reads the corpus and learns the vocabulary. transform counts: it fills one column per word for every review. Then .toarray() converts the result into a plain 2D array. That is the format sklearn models expect.

7. TF-IDF: Weighing Words

Raw counts have a weakness. Every word counts the same. But words are not equally informative. "great" appears in half the reviews. It separates little. "rancid" appears in nine reviews. It separates a lot. TF-IDF — term frequency × inverse document frequency — fixes this. It re-weights every cell:

Now verify by hand on the toy corpus (n = 3 documents; every number below is real library output). great appears in 2 documents → idf = ln(4/3) + 1 = 1.29. not appears in 1 → idf = ln(4/2) + 1 = 1.69. good appears in all 3 → idf = ln(4/4) + 1 = 1.00. Then multiply by the counts:

Same matrix, smarter weights. "not" is rare and full of sentiment. It now out-weighs "great" in its document. TF-IDF repeats the trick of Chapter 21's lift: divide away what is common, keep what is informative.

8. Naive Bayes Returns — and the Beat-73% Challenge

Naive Bayes returns from Chapter 15, as promised. Text is its historic home. Thousands of sparse count features are exactly what the word-by-word probability product was built for. (The classic spam filter uses Multinomial NB.) This chapter uses GaussianNB. The split is 800 train / 200 test, seed 0. The real results:

Two things to notice. First, the errors lean toward false alarms. Only 12 liked reviews were missed (of 103). But 42 reviews got false praise. This is the negation problem of Section 5, made visible: a "not good" bag contains both "not" and "good". It often tips toward liked. Second, the TF-IDF dip. GaussianNB models each feature with a bell curve. TF-IDF values have a long tail, not a bell. The fit is worse than with raw counts. The lesson is not "TF-IDF is bad". The lesson is: match the representation to the model.

8.1 The first three predictions — and the challenge

The build ends with a moment worth keeping. The predictions print. The first three test reviews are all wrong. Each was predicted positive. Each is actually negative. And note: these are random reviews from the test set. They are not the first rows of the file. No test review was seen during training. Do not panic. Scroll on. By the end of the list, the correct answers dominate. The confusion matrix gives the verdict: 73.0%. And now the challenge of this course: beat 73%. Try other classifiers. Improve the cleaning. Remove "isn't" from the stop words.

We accept. Real runs. Same bag, same split:

Challenge answered: 78.0%. But look at the other columns. The rivals win on accuracy and precision (0.817 vs 0.684). They lose on recall (0.738 vs 0.883). NB catches 91 of 103 positive reviews. The SVM catches 76. This is Chapter 18's whole story in one table. "Best" depends on the costs. A site that auto-promotes positive reviews wants precision. A brand hunting for unhappy customers wants recall.

10. Reading the Results

The full confusion matrix, cell by cell (200 test reviews; 103 are actually liked):

The 42 false positives are the bag's typical error. These reviews contain many positive-sounding words. But they mean the opposite. Examples: "not good", "used to be great", "great location, terrible food". Chapter 18's toolkit reads this matrix clearly. Recall is 0.883: only 12 fans missed. The price is precision 0.684. Is that trade right? It depends on what a false "liked" costs. A missed chance to fix a problem? A wrongly promoted review? Note also the class balance: 50/50. A coin flip already scores 50%. So 0.73 is a real 23-point lift over chance.

The challengers sharpen the picture. The linear SVM's matrix is [[80, 17], [27, 76]]. It moves 25 customers from the false-alarm column to the correct-rejection column. Precision rises to 0.817. The cost: 15 more fans missed. Same data. Same representation. Three defensible operating points. The model choice is a business choice.

One last thought before the outlook. We just trained a machine to read English. It read 1,000 reviews. It learned which words praise and which words complain. And it answers correctly 73 times out of 100. A few decades ago this was a hard research problem. Today it is twenty lines of Python.

11. Positive, Neutral, or Negative

Liked or not liked. Yes or no. Every classifier in this chapter is forced to choose one of two answers. But many real reviews are neither happy nor angry. "The food was okay. Nothing special." That is a lukewarm review. A business often wants three piles: promote the positives, fix the negatives, skip the neutrals. Let us build the third pile. There are two ways.

11.1 Way 1: a dataset with three labels

The clean way needs data that already carries three labels. Star ratings are the classic source. Map 1–2 stars to negative, 3 stars to neutral, 4–5 stars to positive. Now y holds three values instead of two:

# stars -> three classes
y = np.where(stars >= 4, 2,            # 4-5 stars -> 2 (positive)
    np.where(stars == 3, 1, 0))        # 3 -> 1 (neutral), 1-2 -> 0 (negative)

Nothing else changes. GaussianNB, logistic regression and most sklearn classifiers accept three classes without a single new parameter. The confusion matrix becomes 3×3. Precision and recall are computed per class, one row each, with Chapter 18's tools.

One honest limit: Restaurant_Reviews.tsv has no neutral label. So this section ships its own small corpus with star ratings. We built it from word pools: 12 positive words ("great", "delicious"…), 9 lukewarm words ("okay", "average"…), 12 negative words ("terrible", "bland"…), plus everyday filler. Half of the neutral reviews use lukewarm words. The other half are mixed: one praise and one complaint in the same review. 300 reviews per class, 900 rows, fixed random seed. Download it and reproduce every number below yourself:

# ---- Listing 25.3: three classes, the full pipeline ----
# data: synthetic_reviews.csv  (900 reviews, star ratings 1-5)
import numpy as np, pandas as pd, re, nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
nltk.download('stopwords', quiet=True)
sw = stopwords.words('english'); sw.remove('not')      # keep negation
ps = PorterStemmer()

ds = pd.read_csv('synthetic_reviews.csv')              # <- downloaded above
corpus = []
for i in range(len(ds)):
    r = re.sub('[^a-zA-Z]', ' ', ds['Review'][i]).lower().split()
    corpus.append(' '.join(ps.stem(w) for w in r if w not in set(sw)))

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
X = CountVectorizer().fit_transform(corpus).toarray()  # (900, 44)
stars = ds['Stars'].values
y = np.where(stars >= 4, 2, np.where(stars == 3, 1, 0))  # 0 neg, 1 neu, 2 pos
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0)               # 720 train / 180 test

from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report

for model in [GaussianNB(), LogisticRegression(max_iter=1000)]:
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print(type(model).__name__,
          'accuracy:', round(accuracy_score(y_test, y_pred), 4))
    print(confusion_matrix(y_test, y_pred))            # a 3 x 3 matrix
    print(classification_report(y_test, y_pred,
          target_names=['neg (0)', 'neutral (1)', 'pos (2)']))

Listing 25.3 — The full three-label pipeline. Download the CSV, run this file unchanged, and every number below prints on your screen. After cleaning, the whole corpus speaks only 44 words.

The test split holds 180 reviews: 58 negative, 58 neutral, 64 positive. The real results:

Read the middle row. That row is the whole three-class story. GaussianNB catches only 24 of the 58 neutrals — recall 0.414 (its precision is 1.000: what it calls neutral is neutral). The 34 misses are the mixed reviews. Logistic regression catches all 58. When a class has its own vocabulary ("okay", "average"), a model that can weight words one by one learns it easily.

Logistic regression still makes 4 mistakes. All four are the same pattern — a positive review with an honest complaint: "The food was perfect, although the meal was disgusting.", "The atmosphere was best, although the service was disappointing." The strong complaint word out-weighs the praise. The bag counts both sides; it cannot weigh "although". One honest caveat the other way: 0.978 is high partly because the corpus is synthetic and speaks only 44 words. Real star-rated corpora (Yelp, Amazon) are far messier — expect less.

11.2 Way 2: the neutral band — the model abstains

The second way works on our dataset today. Every classifier can output a probability, not just a class. The call is predict_proba. A probability near 1 means: the model is sure the review is positive. Near 0: sure it is negative. And near 0.5? The model does not know. So we give those reviews their own label:

Which model's probabilities can we trust? Test both, on the same 200 test reviews:

GaussianNB is overconfident. Its probabilities collapse to almost 0 or almost 1. Every single one. The median is 1.0000. So the band catches nothing — at any width you choose. Logistic regression gives smooth, honest probabilities. Its median is 0.46. The band catches 49 of the 200 reviews. Lesson: the abstain trick needs a model with usable probabilities.

11.3 The real results, and the trade-off

Logistic regression with the band at 0.40/0.60 (real run, same split as Section 8):

Compare. The same model forced to answer everything scores 0.775 (Section 8). With permission to skip 49 unsure reviews, it answers the remaining 151 at 0.834. And the band width is a dial. Turn it:

Read the staircase. Answer everything: 0.775. Skip the 17 most unsure reviews: 0.809. Skip 87: 0.938 — but you now answer only 113 of 200 reviews. There is no free lunch. Better answers cost coverage. The right width is a business choice, exactly like the cost-based thresholds of Chapter 18. An auto-reply system can live at 0.10. A system that escalates reviews to a busy manager might sit at 0.20.

11.4 What lands in the band

The borderline reviews are worth reading (real test reviews, real probabilities):

These are exactly the reviews a word-count model should not decide. An idiom ("can't go wrong" — the bag sees "wrong"). A mixed review (warm words, then "no!"). A thin review with almost no sentiment words. The band routes all of them to a human.

And the most confident mistake of the whole test set is Section 5's lesson again. The review "We made the drive all the way from North Scottsdale... and I was not one bit disappointed!" is positive. The model gives it p = 0.007 — negative, with near-total confidence. "not disappointed" carries both "not" and "disappointed". The bag counts both. Negation fools the counter even when the counter is confident. (Its opposite: "Loved it...friendly servers, great food, wonderful and imaginative menu." gets p = 0.998 — and is right.)

11.5 Checking the band against true neutral labels

The synthetic corpus has real neutral labels. So it can answer the question the restaurant data cannot: when the band says "neutral", how often is the review truly neutral? Train the binary model on positive and negative reviews only. Then predict probabilities for all 180 test reviews — neutral reviews included, exactly like a real deployment where lukewarm reviews arrive unannounced:

# continues from Listing 25.3: X_train, X_test, y_train, y_test
binary = y_train != 1                       # train on pos + neg only
lr2 = LogisticRegression(max_iter=1000).fit(X_train[binary], y_train[binary])
p = lr2.predict_proba(X_test)[:, 1]         # P(positive) for every test review

band = (p >= 0.40) & (p <= 0.60)            # the neutral band
for lab, name in [(0, 'neg'), (1, 'neu'), (2, 'pos')]:
    caught = int(((y_test == lab) & band).sum())
    total  = int((y_test == lab).sum())
    print(f'true {name}: {caught} of {total} in the band')

The real answer (180 test reviews):

The band mostly works. 45 of the 51 reviews inside it (88.2%) are truly neutral. And 45 of the 58 true neutrals (77.6%) land inside. Look at what the band actually caught: "The menu was standard and acceptable." (p = 0.589). "The service was ordinary and average." (p = 0.488). "The restaurant was usual." (p = 0.455). The binary model never saw these words in training. They push the probability nowhere. So it drifts to the middle — and the band catches them.

But 13 true neutrals escape, and almost all of them are the mixed reviews: "The food was loved, but dinner was stale." got p = 0.666 — confidently positive. "The staff was great, but the place was slow." got p = 0.260 — confidently negative. A mixed review carries two real signals. The model adds them and takes a side.

# ---- Listing 25.4: Way 2 on the restaurant reviews (no neutral labels) ----
# continues from Listing 25.1: X_train, X_test, y_train, y_test are ready

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(max_iter=1000).fit(X_train, y_train)
proba = lr.predict_proba(X_test)[:, 1]       # P(positive) per test review

LO, HI = 0.40, 0.60                          # the band; move it as needed
pred3 = np.where(proba >= HI, 1,             # confident positive
       np.where(proba <= LO, 0, 2))          # confident negative; 2 = neutral

answered = pred3 != 2
print(np.unique(pred3, return_counts=True))  # [0 1 2] -> [81 70 49]
print('coverage:', answered.mean())          # 0.755  (151 of 200 answered)
print('accuracy on answered:',
      (pred3[answered] == y_test[answered]).mean())   # 0.834

# why not GaussianNB? look at its probabilities:
nb_probas = classifier.predict_proba(X_test)[:, 1]
print(np.median(nb_probas))                 # 1.0 — every probability extreme

Listing 25.4 — The neutral band on the restaurant corpus: the model abstains when unsure (Way 2, when no neutral labels exist). All printed numbers are from the canonical run.