51. What Is Natural Language Processing (NLP)
51.4 Bag-of-Words Intuition
Why. Text is not one float column. You must turn documents into a matrix before sklearn fit.
What. Classic idea: bag of words — count how often each word appears, ignore grammar order at first. Next chapter upgrades counts to TF-IDF (smarter weights). Labels (y) stay 0/1 for sentiment.
How.
See documents as rows
- Each review = one document (one row).
- Vocabulary = unique tokens across the training set.
- Cell value = count (or later TF-IDF weight) of that token in that review.
- y = sentiment label known from humans.
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
docs = [
"fresh pedha on time",
"sour dahi late delivery",
"fresh dahi good packing",
]
y = [1, 0, 1] # 1=positive, 0=negative (fictional)
vec = CountVectorizer()
X = vec.fit_transform(docs)
print(vec.get_feature_names_out())
print(X.toarray())
print("y:", y)
Ravindra Bagale's Tip
Bag-of-words order ignore karto — "not good" hard case. Tarahi baseline strong; TF-IDF next.