52. Text Cleaning and TF-IDF in Python
52.1 TF-IDF Intuition
Why. Counts treat "the" and "refund" the same. Rare, meaningful words should weigh more when you classify Mauli reviews.
What. TF-IDF = Term Frequency × Inverse Document Frequency. Words common in one review but rare across the corpus get higher weight. sklearn's TfidfVectorizer builds the matrix for you.
How.
Intuition in one India story
- "fresh" appears in many positive dairy reviews → less distinctive alone.
- "sour" appears in few reviews → strong negative signal.
- TF-IDF down-weights boring common tokens, up-weights distinctive ones.
from sklearn.feature_extraction.text import TfidfVectorizer
docs = [
"fresh pedha good packing",
"sour dahi bad packing",
"fresh dahi on time delivery",
]
vec = TfidfVectorizer()
X = vec.fit_transform(docs)
print(vec.get_feature_names_out())
print(X.toarray().round(2))
Fit on train only
Same rule as scalers (ch40): fit / fit_transform on training text; transform only on test text.