52. Text Cleaning and TF-IDF in Python
52.4 Inspect Vocabulary and Rows
Why. Seeing the sparse matrix and vocabulary makes debugging less scary when a model "sees" the wrong words.
What. TfidfVectorizer returns a sparse matrix. Use .toarray() only on tiny demos. Inspect get_feature_names_out() to confirm bigrams and cleaning.
How.
Inspect what the model will see
- Fit on a tiny train list.
- Print feature names.
- Print one row of weights (rounded).
- Confirm test uses
transform, not a secondfit.
from sklearn.feature_extraction.text import TfidfVectorizer
train = ["fresh pedha good", "sour dahi bad", "fresh dahi good"]
test = ["fresh pedha bad"]
vec = TfidfVectorizer(ngram_range=(1, 2))
Xtr = vec.fit_transform(train)
Xte = vec.transform(test)
print(vec.get_feature_names_out())
print("test row:", Xte.toarray().round(2))