52. Text Cleaning and TF-IDF in Python
52.2 TfidfVectorizer Beginner Knobs
Why. Vectoriser settings change quality: stop words, n-grams, min document frequency. Beginners should know the knobs without drowning.
What. Useful knobs: stop_words="english", ngram_range=(1,2) for bigrams ("not good"), min_df to drop ultra-rare typos, max_features to cap columns on tiny laptops.
How.
Try a safer beginner config
- Start with unigrams + bigrams.
- Drop English stop words for English-heavy text.
- Cap features (e.g. 500) on practice data.
- Always split train/test before fitting the vectoriser.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
docs = [
"fresh pedha good packing",
"not good packing torn box",
"sour dahi late delivery",
"fresh dahi on time",
"bad smell sour tray",
"sweet pedha excellent",
]
y = [1, 0, 0, 1, 0, 1]
X_train, X_test, y_train, y_test = train_test_split(
docs, y, test_size=0.33, random_state=42, stratify=y
)
vec = TfidfVectorizer(
stop_words="english",
ngram_range=(1, 2),
min_df=1,
max_features=500,
)
X_train_m = vec.fit_transform(X_train)
X_test_m = vec.transform(X_test)
print("Train shape:", X_train_m.shape, "Test shape:", X_test_m.shape)
print("Sample features:", list(vec.get_feature_names_out())[:8])