51. What Is Natural Language Processing (NLP)
51.3 Tiny Text Cleaning Pipeline
Why. Raw text is messy: emoji, Marathi–English mix, extra spaces, "Pune!!!" vs "pune". Models need a calm pipeline before features.
What. A beginner pipeline: lowercase → strip punctuation/noise → split into tokens (words) → optional stop-word drop → then vectorise (Chapter 52). You stay in Python strings and lists first.
How.
Clean one fictional review
- Lowercase the string.
- Keep letters/numbers/spaces; drop most punctuation.
- Split on spaces into tokens.
- Drop empty tokens.
import re
raw = " Fresh PEDHA from Nashik!!! Delivery was OK :) "
text = raw.lower().strip()
text = re.sub(r"[^a-z0-9\s]", " ", text)
tokens = [t for t in text.split() if t]
print(tokens)
# ['fresh', 'pedha', 'from', 'nashik', 'delivery', 'was', 'ok']
Fictional data
Reviews are invented for teaching — not scraped from a real storefront.