49. What Transformers Are (Text and Sequences)
49.3 Tokens and Embeddings
Why. Models read numbers, not letters. Tokenisation splits text; embeddings map token ids to vectors.
What. Beginner view: each word piece gets an id; nn.Embedding turns ids into trainable vectors.
How.
Tiny vocabulary demo
- Build a dict word→id.
- Encode a short review.
- Embed with nn.Embedding.
import torch
import torch.nn as nn
vocab = {"pad": 0, "pedha": 1, "tasty": 2, "not": 3, "good": 4, "stale": 5}
ids = torch.tensor([[1, 2, 0, 0], [3, 4, 0, 0]]) # batch of 2
emb = nn.Embedding(num_embeddings=6, embedding_dim=4)
print(emb(ids).shape) # (2, 4, 4) = batch, seq, dim
What you should see. Shape (batch, sequence length, embed dim).