Ravi Sir's Cards › Generative AI
Tokens and next-token prediction
Revision card: Tokens and next-token prediction. Example sentence “Pune is famous for vada pav.” split into 9 tokens; the AI guesses the next token; 1 token is about three quarters of an English word.
Read card text
What this card says
Tokens and next-token prediction
Example
sentence “Pune is famous for vada pav.” split into 9 tokens; the AI guesses the next token; 1 token is about three quarters of an English word