Large Language Models
31 views
Tokenization
Quick Definition
Breaking text into tokens for language model processing
Full Definition
Breaking text into smaller units called tokens that language models can process.
Examples
BPE, SentencePiece, word embeddings, vocabulary building
Related Terms
large-language-model
embedding
bpe
More Large Language Models Terms
Distillation LLM
Training smaller LLMs to replicate larger model behavior
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Chain-of-Thought Prompting
Technique encouraging step-by-step reasoning in LLMs
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Context Window
Maximum tokens a language model can process at once
Pre-training
Initial training learning general language patterns from text