Large Language Models
165 views
Tokenization
Quick Definition
Breaking text into tokens for language model processing
Full Definition
Breaking text into smaller units called tokens that language models can process.
Examples
BPE, SentencePiece, word embeddings, vocabulary building
Related Terms
large-language-model
embedding
bpe
More Large Language Models Terms
Prompt Caching
Storing key-value states for repeated prompt prefixes
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Model Merging
Combining fine-tuned LLMs without additional training
GGUF
File format for LLMs optimized for local inference
Embeddings LLM
Dense vectors capturing semantic meaning of text
Guardrails LLM
Frameworks for monitoring and controlling LLM I/O