Large Language Models
153 views
Tokenizer Vocabulary
Quick Definition
Complete set of tokens a language model can process
Full Definition
The complete set of tokens a language model can recognize and process.
Examples
word pieces, subwords, special tokens
Related Terms
tokenization
bpe
More Large Language Models Terms
Temperature
Parameter controlling randomness in text generation
Pre-training
Initial training learning general language patterns from text
Guardrails LLM
Frameworks for monitoring and controlling LLM I/O
Mixture of Experts
Architecture routing tokens to specialized expert sub-networks
Structured Output
Generating LLM responses in predefined machine-readable formats
Chain-of-Thought Prompting
Technique encouraging step-by-step reasoning in LLMs