Large Language Models
29 views
Tokenizer Vocabulary
Quick Definition
Complete set of tokens a language model can process
Full Definition
The complete set of tokens a language model can recognize and process.
Examples
word pieces, subwords, special tokens
Related Terms
tokenization
bpe
More Large Language Models Terms
Reward Model
Model scoring LLM outputs based on human preferences
Tokenization
Breaking text into tokens for language model processing
Quantization LLM
Reducing LLM weight precision for efficient inference
Pre-training
Initial training learning general language patterns from text
Attention Head
Individual attention mechanism learning specific patterns
Mistral
Efficient open-source LLMs with strong performance