Large Language Models
289 views
Tokenizer Vocabulary
Quick Definition
Complete set of tokens a language model can process
Full Definition
The complete set of tokens a language model can recognize and process.
Examples
word pieces, subwords, special tokens
Related Terms
tokenization
bpe
More Large Language Models Terms
GGUF
File format for LLMs optimized for local inference
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
Prompt Caching
Storing key-value states for repeated prompt prefixes
GPT
OpenAI's family of autoregressive large language models
Attention Head
Individual attention mechanism learning specific patterns
Adapter Layer
Small trainable modules in frozen transformer layers