Large Language Models
302 views
SentencePiece
Quick Definition
Language-agnostic tokenizer operating on raw text bytes
Full Definition
A language-agnostic tokenizer treating text as raw bytes enabling tokenization without pre-segmentation.
Examples
multilingual models, Llama tokenization, open vocabulary
Related Terms
tokenization
bpe
wordpiece
More Large Language Models Terms
Guardrails LLM
Frameworks for monitoring and controlling LLM I/O
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Linear Attention
Attention with linear complexity for long sequences
Semantic Search
Search technique understanding meaning rather than keywords
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
Attention Head
Individual attention mechanism learning specific patterns