Large Language Models
158 views
SentencePiece
Quick Definition
Language-agnostic tokenizer operating on raw text bytes
Full Definition
A language-agnostic tokenizer treating text as raw bytes enabling tokenization without pre-segmentation.
Examples
multilingual models, Llama tokenization, open vocabulary
Related Terms
tokenization
bpe
wordpiece
More Large Language Models Terms
Masked Language Model
Training objective predicting randomly masked input tokens
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
RoPE
Position encoding using rotation matrices for relative positions
Autoregressive Model
Generative model predicting next token from previous tokens
Grounded Generation
Generating outputs faithful to provided source documents
Vector Database
Specialized database for high-dimensional embedding queries