Large Language Models
26 views
SentencePiece
Quick Definition
Language-agnostic tokenizer operating on raw text bytes
Full Definition
A language-agnostic tokenizer treating text as raw bytes enabling tokenization without pre-segmentation.
Examples
multilingual models, Llama tokenization, open vocabulary
Related Terms
tokenization
bpe
wordpiece
More Large Language Models Terms
Chain-of-Thought Prompting
Technique encouraging step-by-step reasoning in LLMs
Constitutional AI
Training AI to follow explicit principles for consistency
QLoRA
Combining 4-bit quantization with LoRA for efficiency
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
Embeddings LLM
Dense vectors capturing semantic meaning of text
Instruction Tuning
Fine-tuning on instruction-response pairs for better following