Large Language Models
177 views
Speculative Decoding
Quick Definition
Using draft model candidates verified by large model
Full Definition
Using a small draft model to generate candidates verified in parallel by the large target model.
Examples
inference acceleration, LLM serving, throughput optimization
Related Terms
inference-optimization
kv-cache
More Large Language Models Terms
Guardrails LLM
Frameworks for monitoring and controlling LLM I/O
Mixture of Experts
Architecture routing tokens to specialized expert sub-networks
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
Alignment Tax
Performance cost of aligning AI with human preferences
Linear Attention
Attention with linear complexity for long sequences
SentencePiece
Language-agnostic tokenizer operating on raw text bytes