Large Language Models
22 views
Distillation LLM
Quick Definition
Training smaller LLMs to replicate larger model behavior
Full Definition
Training smaller student LLMs to replicate behavior of larger teacher models.
Examples
student-teacher training, model compression, edge deployment
Related Terms
model-distillation
quantization
More Large Language Models Terms
Alignment Tax
Performance cost of aligning AI with human preferences
Tokenizer Vocabulary
Complete set of tokens a language model can process
Linear Attention
Attention with linear complexity for long sequences
Flash Attention
Memory-efficient attention using GPU SRAM block computation
Prompt Template
Reusable structured format with dynamic input placeholders
PEFT
Fine-tuning methods updating only small parameter subsets