Large Language Models
142 views
Distillation LLM
Quick Definition
Training smaller LLMs to replicate larger model behavior
Full Definition
Training smaller student LLMs to replicate behavior of larger teacher models.
Examples
student-teacher training, model compression, edge deployment
Related Terms
model-distillation
quantization
More Large Language Models Terms
Structured Output
Generating LLM responses in predefined machine-readable formats
Mixture of Experts
Architecture routing tokens to specialized expert sub-networks
Reward Model
Model scoring LLM outputs based on human preferences
Multi-Head Attention
Parallel attention operations concatenated for richer representations
BPE
Subword tokenization algorithm merging frequent character pairs
PEFT
Fine-tuning methods updating only small parameter subsets