Large Language Models
144 views
Mixture of Experts
Quick Definition
Architecture routing tokens to specialized expert sub-networks
Full Definition
An architecture routing tokens to specialized expert sub-networks via a gating mechanism.
Examples
Mixtral, Switch Transformer, sparse scaling
Related Terms
transformer
sparse-model
More Large Language Models Terms
Sparse Attention
Attention attending to only a subset of tokens
Linear Attention
Attention with linear complexity for long sequences
Embeddings LLM
Dense vectors capturing semantic meaning of text
Distillation LLM
Training smaller LLMs to replicate larger model behavior
Mistral
Efficient open-source LLMs with strong performance
QLoRA
Combining 4-bit quantization with LoRA for efficiency