Large Language Models
23 views
Mixture of Experts
Quick Definition
Architecture routing tokens to specialized expert sub-networks
Full Definition
An architecture routing tokens to specialized expert sub-networks via a gating mechanism.
Examples
Mixtral, Switch Transformer, sparse scaling
Related Terms
transformer
sparse-model
More Large Language Models Terms
Model Merging
Combining fine-tuned LLMs without additional training
PEFT
Fine-tuning methods updating only small parameter subsets
Tree-of-Thought
Framework exploring multiple reasoning paths for optimal solutions
Semantic Search
Search technique understanding meaning rather than keywords
Scaling Laws
Relationships between model size, data, compute, and performance
Inference LLM
Using a trained LLM to generate outputs from inputs