Large Language Models
278 views
Mixture of Experts
Quick Definition
Architecture routing tokens to specialized expert sub-networks
Full Definition
An architecture routing tokens to specialized expert sub-networks via a gating mechanism.
Examples
Mixtral, Switch Transformer, sparse scaling
Related Terms
transformer
sparse-model
More Large Language Models Terms
Semantic Search
Search technique understanding meaning rather than keywords
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Grounding LLM
Connecting LLM outputs to verifiable external sources
Grounded Generation
Generating outputs faithful to provided source documents
PEFT
Fine-tuning methods updating only small parameter subsets
Top-K Sampling
Generation strategy considering top k most probable tokens