Large Language Models
277 views
Sparse Attention
Quick Definition
Attention attending to only a subset of tokens
Full Definition
Attention where each token attends to only a subset of other tokens for efficiency.
Examples
Longformer, BigBird, efficient transformers
Related Terms
attention-mechanism
sliding-window-attention
More Large Language Models Terms
Mixture of Experts
Architecture routing tokens to specialized expert sub-networks
Multi-Head Attention
Parallel attention operations concatenated for richer representations
Quantization LLM
Reducing LLM weight precision for efficient inference
Autoregressive Model
Generative model predicting next token from previous tokens
Top-K Sampling
Generation strategy considering top k most probable tokens
Llama
Meta AI's open-source large language models