Large Language Models
144 views
Sparse Attention
Quick Definition
Attention attending to only a subset of tokens
Full Definition
Attention where each token attends to only a subset of other tokens for efficiency.
Examples
Longformer, BigBird, efficient transformers
Related Terms
attention-mechanism
sliding-window-attention
More Large Language Models Terms
PEFT
Fine-tuning methods updating only small parameter subsets
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Semantic Search
Search technique understanding meaning rather than keywords
Embeddings LLM
Dense vectors capturing semantic meaning of text
Speculative Decoding
Using draft model candidates verified by large model
QLoRA
Combining 4-bit quantization with LoRA for efficiency