Large Language Models
276 views
Sparse Attention
Quick Definition
Attention attending to only a subset of tokens
Full Definition
Attention where each token attends to only a subset of other tokens for efficiency.
Examples
Longformer, BigBird, efficient transformers
Related Terms
attention-mechanism
sliding-window-attention
More Large Language Models Terms
Tool Use
Capability of LLMs to call external tools and APIs
Distillation LLM
Training smaller LLMs to replicate larger model behavior
RAG
Enhancing LLMs by retrieving relevant external knowledge
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Speculative Decoding
Using draft model candidates verified by large model
Jailbreaking
Circumventing AI safety restrictions through prompt techniques