Large Language Models
137 views
Linear Attention
Quick Definition
Attention with linear complexity for long sequences
Full Definition
Attention mechanism with linear complexity for efficient long-sequence processing.
Examples
efficient transformers, long documents, streaming models
Related Terms
attention-mechanism
sparse-attention
More Large Language Models Terms
WordPiece
Subword tokenization splitting words into frequent units
Attention Head
Individual attention mechanism learning specific patterns
RoPE
Position encoding using rotation matrices for relative positions
Nucleus Sampling
Generation using cumulative probability threshold for token selection
Masked Language Model
Training objective predicting randomly masked input tokens
GGUF
File format for LLMs optimized for local inference