Large Language Models
313 views
Sliding Window Attention
Quick Definition
Attention limiting tokens to nearby window only
Full Definition
Attention limiting each token to attend to a fixed-size local window of nearby tokens.
Examples
efficient attention, long sequences, Mistral architecture
Related Terms
attention-mechanism
sparse-attention
More Large Language Models Terms
Reward Model
Model scoring LLM outputs based on human preferences
Flash Attention
Memory-efficient attention using GPU SRAM block computation
Autoregressive Model
Generative model predicting next token from previous tokens
BERT
Google's bidirectional transformer for context understanding
Prompt Caching
Storing key-value states for repeated prompt prefixes
Sparse Attention
Attention attending to only a subset of tokens