Large Language Models
29 views
Sliding Window Attention
Quick Definition
Attention limiting tokens to nearby window only
Full Definition
Attention limiting each token to attend to a fixed-size local window of nearby tokens.
Examples
efficient attention, long sequences, Mistral architecture
Related Terms
attention-mechanism
sparse-attention
More Large Language Models Terms
KV Cache
Memory optimization storing key-values for faster generation
Temperature
Parameter controlling randomness in text generation
Model Merging
Combining fine-tuned LLMs without additional training
Quantization LLM
Reducing LLM weight precision for efficient inference
QLoRA
Combining 4-bit quantization with LoRA for efficiency
Embeddings LLM
Dense vectors capturing semantic meaning of text