Large Language Models
264 views
Attention Head
Quick Definition
Individual attention mechanism learning specific patterns
Full Definition
Individual attention mechanisms within multi-head attention each learning different relationship patterns.
Examples
multi-head attention, self-attention, transformer layers
Related Terms
attention-mechanism
transformer
More Large Language Models Terms
PEFT
Fine-tuning methods updating only small parameter subsets
Multi-Head Attention
Parallel attention operations concatenated for richer representations
Fine-Tuning
Further training pre-trained models on specific domain data
Chain-of-Thought Prompting
Technique encouraging step-by-step reasoning in LLMs
Sliding Window Attention
Attention limiting tokens to nearby window only
Model Merging
Combining fine-tuned LLMs without additional training