Large Language Models
144 views
Multi-Head Attention
Quick Definition
Parallel attention operations concatenated for richer representations
Full Definition
A transformer mechanism running multiple attention operations in parallel and concatenating their outputs.
Examples
transformer layers, contextual learning, multi-perspective analysis
Related Terms
attention-mechanism
self-attention
transformer
More Large Language Models Terms
Linear Attention
Attention with linear complexity for long sequences
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
PEFT
Fine-tuning methods updating only small parameter subsets
Positional Encoding
Incorporating token position information into transformers
Distillation LLM
Training smaller LLMs to replicate larger model behavior
RoPE
Position encoding using rotation matrices for relative positions