Large Language Models
281 views
Multi-Head Attention
Quick Definition
Parallel attention operations concatenated for richer representations
Full Definition
A transformer mechanism running multiple attention operations in parallel and concatenating their outputs.
Examples
transformer layers, contextual learning, multi-perspective analysis
Related Terms
attention-mechanism
self-attention
transformer
More Large Language Models Terms
Autoregressive Model
Generative model predicting next token from previous tokens
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
QLoRA
Combining 4-bit quantization with LoRA for efficiency
Next Token Prediction
Core LLM objective predicting the next token from context
Hallucination LLM
LLMs generating plausible but incorrect information
Attention Head
Individual attention mechanism learning specific patterns