Large Language Models
266 views
Attention Head
Quick Definition
Individual attention mechanism learning specific patterns
Full Definition
Individual attention mechanisms within multi-head attention each learning different relationship patterns.
Examples
multi-head attention, self-attention, transformer layers
Related Terms
attention-mechanism
transformer
More Large Language Models Terms
Speculative Decoding
Using draft model candidates verified by large model
Hallucination LLM
LLMs generating plausible but incorrect information
RoPE
Position encoding using rotation matrices for relative positions
Inference LLM
Using a trained LLM to generate outputs from inputs
Embeddings LLM
Dense vectors capturing semantic meaning of text
DPO
Alignment method bypassing reward model for preference learning