Large Language Models
133 views
Attention Head
Quick Definition
Individual attention mechanism learning specific patterns
Full Definition
Individual attention mechanisms within multi-head attention each learning different relationship patterns.
Examples
multi-head attention, self-attention, transformer layers
Related Terms
attention-mechanism
transformer
More Large Language Models Terms
Constitutional AI
Training AI to follow explicit principles for consistency
Streaming
Sending LLM tokens to clients as they are generated
WordPiece
Subword tokenization splitting words into frequent units
Positional Encoding
Incorporating token position information into transformers
Model Merging
Combining fine-tuned LLMs without additional training
Flash Attention
Memory-efficient attention using GPU SRAM block computation