Large Language Models
25 views
Multi-Head Attention
Quick Definition
Parallel attention operations concatenated for richer representations
Full Definition
A transformer mechanism running multiple attention operations in parallel and concatenating their outputs.
Examples
transformer layers, contextual learning, multi-perspective analysis
Related Terms
attention-mechanism
self-attention
transformer
More Large Language Models Terms
Mistral
Efficient open-source LLMs with strong performance
Reward Model
Model scoring LLM outputs based on human preferences
Sparse Attention
Attention attending to only a subset of tokens
Next Token Prediction
Core LLM objective predicting the next token from context
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
RoPE
Position encoding using rotation matrices for relative positions