Large Language Models
23 views
Attention Head
Quick Definition
Individual attention mechanism learning specific patterns
Full Definition
Individual attention mechanisms within multi-head attention each learning different relationship patterns.
Examples
multi-head attention, self-attention, transformer layers
Related Terms
attention-mechanism
transformer
More Large Language Models Terms
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
GGUF
File format for LLMs optimized for local inference
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Fine-Tuning
Further training pre-trained models on specific domain data
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Nucleus Sampling
Generation using cumulative probability threshold for token selection