Large Language Models
154 views
RoPE
Quick Definition
Position encoding using rotation matrices for relative positions
Full Definition
Rotary Position Embedding encoding relative positions using rotation matrices.
Examples
Llama, Mistral, length generalization
Related Terms
positional-encoding
transformer
More Large Language Models Terms
Pre-training
Initial training learning general language patterns from text
PEFT
Fine-tuning methods updating only small parameter subsets
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Multi-Head Attention
Parallel attention operations concatenated for richer representations
Linear Attention
Attention with linear complexity for long sequences
Guardrails LLM
Frameworks for monitoring and controlling LLM I/O