Large Language Models
285 views
RoPE
Quick Definition
Position encoding using rotation matrices for relative positions
Full Definition
Rotary Position Embedding encoding relative positions using rotation matrices.
Examples
Llama, Mistral, length generalization
Related Terms
positional-encoding
transformer
More Large Language Models Terms
Linear Attention
Attention with linear complexity for long sequences
Tool Use
Capability of LLMs to call external tools and APIs
Hallucination LLM
LLMs generating plausible but incorrect information
Quantization LLM
Reducing LLM weight precision for efficient inference
Model Merging
Combining fine-tuned LLMs without additional training
Multi-Head Attention
Parallel attention operations concatenated for richer representations