Large Language Models
270 views
KV Cache
Quick Definition
Memory optimization storing key-values for faster generation
Full Definition
A memory optimization storing computed key-value pairs to avoid redundant attention computation.
Examples
inference optimization, generation speedup, memory management
Related Terms
attention-mechanism
inference-optimization
More Large Language Models Terms
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
Positional Encoding
Incorporating token position information into transformers
Linear Attention
Attention with linear complexity for long sequences
Temperature
Parameter controlling randomness in text generation
Embeddings LLM
Dense vectors capturing semantic meaning of text
Quantization LLM
Reducing LLM weight precision for efficient inference