Large Language Models
281 views
Prompt Caching
Quick Definition
Storing key-value states for repeated prompt prefixes
Full Definition
Storing computed key-value states for repeated prompt prefixes to reduce inference latency and cost.
Examples
reduced latency, cost optimization, system prompt reuse
Related Terms
kv-cache
inference-optimization
prompt-engineering
More Large Language Models Terms
RAG
Enhancing LLMs by retrieving relevant external knowledge
Sliding Window Attention
Attention limiting tokens to nearby window only
Masked Language Model
Training objective predicting randomly masked input tokens
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Top-K Sampling
Generation strategy considering top k most probable tokens
Prompt Template
Reusable structured format with dynamic input placeholders