Large Language Models
148 views
Prompt Caching
Quick Definition
Storing key-value states for repeated prompt prefixes
Full Definition
Storing computed key-value states for repeated prompt prefixes to reduce inference latency and cost.
Examples
reduced latency, cost optimization, system prompt reuse
Related Terms
kv-cache
inference-optimization
prompt-engineering
More Large Language Models Terms
QLoRA
Combining 4-bit quantization with LoRA for efficiency
Adapter Layer
Small trainable modules in frozen transformer layers
Top-K Sampling
Generation strategy considering top k most probable tokens
Masked Language Model
Training objective predicting randomly masked input tokens
Training LLM
Adjusting parameters on large datasets to learn language
Speculative Decoding
Using draft model candidates verified by large model