Large Language Models
32 views
Prompt Caching
Quick Definition
Storing key-value states for repeated prompt prefixes
Full Definition
Storing computed key-value states for repeated prompt prefixes to reduce inference latency and cost.
Examples
reduced latency, cost optimization, system prompt reuse
Related Terms
kv-cache
inference-optimization
prompt-engineering
More Large Language Models Terms
Vector Database
Specialized database for high-dimensional embedding queries
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
Inference LLM
Using a trained LLM to generate outputs from inputs
Masked Language Model
Training objective predicting randomly masked input tokens
Tokenization
Breaking text into tokens for language model processing
GPT
OpenAI's family of autoregressive large language models