Large Language Models
24 views
Quantization LLM
Quick Definition
Reducing LLM weight precision for efficient inference
Full Definition
Reducing LLM weight precision to lower bits like INT8 or INT4 for efficient inference.
Examples
4-bit inference, GPTQ, AWQ, model deployment
Related Terms
quantization
inference-optimization
More Large Language Models Terms
Scaling Laws
Relationships between model size, data, compute, and performance
Vector Database
Specialized database for high-dimensional embedding queries
KV Cache
Memory optimization storing key-values for faster generation
Sliding Window Attention
Attention limiting tokens to nearby window only
Hallucination LLM
LLMs generating plausible but incorrect information
GGUF
File format for LLMs optimized for local inference