Flash Attention
Quick Definition
Memory-efficient attention using GPU SRAM block computation
Full Definition
An optimized attention algorithm computing in GPU SRAM blocks for reduced memory and increased speed.
Examples
fast training, memory optimization, transformer acceleration
Related Terms
attention-mechanism
gpu-optimization
More Large Language Models Terms
BPE
Subword tokenization algorithm merging frequent character pairs
Distillation LLM
Training smaller LLMs to replicate larger model behavior
RAG
Enhancing LLMs by retrieving relevant external knowledge
Sparse Attention
Attention attending to only a subset of tokens
Tokenizer Vocabulary
Complete set of tokens a language model can process
Attention Head
Individual attention mechanism learning specific patterns