Large Language Models
265 views
Pre-training
Quick Definition
Initial training learning general language patterns from text
Full Definition
Initial training phase where models learn general language patterns from massive text corpora.
Examples
GPT pre-training, BERT training, foundation model creation
Related Terms
self-supervised-learning
fine-tuning
More Large Language Models Terms
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Grounded Generation
Generating outputs faithful to provided source documents
Context Window
Maximum tokens a language model can process at once
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Tree-of-Thought
Framework exploring multiple reasoning paths for optimal solutions
Positional Encoding
Incorporating token position information into transformers