Large Language Models
150 views
WordPiece
Quick Definition
Subword tokenization splitting words into frequent units
Full Definition
A subword tokenization algorithm that splits words into frequent subword units based on likelihood.
Examples
BERT tokenization, vocabulary building, unknown word handling
Related Terms
tokenization
bpe
sentencepiece
More Large Language Models Terms
Grounded Generation
Generating outputs faithful to provided source documents
Tool Use
Capability of LLMs to call external tools and APIs
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
Nucleus Sampling
Generation using cumulative probability threshold for token selection
Fine-Tuning
Further training pre-trained models on specific domain data