Build a GPT-like LLM from Scratch: Why This PyTorch Repo Has 30K+ Stars

Stop treating large language models like black boxes. Every day, thousands of developers call OpenAI's API, pipe text through transformers.AutoModel, and pretend they understand what happens inside. They don't. And the worst part? They know it. That nagging feeling that you're just a user of someone else's magic—not a builder of your own.

What if you could strip away every abstraction? What if you could code attention mechanisms by hand, watch your own GPT model generate its first coherent sentence, and finally grok why ChatGPT works the way it does?

That's exactly what rasbt/LLMs-from-scratch delivers. Created by Sebastian Raschka—renowned machine learning researcher and author—this repository isn't another wrapper around Hugging Face. It's a from-the-ground-up educational journey that takes you from Python basics to a functioning, pretrained, finetuned language model. No shortcuts. No hidden libraries. Just you, PyTorch, and the raw architecture of modern AI.

If you're tired of consuming AI and ready to create it, this is your moment. Let's dive into why this repository is dominating developer circles and how you can use it to finally master LLMs.

What Is LLMs-from-scratch?

LLMs-from-scratch is the official code repository for Sebastian Raschka's book Build a Large Language Model (From Scratch), published by Manning in 2024 (ISBN 9781633437166). But calling it a "companion repo" sells it short. This is a standalone educational powerhouse that has attracted tens of thousands of stars, forks, and active community discussions.

Sebastian Raschka isn't some random GitHub contributor. He's a machine learning researcher with deep expertise in deep learning, the author of Python Machine Learning, and a voice respected across the AI community. When he decided to demystify LLMs, he didn't write another high-level tutorial. He committed to implementing every core component by hand in pure PyTorch—no transformers library, no pre-built tokenizers hiding the mechanics.

The repository's philosophy is radical in its simplicity: mirroring the actual approach used to build foundational models like GPT-4, but at an educational scale. You build a "small-but-functional" model that runs on ordinary laptops, yet follows the same architectural principles as the multi-billion parameter behemoths. The repository also includes code for loading weights from larger pretrained models for finetuning—bridging the gap between educational toy models and production-grade systems.

What's driving its explosive popularity? Three forces converged. First, the AI boom created massive demand for genuine understanding rather than API fluency. Second, Raschka's reputation guaranteed quality. Third, the repository arrived at the perfect moment—when developers realized that prompt engineering had limits, and model engineering was the next frontier. The included 17-hour companion video course, organized chapter-by-chapter, transformed it into a complete self-study curriculum.

Key Features That Set This Apart

Let's dissect what makes this repository genuinely exceptional—not just "good for beginners," but structurally transformative for how you understand AI:

• Pure PyTorch Implementation: Zero dependencies on external LLM libraries. Every attention head, every layer normalization, every feed-forward block is coded explicitly. This isn't pedagogical purism—it's how you build intuition that transfers to any framework.

• Complete LLM Lifecycle Coverage: The repository spans the entire pipeline—from text tokenization (Chapter 2), through attention mechanisms (Chapter 3), GPT architecture construction (Chapter 4), pretraining on unlabeled data (Chapter 5), classification finetuning (Chapter 6), to instruction following via finetuning (Chapter 7). Most tutorials cover one phase; this covers all of them.

• Hardware Accessibility: The main chapters run on conventional laptops within reasonable timeframes. GPU acceleration is automatically utilized when available, but never required. This democratizes access in a field often gated by $10K GPU clusters.

• Production-Ready Extensions: Beyond the educational core, bonus materials include KV caching for efficient inference, grouped-query attention, mixture-of-experts architectures, LoRA for parameter-efficient finetuning, and even conversions from GPT to Llama architectures. You're not just learning 2023's GPT-2; you're touching 2024's frontier.

• Cross-Platform Reliability: Automated testing across Linux, Windows, and macOS ensures the code actually works on your machine—not just the author's.

• Massive Supplementary Ecosystem: 170 pages of self-test questions, exercise solutions for every chapter, Docker environments, and bonus notebooks on topics from BPE tokenizers to DPO alignment. This isn't a repo; it's a curriculum engine.

Real-World Use Cases Where This Shines

Theory without application is just trivia. Here's where developers are actually deploying insights from this repository:

1. Breaking Into AI Engineering Roles The brutal truth? Interviewers can spot API users versus model builders instantly. Walking through this repository gives you the vocabulary and intuition to discuss attention patterns, training dynamics, and architectural trade-offs with the confidence of someone who's touched the metal. Candidates who've built models from scratch consistently outperform those who've only fine-tuned bert-base-uncased.

2. Debugging Production Model Failures When your finetuned model hallucinates, generates repetitive outputs, or collapses during training, surface-level debugging fails. Understanding the actual forward pass—because you coded it—lets you trace issues to their source: attention score distributions, gradient flow problems, or tokenization edge cases.

3. Research Prototyping and Architecture Innovation Want to experiment with a modified attention mechanism? Test a new normalization scheme? The repository's clean, modular structure lets you swap components without fighting against abstractions designed for different purposes. Researchers use this as a springboard for novel architectures.

4. Educational Content Creation and Team Training Technical leads use this repository to onboard teams to LLM fundamentals. The chapter-by-chapter structure, complete with diagrams and progressive complexity, makes it ideal for structured learning programs. The included video course enables self-paced upskilling without consuming senior engineer time.

5. Bridging to Advanced Techniques The bonus materials on LoRA, DPO, KV caching, and model distillation provide direct pathways from educational basics to implementation of cutting-edge efficiency and alignment techniques used in production systems.

Step-by-Step Installation & Setup Guide

Ready to stop reading and start building? Here's your exact path from zero to running code:

Step 1: Clone the Repository

# Shallow clone to save space while getting everything you need
git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git

# Enter the project directory
cd LLMs-from-scratch

The --depth 1 flag creates a shallow clone without full git history—perfect for getting started quickly. The full repository is available at https://github.com/rasbt/LLMs-from-scratch for updates.

Step 2: Configure Your Python Environment

The repository requires Python with PyTorch installed. Navigate to the setup directory for detailed guidance:

# Check the setup README for environment-specific instructions
cat setup/README.md

For most users, a standard virtual environment suffices:

# Create isolated Python environment
python -m venv llm-env

# Activate it (Linux/macOS)
source llm-env/bin/activate

# Activate it (Windows)
llm-env\Scripts\activate

Step 3: Install Dependencies

The repository uses pure PyTorch without external LLM libraries, keeping dependencies minimal:

# Install core requirements
pip install torch torchvision torchaudio

# Additional utilities used across chapters
pip install numpy matplotlib tiktoken

For the complete dependency set, refer to the setup directory's installation guides at setup/02_installing-python-libraries.

Step 4: Verify Your Installation

# Quick smoke test—PyTorch should detect GPU if available
python -c "import torch; print(f'PyTorch {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')"

Step 5: Launch Your First Notebook

# Start with Chapter 2: Working with Text Data
jupyter notebook ch02/01_main-chapter-code/ch02.ipynb

Each chapter's 01_main-chapter-code directory contains the primary notebooks, with exercise-solutions.ipynb for self-testing. The mental model diagram in the main README provides a visual roadmap of how chapters connect.

REAL Code Examples from the Repository

Let's examine actual code patterns from the repository, with detailed explanations of what makes each significant.

Example 1: The Core GPT Architecture (from ch04/01_main-chapter-code/gpt.py)

This is the architectural heart—every component you interact with in ChatGPT, stripped to essentials:

import torch
import torch.nn as nn

class GPTModel(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        # Token embeddings: convert vocabulary IDs to dense vectors
        self.tok_emb = nn.Embedding(cfg["vocab_size"], cfg["emb_dim"])
        
        # Positional embeddings: encode where in the sequence a token appears
        self.pos_emb = nn.Embedding(cfg["context_length"], cfg["emb_dim"])
        
        # Dropout for regularization during training
        self.drop_emb = nn.Dropout(cfg["drop_rate"])
        
        # Stack of transformer blocks—the core processing engine
        self.trf_blocks = nn.Sequential(
            *[TransformerBlock(cfg) for _ in range(cfg["n_layers"])]
        )
        
        # Final layer normalization stabilizes training at depth
        self.final_norm = LayerNorm(cfg["emb_dim"])
        
        # Output projection to vocabulary space for next-token prediction
        self.out_head = nn.Linear(cfg["emb_dim"], cfg["vocab_size"], bias=False)

    def forward(self, in_idx):
        batch_size, seq_len = in_idx.shape
        
        # Look up token embeddings for input indices
        tok_embeds = self.tok_emb(in_idx)
        
        # Create and look up positional embeddings
        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))
        
        # Combine token and positional information—this is the input representation
        x = tok_embeds + pos_embeds
        x = self.drop_emb(x)
        
        # Pass through transformer blocks for contextualized representations
        x = self.trf_blocks(x)
        
        # Normalize before final prediction
        x = self.final_norm(x)
        
        # Project to vocabulary logits—higher values = more likely next tokens
        logits = self.out_head(x)
        return logits

Why this matters: This isn't pseudocode—it's the actual architectural pattern used in GPT-2, GPT-3, and their successors. The tok_emb + pos_embeds combination is the only way the model knows both what each token is and where it appears. The TransformerBlock stack (detailed in Chapter 3) applies self-attention repeatedly, building increasingly sophisticated contextual representations. The final out_head projection is what makes next-token prediction possible—every token position predicts a probability distribution over the entire vocabulary.

Example 2: Multi-Head Attention Implementation (from ch03/01_main-chapter-code/multihead-attention.ipynb)

Attention is where the "magic" happens. Here's the scaled dot-product mechanism, implemented explicitly:

class MultiHeadAttention(nn.Module):
    def __init__(self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False):
        super().__init__()
        assert d_out % num_heads == 0, "d_out must be divisible by num_heads"
        
        self.d_out = d_out
        self.num_heads = num_heads
        # Each head gets a slice of the output dimension
        self.head_dim = d_out // num_heads
        
        # Linear projections for queries, keys, and values
        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)
        
        # Final output projection combines all heads
        self.out_proj = nn.Linear(d_out, d_out)
        self.dropout = nn.Dropout(dropout)
        
        # Causal mask prevents attending to future tokens (critical for autoregressive generation)
        self.register_buffer(
            "mask",
            torch.triu(torch.ones(context_length, context_length), diagonal=1)
        )

    def forward(self, x):
        b, num_tokens, d_in = x.shape
        
        # Project inputs to Q, K, V spaces
        keys = self.W_key(x)      # Shape: (b, num_tokens, d_out)
        queries = self.W_query(x)
        values = self.W_value(x)
        
        # Reshape for multi-head processing: split d_out into num_heads × head_dim
        # We implicitly split the matrix by adding a num_heads dimension
        keys = keys.view(b, num_tokens, self.num_heads, self.head_dim).transpose(1, 2)
        queries = queries.view(b, num_tokens, self.num_heads, self.head_dim).transpose(1, 2)
        values = values.view(b, num_tokens, self.num_heads, self.head_dim).transpose(1, 2)
        
        # Compute attention scores: how much each token attends to each other token
        attn_scores = queries @ keys.transpose(2, 3)  # Dot product attention
        
        # Apply causal mask: set future positions to negative infinity (before softmax)
        mask_bool = self.mask.bool()[:num_tokens, :num_tokens]
        attn_scores.masked_fill_(mask_bool, -torch.inf)
        
        # Scale by sqrt(head_dim) for training stability—prevents extreme softmax values
        attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)
        attn_weights = self.dropout(attn_weights)
        
        # Weighted sum of values produces context-aware representations
        context_vec = (attn_weights @ values).transpose(1, 2)
        
        # Reshape back and apply final projection
        context_vec = context_vec.contiguous().view(b, num_tokens, self.d_out)
        context_vec = self.out_proj(context_vec)
        return context_vec

Critical insight: The causal mask (torch.triu) is what makes GPT autoregressive—it can only attend to previous and current positions, never future ones. This is why GPT generates text left-to-right. The scaling factor (keys.shape[-1]**0.5) isn't arbitrary; it prevents dot products from growing too large with dimension, which would push softmax into extremely sharp distributions and destroy gradient flow. These aren't implementation details—they're why the architecture works.

Example 3: Text Generation with Temperature Scaling (from ch05/01_main-chapter-code/gpt_generate.py)

Once trained, here's how the model actually produces text:

def generate_text_simple(model, idx, max_new_tokens, context_size):
    """
    idx: (batch, n_tokens) array of indices in the current context
    """
    for _ in range(max_new_tokens):
        # Crop context to maximum size the model can handle
        idx_cond = idx[:, -context_size:]
        
        # Get predictions—model returns logits for all positions
        with torch.no_grad():  # No need to track gradients during inference
            logits = model(idx_cond)
        
        # Focus only on the last time step's predictions
        logits = logits[:, -1, :]  # Shape: (batch, vocab_size)
        
        # Convert logits to probabilities
        probas = torch.softmax(logits, dim=-1)
        
        # Sample from the probability distribution (greedy: take most likely)
        idx_next = torch.argmax(probas, dim=-1, keepdim=True)
        
        # Append to running sequence for next iteration
        idx = torch.cat((idx, idx_next), dim=1)
    
    return idx

The generation loop exposed: This reveals the fundamental simplicity of autoregressive generation. The model never "knows" it's generating a sentence—it just predicts one next token, appends it, and repeats. The context_size crop is crucial; without it, sequences longer than training context would cause errors. For more sophisticated sampling (temperature, top-k, top-p), the repository extends this with probability reweighting—controlling the randomness-creativity tradeoff that distinguishes coherent text from gibberish.

Advanced Usage & Best Practices

Moving beyond the basics, here's how to extract maximum value:

• Progressive Complexity Path: Don't jump to Chapter 5. The repository is architected deliberately—each chapter builds mental models that later chapters assume. Skipping attention mechanisms to "get to the fun stuff" creates gaps that haunt you during debugging.

• Exercise-Driven Learning: Every chapter includes exercises with solutions in exercise-solutions.ipynb. Actually attempt them before checking answers. The 170-page self-test PDF provides additional validation of understanding.

• GPU Utilization Strategy: While laptop-compatible, pretraining benefits enormously from GPU acceleration. The code automatically detects CUDA; for cloud training, the repository's structure ports cleanly to single-GPU instances.

• Bonus Material Sequencing: After completing core chapters, explore KV caching (inference optimization), LoRA (efficient finetuning), and the GPT-to-Llama conversion to understand modern architectural evolution.

• Community Engagement: The GitHub Discussions forum contains clarifications, common pitfalls, and extended explanations from Raschka himself. Search before asking—your confusion has likely been addressed.

Comparison with Alternatives

The verdict: Choose LLMs-from-scratch when you need foundational understanding that transfers across frameworks and architectures. Choose Hugging Face when you need immediate production deployment. Use nanoGPT for research experimentation with minimal overhead. The repositories complement rather than replace each other—many developers use this repository to understand what Hugging Face abstracts away.

Frequently Asked Questions

Q: Do I need a PhD in machine learning to follow this repository? A: Absolutely not. Strong Python fundamentals are the primary prerequisite. Some neural network familiarity helps but isn't required—Appendix A provides a PyTorch crash course. The book targets motivated developers, not researchers.

Q: Can I actually train a useful model on my laptop? A: The educational model trains on laptop CPUs/GPUs, but won't match ChatGPT's capabilities—it's designed for understanding, not production. However, the repository includes code for loading weights from larger pretrained models and finetuning them, bridging to practical applications.

Q: How does this differ from just reading the Transformer paper? A: The "Attention Is All You Need" paper describes architecture; this repository implements it with progressive complexity, training dynamics, and practical considerations (numerical stability, memory efficiency, convergence) that papers omit.

Q: Is the code up-to-date with 2024 architectures? A: The bonus materials actively extend coverage—Llama 3.2, Qwen3, Gemma 3, MoE, and more are implemented. The core educational model remains GPT-like for pedagogical clarity, but modern variants are available.

Q: Can I contribute improvements to the repository? A: The maintainer currently doesn't accept contributions that modify main chapter code (to preserve book consistency), but welcomes discussions, issue reports, and community extensions via GitHub Discussions.

Q: How long does complete mastery take? A: The 17-hour video course suggests intensive study. Most developers report 4-8 weeks for thorough coverage with exercises, depending on prior PyTorch experience.

Q: Will this help me get a job in AI? A: The skills developed—understanding attention, training dynamics, finetuning strategies, and architectural trade-offs—are precisely what technical interviews assess. Portfolio projects built from this repository demonstrate capability beyond API usage.

Conclusion: Your Invitation to Build, Not Just Consume

The AI landscape is bifurcating: those who use models, and those who understand them. The first group will always be dependent, always paying per token, always guessing why outputs fail. The second group—the builders, the engineers who've traced gradients through attention heads and watched loss curves converge—that's where genuine innovation lives.

rasbt/LLMs-from-scratch isn't just code. It's a declaration of independence from black-box dependence. Sebastian Raschka has constructed perhaps the most accessible yet rigorous pathway from curious developer to competent LLM engineer, and he's given it away freely.

The repository awaits. The notebooks are ready. The only question is whether you'll remain a consumer of AI magic—or become the magician yourself.

Clone it today. Build your first model this week. Thank yourself a year from now.

git clone --depth 1 https://github.com/rasbt/LLMs-from-scratch.git

Your journey from API caller to model architect starts with this single command. Don't let another tutorial scroll by without acting. The future belongs to builders—and now you have the blueprint.