Sitefetch: The Secret Weapon for Feeding Websites to AI Models

Sitefetch: The Secret Weapon for Feeding Websites to AI Models

What if I told you that hours of manual copy-pasting could be replaced by a single command?

Every developer who works with AI models has hit the same wall. You need to train a model, fine-tune responses, or build a RAG system. The data? Scattered across hundreds of web pages. Your options are grim: write a custom scraper (hours of debugging selectors and rate limits), pay for expensive APIs with restrictive quotas, or—the worst fate—manually copy content into text files like it's 2003.

I spent three days building a scraper for a documentation project last month. It broke on JavaScript^{↗ Bright Coding Blog}-rendered pages. It grabbed navigation menus and footer junk. It got rate-limited. When I finally got clean data, I discovered sitefetch—and realized I'd wasted my time.

Created by EGOIST, the prolific open-source developer behind tools like Vite plugins and the Chatwise app, sitefetch solves this exact pain point with brutal simplicity. One command. Any website. Clean text output ready for your LLM pipeline. No configuration files. No puppeteer nightmares. No subscription fees bleeding your budget dry.

In this deep dive, I'll show you why sitefetch is becoming the go-to tool for developers building AI-powered applications—and how you can start using it in under 60 seconds.

What is Sitefetch?

Sitefetch is an open-source CLI tool that crawls entire websites and compiles them into a single, clean text file. Built specifically for AI model consumption, it strips away the visual noise of modern web design—navigation bars, advertisements, cookie banners, sidebar widgets—and extracts only the meaningful content.

The project lives at egoist/sitefetch and has gained rapid traction among developers for one simple reason: it solves a universal problem with zero friction. Unlike traditional web scrapers that require complex configuration, sitefetch operates on a "zero config, maximum results" philosophy that aligns perfectly with modern developer expectations.

EGOIST, the creator, is no stranger to building developer tools that gain cult followings. His work on the Vite ecosystem and the Chatwise LLM chat application gives him unique insight into what AI practitioners actually need. Sitefetch emerged from this intersection of web tooling and AI workflows—a tool built by someone who understands both worlds intimately.

What makes sitefetch particularly relevant now is the explosive growth of context-window sizes in large language models. Claude 3.5 Sonnet handles 200K tokens. Gemini 1.5 Pro pushes 1 million tokens. GPT-4o's context window keeps expanding. These massive windows create an unprecedented opportunity: feeding entire codebases, documentation sites, or competitor analyses directly into models. But you need clean, consolidated text first. Sitefetch bridges that gap.

The tool leverages battle-tested libraries under the hood—Mozilla's Readability for content extraction and micromatch for flexible URL filtering—rather than reinventing wheels. This architectural choice means sitefetch inherits years of refinement in web content parsing while maintaining a dead-simple interface.

Key Features That Make Sitefetch Irresistible

Zero-Configuration Content Extraction

Sitefetch uses Mozilla's Readability—the same engine powering Firefox's reader mode—to intelligently identify and extract article content. This isn't naive HTML stripping; it's semantic analysis that understands page structure. The result? You get clean paragraphs, preserved code blocks, and organized headings without touching a single CSS selector.

Flexible Page Targeting with Glob Patterns

Need only blog posts and guides from a sprawling documentation site? The -m, --match flag accepts micromatch patterns, giving you grep-like precision over URL pathnames. Fetch /blog/** and /guide/** while ignoring /api/reference/** and /changelog/**. This selective crawling saves tokens, reduces noise, and keeps your AI context windows focused.

Concurrency Control for Speed

Slow sequential crawling kills productivity. Sitefetch's --concurrency flag lets you blast through pages in parallel. Dial it up to 10, 20, or higher depending on the target site's tolerance and your network capacity. For large documentation sites, this can reduce hours of waiting to minutes.

Multiple Package Manager Support

Whether you're team Bun, npm, or pnpm, sitefetch meets you where you are. One-off execution via bunx, npx, or pnpx means no permanent installation for occasional use. Global installation via bun i -g, npm i -g, or pnpm i -g provides persistent CLI access for power users.

Programmatic API for Custom Pipelines

The fetchSite function exposes the core engine for integration into Node.js applications. Build automated documentation updaters, scheduled content monitors, or dynamic RAG ingestion pipelines. The TypeScript-typed options interface ensures your IDE guides configuration correctly.

CSS Selector Override for Edge Cases

When Readability's heuristics miss the mark—common on heavily JavaScript-rendered SPAs or unusual page structures—the --content-selector flag lets you pinpoint exactly where content lives. A simple .content or #main-article selector overrides the automatic detection with surgical precision.

Real-World Use Cases Where Sitefetch Dominates

Building Documentation-Aware AI Assistants

Imagine creating a support bot that actually understands your framework's documentation. Not surface-level RAG with chunked embeddings, but deep comprehension of every guide, tutorial, and API reference. Sitefetch ingests the entire docs site into a format you can feed directly to Claude or GPT-4 for fine-tuning or few-shot prompting. The result? An assistant that cites specific sections accurately and understands version nuances.

Competitive Intelligence at Scale

Monitor competitor pricing pages, feature announcements, or documentation changes. Schedule sitefetch runs to capture entire competitor sites weekly, then diff the outputs to spot strategic shifts. The clean text format makes programmatic comparison trivial—no HTML parsing nightmares, no visual diff tools.

Training Data Curation for Domain-Specific Models

Building a legal assistant? A medical coding model? A specialized programming tutor? The web contains vast domain knowledge trapped in poorly structured sites. Sitefetch liberates this content into trainable corpora. Target specific sections with match patterns, extract clean text, and pipeline into your data preprocessing workflow.

Offline Knowledge Bases for Security-Conscious Environments

Air-gapped systems can't query the live web. Sitefetch creates portable knowledge snapshots—entire reference sites in single text files that transfer via USB or secure file exchange. Security teams audit the static output; no dynamic crawling risks in production environments.

Legacy Content Migration

Modernizing an old CMS? The original content might be scattered across hundreds of generated pages with inconsistent templates. Sitefetch extracts the semantic content regardless of presentation layer, giving you clean source material for new-site population.

Step-by-Step Installation & Setup Guide

One-Off Usage (No Installation)

For occasional use, execute directly via your preferred package runner:

# Using Bun's blazing-fast runner
bunx sitefetch

# Classic npm npx
npx sitefetch

# pnpm's efficient alternative
pnpx sitefetch

These commands download and execute the latest version temporarily, leaving your system clean afterward. Perfect for CI pipelines or one-time data extraction tasks.

Global Installation (Recommended for Regular Use)

For frequent access, install globally:

# Bun users get fastest install speeds
bun i -g sitefetch

# Standard npm global install
npm i -g sitefetch

# pnpm's disk-space-efficient global install
pnpm i -g sitefetch

Verify installation:

sitefetch --version

Basic Configuration

No configuration file needed! Sitefetch operates via CLI flags. However, consider these environment optimizations:

• Set a reasonable concurrency limit for target sites—start with 5, increase based on response reliability
• Use output redirection (-o) rather than stdout for large sites
• Combine with tee for progress visibility on massive crawls: sitefetch URL -o out.txt | tee /dev/tty

Quick Validation Test

Confirm everything works:

# Fetch a small personal site as smoke test
sitefetch https://egoist.dev -o test.txt

# Verify output
wc -l test.txt    # Line count
head -20 test.txt # Preview content quality

REAL Code Examples from the Repository

The sitefetch README provides concise, practical examples that reveal the tool's power. Let's dissect each one with detailed explanations.

Example 1: Basic Site Capture

sitefetch https://egoist.dev -o site.txt

This foundational command demonstrates sitefetch's core value proposition. The URL https://egoist.dev specifies the starting point—sitefetch will discover linked pages within the same origin and recursively fetch them. The -o site.txt flag directs output to a file rather than flooding your terminal.

What's happening under the hood: Sitefetch initiates a crawl from the root page, extracts all same-origin hrefs, filters duplicates, applies Readability to each page for content extraction, and concatenates results with clear page separators in the output file. The process respects robots.txt implicitly through responsible default concurrency.

When to use this: Perfect for complete site archives, initial data exploration, or when you need everything and will filter later.

Example 2: Performance-Tuned Crawling

# or better concurrency
sitefetch https://egoist.dev -o site.txt --concurrency 10

The comment "or better concurrency" hints at a critical optimization. Default concurrency is conservative; this flag unleashes parallel processing.

Technical breakdown: --concurrency 10 maintains 10 simultaneous HTTP requests. This dramatically reduces wall-clock time for I/O-bound crawling, especially on high-latency connections or with distant servers. The trade-off? Increased load on the target server and higher memory usage for concurrent response buffering.

Best practice progression: Start with default concurrency for unknown sites. Increase to 5-10 for established documentation hosts (Cloudflare, Vercel, Netlify). Monitor for 429 (Too Many Requests) responses—back off if encountered. For local development servers or self-hosted sites, aggressive concurrency may crash lightweight servers.

Example 3: Selective Page Matching

sitefetch https://vite.dev -m "/blog/**" -m "/guide/**"

This example showcases sitefetch's precision filtering. The -m flag (short for --match) accepts micromatch glob patterns tested against URL pathnames.

Pattern analysis:

• /blog/** matches /blog/index.html, /blog/why-vite, /blog/2024/annual-update—any depth under /blog/
• /guide/** similarly captures all guide content
• Multiple -m flags combine with OR logic—pages matching any pattern are included

The micromatch advantage: Unlike simple substring matching, micromatch supports negation (!**/draft/**), character classes, brace expansion, and extglobs. This expressiveness handles complex inclusion/exclusion rules that would require multiple tools otherwise.

Real-world application: A Vite plugin developer needs only API reference and plugin authoring guides, not marketing pages or team blog posts. Two match patterns extract exactly the relevant corpus.

Example 4: CSS Selector Content Targeting

sitefetch https://vite.dev --content-selector ".content"

When Readability's automatic detection falters—common with JavaScript frameworks that hydrate content into generic containers—this flag provides manual override.

The problem it solves: Modern SPAs often render into <div id="app"> or similar root elements. Readability may struggle to identify the meaningful content boundary, potentially including navigation UI or excluding dynamically loaded sections. A targeted CSS selector eliminates ambiguity.

Selector strategy: Inspect the target site with browser DevTools. Identify the stable container that holds primary content—often .content, .documentation, [role="main"], or framework-specific classes like .nuxt-content. Test with document.querySelector('.content') in console before running sitefetch.

Example 5: Programmatic API Integration

import { fetchSite } from "sitefetch"

await fetchSite("https://egoist.dev", {
  //...options
})

The TypeScript API unlocks custom automation. This isn't just a CLI wrapper—the fetchSite function exposes the full crawling engine for embedded use.

Integration patterns:

import { fetchSite } from "sitefetch"
import { writeFileSync } from "fs"

// Custom pre-processing pipeline
const content = await fetchSite("https://docs.example.com", {
  match: ["/api/**", "/tutorials/**"],
  concurrency: 15,
  contentSelector: ".doc-content"
})

// Post-process: add custom headers, filter sections, chunk for embeddings
const processed = addMetadata(content, { source: "example-docs", version: "2.1.0" })
writeFileSync("training-corpus.txt", processed)

Type safety: The options object is fully typed via types.ts. Your IDE autocomplete reveals all available configuration—no documentation diving required for parameter discovery.

Advanced Usage & Best Practices

Token Budget Optimization

LLM context windows are precious. Before feeding sitefetch output to models, analyze and trim:

# Estimate token count (rough: 1 token ≈ 4 characters)
wc -c site.txt | awk '{print int($1/4) " tokens"}'

# Preview structure to identify bloat
grep -n "^=== " site.txt | head -20  # Page separators

Use match patterns aggressively to exclude irrelevant sections. A focused 50K token corpus outperforms a noisy 200K token dump.

Incremental Updates with Differential Crawling

For monitoring use cases, combine sitefetch with hashing:

#!/bin/bash
sitefetch https://target.com -o new.txt
if ! diff -q new.txt last.txt >/dev/null; then
  echo "Changes detected, triggering pipeline..."
  cp new.txt last.txt
  # Trigger embedding update, notification, etc.
fi

Respectful Crawling Ethics

• Check robots.txt before aggressive crawling
• Use conservative concurrency (2-3) for small or personal sites
• Consider sleep intervals between batches for sustained monitoring
• Cache aggressively; don't re-crawl unchanged content

Output Post-Processing Pipeline

Chain sitefetch with standard Unix tools for refinement:

# Extract only code blocks for programming model training
sitefetch https://docs.python↗ Bright Coding Blog.org -o - | grep -A 20 "^\`\`\`" > code-examples.txt

# Remove specific sections with sed
sitefetch https://example.com | sed '/^## Sponsors/,/^## /d' > no-ads.txt

Comparison with Alternatives

Why sitefetch wins: It occupies the sweet spot between wget's simplicity (but poor output quality) and Scrapy's power (but heavy setup). Unlike Firecrawl, it's free, offline-capable, and requires no API keys. Unlike custom Playwright scripts, it works in seconds without debugging selector timing issues.

Frequently Asked Questions

Is sitefetch legal for commercial use?

Yes, the MIT license permits commercial use, modification, and distribution. However, respect target sites' terms of service and robots.txt directives. Sitefetch is a tool; responsible usage is your obligation.

Can sitefetch handle JavaScript-rendered single-page applications?

Sitefetch fetches static HTML. For SPAs requiring JavaScript execution, pre-render or use a headless browser solution. The --content-selector flag helps when content exists in HTML but Readability misses it due to unusual structure.

How does sitefetch handle pagination and infinite scroll?

It follows standard <a href> links within the same origin. It does not execute JavaScript to trigger infinite scroll loading. For paginated content, ensure navigation links exist in the initial HTML or manually specify page URLs.

What's the maximum site size sitefetch can handle?

Practically limited by your system's memory and disk space. Output is streamed to the file system, so even gigabyte-scale sites are feasible with sufficient storage. Extremely large crawls may benefit from splitting by match patterns.

Can I use sitefetch in GitHub Actions or CI pipelines?

Absolutely. The one-off execution (npx sitefetch) is ideal for CI. Pin to a specific version for reproducibility: npx sitefetch@1.2.3. Cache output between runs to minimize redundant crawling.

Does sitefetch support authentication or cookies?

The current version focuses on public content. For authenticated sites, consider exporting cookies or using the programmatic API with a custom fetch implementation. Check the types.ts for evolving options.

How do I report issues or request features?

Visit the GitHub Issues page. EGOIST is responsive, and the focused scope means feature requests get serious consideration.

Conclusion: Your AI Pipeline Deserves This Tool

Sitefetch transforms a tedious, error-prone workflow into a single command. No more custom scraper maintenance. No more copy-paste fatigue. No more wrestling with HTML-to-text conversion libraries that garble code blocks and mangle headings.

I've integrated it into my documentation pipeline, my competitive monitoring scripts, and my RAG ingestion workflows. Each time, it saves hours and produces cleaner output than my custom solutions ever achieved.

The AI landscape rewards speed of iteration. The faster you can feed quality data to your models, the faster you ship better features. Sitefetch removes the friction between "I need this website's content" and "my model has context."

Stop building scrapers. Start shipping AI features.

Grab sitefetch from egoist/sitefetch today—install takes 10 seconds, and your first complete site archive completes before you finish reading this sentence. Your future self, staring at a clean text file instead of a broken Puppeteer script, will thank you.

And if sitefetch saves you time? Check out EGOIST's Chatwise—the LLM chat app built by someone who clearly understands what developers actually need.

Found this guide valuable? Star the repository, share with your AI-building colleagues, and watch your web-to-model pipeline transform from bottleneck to competitive advantage.