tb0hdan/domains: 1.7 Billion Internet Domains Dataset

tb0hdan/domains: 1.7 Billion Internet Domains Dataset

Finding comprehensive, freely available domain data at scale is a persistent challenge for security researchers, data scientists, and infrastructure engineers. Most datasets are either proprietary, fragmented across sources, or too small to yield meaningful statistical insights. The tb0hdan/domains repository on GitHub addresses this gap directly: it hosts what its maintainers describe as the world's single largest public Internet domains dataset, currently at 1.7 billion domains and counting. This article examines what the dataset contains, how to access it, and where it fits in modern data pipelines.

What is tb0hdan/domains?

tb0hdan/domains is a public GitHub repository maintained by the Domains Project (domainsproject.org). Its stated purpose is straightforward: provide a freely available, sorted list of Internet domains without requiring users to process petabytes of raw crawl data themselves. The repository has accumulated 1,144 stars, 179 forks, and uses JavaScript^{↗ Bright Coding Blog} as its primary language under the BSD 3-Clause license. The last commit was on 2026-05-03, indicating active maintenance.

The dataset's scale is its defining characteristic. The repository contains 1.7 billion domains in its public GitHub form, with subscription tiers extending to 2.9 billion and 3 billion domains. The project has processed 8.1 petabytes of Internet traffic to build this corpus—a figure that underscores the infrastructure challenge of domain enumeration at global scale.

The maintainer, tb0hdan, also develops related tooling: Freya (DNS checks client, early stages) and Idun (HTTP crawler, being rewritten). The crawler itself, versions 1.0.7+, implements partial robots.txt support and rate limiting. Data collection uses Scrapy and Colly frameworks.

A notable commercial connection exists: the maintainer runs LLMSE.ai, an LLM-based search engine using Ollama under the hood, though this appears separate from the domains dataset itself.

Key Features

Massive Scale with Efficient Storage. The compression efficiency is remarkable: 1.7 billion domains require only 4.6 GB of compressed disk space. The project notes that 1 million domains compress to approximately 5 MB, and more than 1 TB of Internet traffic reduces to roughly 3 MB compressed. This makes the dataset practical to transfer and store even on modest hardware.

Git LFS Distribution. The repository uses Git LFS (Large File Storage) to manage dataset files, with actual domain lists stored in /data after the 30 million record threshold was crossed. Files are compressed with XZ, requiring standard decompression tools.

Sorted, Line-Delimited Format. After unpacking, domains are plain text files with one domain per line. The repository totals approximately 49 GB uncompressed for the 1.7 billion domain set. This format maximizes compatibility with Unix tooling, databases, and custom processing pipelines.

Active DNS Sensor Network. The project accepts community contributions through passive DNS sensors, documented in a separate repository (tb0hdan/pdns-sensor). This distributed collection model helps sustain growth without centralized infrastructure costs.

Subscription Tier for Raw Data. While the GitHub repository provides filtered, sorted lists, subscribers gain access to unfiltered data at dataset.domainsproject.org with additional features: TLD-only filtering, WebSocket feeds for new domains, and DNS JSON with historical data.

Research-Grade Provenance. The dataset has been cited in multiple peer-reviewed publications, which we detail in the Use Cases section. This academic validation distinguishes it from commercially scraped lists of uncertain methodology.

Use Cases

Security Research and Threat Intelligence. The dataset enables large-scale analysis of domain registration patterns, typosquatting detection, and homoglyph identification. Published research using this data includes "GlyphNet: Homoglyph domains dataset and detection using attention-based Convolutional Neural Networks" and "Misty Registry: An Empirical Study of Flawed Domain Registry Operation." Security teams can cross-reference against this baseline to identify anomalous registrations.

Academic Network Measurement. Researchers studying Internet structure rely on comprehensive domain lists for sampling and validation. Papers such as "Understanding and Characterizing the Adoption of Internationalized Domain Names in Practice" (IEEE) and "Cloudy with a Chance of Cyberattacks: Dangling Resources Abuse on Cloud Platforms" demonstrate the dataset's utility for peer-reviewed work.

Search Engine Bootstrapping. The YaCy open-source search engine community has explicitly referenced this dataset for easier search bootstrapping. The sorted, deduplicated nature of the list reduces the cold-start problem for new crawlers.

Email Infrastructure Analysis. Research on SPF, DKIM, and DMARC deployment rates—such as "Phishing Protection SPF, DKIM, DMARC" and a Czech-language email address analysis thesis—leverages the domain list as a representative sample of the global namespace.

Database and Algorithm Testing. Computer science research on data structures benefits from realistic, large-scale string datasets. Papers including "Proteus: A Self-Designing Range Filter" and "Large Scale String Analytics in Arkouda" use this corpus for benchmarking.

Installation & Setup

Accessing the dataset requires Git, Git LFS, and XZ utilities. The repository provides exact commands:

# Clone the repository
git clone https://github.com/tb0hdan/domains.git

# Enter the directory
cd domains

# Initialize Git LFS for large file handling
git lfs install

# Decompress all dataset files
./unpack.sh

Step-by-step explanation:

1. git clone retrieves the repository structure and LFS pointers. The actual data files are not downloaded until LFS is initialized.

2. git lfs install configures your local Git to handle LFS-tracked files. Without this step, you will only receive text pointers, not the compressed domain lists.

3. ./unpack.sh runs the provided unpacking script, which invokes XZ decompression across the /data directory. The README explicitly notes that XZ is freely available at tukaani.org/xz if your system lacks it.

System requirements: The project notes that an 8-core/16-thread machine with 64 GB RAM can process approximately 2 million new domains daily. This specification reflects their crawler infrastructure, not client-side unpacking requirements. For most users, decompression is I/O-bound and completes on standard development hardware.

Network considerations: A 1 Gbps saturated link yields roughly 2 million new domains daily in their collection pipeline. For downloading the static dataset, the compressed 4.6 GB transfers in minutes on modern connections.

Real Code Examples

The repository's documentation is primarily procedural rather than API-oriented. We present the available examples with context.

Example 1: Basic Clone and Unpack Workflow

The core interaction pattern, reproduced from the README:

# Standard retrieval workflow
git clone https://github.com/tb0hdan/domains.git
cd domains
git lfs install
./unpack.sh

This sequence is the documented entry point for all users. The unpack.sh script abstracts the XZ decompression across the /data directory structure, which organizes domains by country/region (e.g., data/afghanistan/domain2multi-af.txt).

Example 2: Sample Data Format

After unpacking, files contain one domain per line. The README provides this verified sample from data/afghanistan/domain2multi-af.txt:

1tv.af
1tvnews.af
3rdeye.af
8am.af
aan.af
acaa.gov.af
acb.af
acbr.gov.af
acci.org.af
ach.af
acku.edu.af
acsf.af
adras.af
aeiti.af

This format requires no parsing library—standard Unix tools (grep, awk, sort, comm) operate directly. The .af TLD example illustrates country-code organization; other files follow similar patterns.

Example 3: Subscription Dataset Mirroring

For subscribers with unfiltered data access, the README documents wget mirroring:

# Mirror the subscription dataset locally
wget -m https://dataset.domainsproject.org

The -m flag enables recursive mirroring with timestamping. This retrieves additional features unavailable in the GitHub repository: TLD-only views, WebSocket endpoints for real-time new domain feeds, and DNS JSON with historical records.

Note on code availability: The README does not contain extensive programmatic examples for consuming the data. Users are expected to integrate the line-delimited format into their existing tooling. The [INTERNAL_LINK: data-processing-pipelines] article covers patterns for large-scale text processing that apply directly.

Advanced Usage & Best Practices

Memory-mapped processing for large files. At 49 GB uncompressed, full dataset loads into memory may strain typical workstations. Consider streaming processing (mmap, iterator patterns) or database ingestion (PostgreSQL^{↗ Bright Coding Blog} COPY, SQLite .import) rather than loading complete files.

Incremental updates via Git LFS. Since the repository receives updates, track which commit you've processed. The git lfs fetch and git lfs checkout commands allow selective retrieval if you only need specific regions or time slices.

Respect crawler boundaries. If you're building on this data to drive your own crawlers, note the Domains Project bot's robots.txt handling. The documented user-agent patterns are:

Mozilla/5.0 (compatible; Domains Project/1.0.8; +https://domainsproject.org)
Mozilla/5.0 (compatible; Domains Project/1.0.4; +https://github.com/tb0hdan/domains)

To block this bot, use either:

User-agent: domainsproject.org
Disallow:/

or:

User-agent: Domains Project
Disallow:/

Compression ratio awareness. The 1.7 billion → 4.6 GB ratio means archival storage is trivial, but working sets expand 10×. Plan disk space accordingly for uncompressed analysis pipelines.

Subscription evaluation. The GitHub repository's 1.7 billion domains may suffice for many research purposes. Evaluate whether TLD filtering, WebSocket feeds, or DNS JSON historical data justify subscription costs for your specific use case.

Comparison with Alternatives

Trade-offs: Majestic Million offers ranking data but at 1/1700th the scale. Common Crawl provides raw content but requires substantial extraction to yield domain lists. Rapid7 Sonar was comparable for DNS research but closed open access. The Domains Project's unique position is its combination of scale, open accessibility, and active maintenance—though researchers needing content (not just domain names) must pair it with crawlers.

FAQ

Q: Is the dataset truly free? A: The 1.7 billion domain GitHub repository is free under BSD 3-Clause. Subscription tiers for raw data and advanced features are paid.

Q: Why do I need Git LFS? A: Standard Git cannot efficiently handle multi-gigabyte files. Git LFS stores pointers in Git and actual data in separate storage.

Q: Can I use this commercially? A: BSD 3-Clause permits commercial use with attribution. Verify subscription terms separately if using paid features.

Q: How current is the data? A: Last commit was 2026-05-03. The crawler infrastructure adds ~2 million domains daily under ideal conditions.

Q: What hardware do I need to process it? A: Unpacking runs on standard hardware. The 64 GB RAM / 8-core spec refers to their crawler infrastructure, not client requirements.

Q: Are IDN (internationalized) domains included? A: Research papers using this dataset specifically study IDN adoption, confirming their presence.

Q: How do I cite this in academic work? A: Link to https://github.com/tb0hdan/domains and domainsproject.org. Multiple published papers have established precedent.

Conclusion

tb0hdan/domains occupies a distinctive position in the open data ecosystem: a actively maintained, billion-scale domain corpus with demonstrated academic credibility and minimal access friction. For security researchers validating detection algorithms, network scientists measuring Internet structure, or engineers bootstrapping crawlers, the 4.6 GB compressed dataset eliminates the infrastructure burden of independent collection.

The trade-off is specificity—you get domain names, not content, rankings, or reputation scores. Users requiring those dimensions must layer additional data sources. But as a foundational layer for domain-aware analysis, the scale and openness are difficult to match.

If your work involves large-scale domain analysis, clone the repository, verify the format against your pipeline, and evaluate whether the free tier meets your needs before considering subscription features. The project's academic citations and active maintenance suggest it will remain a standard reference for years.

Get started: https://github.com/tb0hdan/domains