Stop Wasting Hours Hunting Datasets! Use awesome-public-datasets Instead

Every data scientist, researcher, and ML engineer knows the soul-crushing reality: you spend 80% of your time finding data, and only 20% actually doing science.

You've been there. Scrolling through broken links on government websites. Wrestling with APIs that rate-limit you into oblivion. Discovering that "open" dataset requires a three-week approval process and a blood sample. The dirty secret of the data world isn't that good datasets are rare—it's that they're scattered across the internet like digital shrapnel, buried in academic papers, hidden in university FTP servers, or locked behind interfaces designed by sadists.

But what if I told you there's a single repository that solves this problem forever? A living, breathing, community-curated goldmine that collects the highest-quality public datasets across every domain imaginable?

Enter awesome-public-datasets—the secret weapon that top-tier researchers at MIT, Stanford, and Shanghai Jiao Tong University have been using for years. Born from OMNILab and now part of the BaiYuLan Open AI community, this isn't just another list. It's the definitive index of topic-centric public data sources, automatically maintained through a sophisticated metadata pipeline. And it's about to transform how you discover data.

Ready to never waste another afternoon on dataset archaeology? Let's dive in.

What is awesome-public-datasets?

awesome-public-datasets is a meticulously curated, topic-organized collection of high-quality open datasets spanning dozens of domains—from Agriculture to Finance, Climate Science to Cybersecurity. Unlike generic search engines or stale government portals, this repository represents crowdsourced intelligence at scale: datasets are collected, verified, and tidied from blogs, research answers, and direct user contributions across the global data community.

The project originated at OMNILab, Shanghai Jiao Tong University, during Xiaming Chen's Ph.D. studies—a research group focused on network intelligence and open data infrastructure. Today, it operates under the BaiYuLan Open AI community, maintaining its academic rigor while scaling to serve hundreds of thousands of developers and researchers worldwide.

What makes this repository genuinely awesome (and worthy of the sindresorhus/awesome badge it proudly displays)? Three critical differentiators:

First, automated curation through apd-core. The repository isn't manually edited—it's automatically generated by the apd-core system, which processes YAML metadata files into the final README. This means contributions follow a structured pipeline: you submit metadata, the system validates it, and the list updates automatically. No more stale links persisting for years.

Second, health indicators for every entry. Each dataset carries a visual status badge—|OK_ICON| for verified, accessible resources or |FIXME_ICON| for entries needing attention. This transparency lets you instantly assess dataset reliability before clicking.

Third, deep metadata integration. Every entry links to its raw YAML configuration in the apd-core repository, showing provenance, update history, and categorization logic. This isn't surface-level linking—it's provenance-aware data discovery.

The repository also maintains an active Slack community for real-time updates, making it a living ecosystem rather than a static dump.

Key Features That Make It Irreplaceable

Let's dissect what elevates awesome-public-datasets above every alternative you've tried:

Topic-Centric Organization

Datasets aren't dumped alphabetically or by arbitrary tags. They're grouped by domain expertise: Agriculture, Biology, Chemistry, Climate+Weather, ComplexNetworks, ComputerNetworks, CyberSecurity, DataChallenges, EarthScience, Economics, Education, Energy, Entertainment, Finance, GIS, and dozens more. This mirrors how researchers actually think—when you're building a climate model, you want all climate data, not data scattered between "C" for "climate" and "W" for "weather."

Quality-First Curation

The "high quality" mandate isn't marketing fluff. The maintainers explicitly filter for datasets that are: actively maintained, properly documented, legally unencumbered (mostly free, with exceptions clearly noted), and structurally sound. The FIXME_ICON system creates accountability—problematic entries are flagged, not hidden.

Automated Maintenance Pipeline

The apd-core generation system eliminates human error in list maintenance. Contributors submit YAML files like this structure:

# Example metadata structure from apd-core
- title: "Global Power Plant Database"
  description: "Comprehensive, open source database of power plants"
  url: "http://datasets.wri.org/dataset/globalpowerplantdatabase"
  category: Energy
  status: OK

The core engine renders these into the final README, ensuring consistent formatting, working links, and automatic validation.

Massive Scale with Surgical Precision

With 2000+ datasets across 40+ categories, the repository balances breadth with discoverability. You're not drowning in noise—you're navigating a hierarchical taxonomy designed by people who understand both data science and information architecture.

Academic Provenance & Community Governance

Originating from university research and transitioning to community-driven maintenance, the project combines institutional credibility with open-source agility. The BaiYuLan Open AI community provides sustainable governance without corporate capture.

6 Concrete Use Cases Where This Repository Shines

1. Climate Research & Environmental Modeling

Need to validate your temperature prediction model? The Climate+Weather section delivers NOAA SURFRAD radiation datasets, WorldClim global climate data, European Climate Assessment & Dataset, and NASA Global Imagery Browse Services. Instead of navigating five different government portals, you get unified access to century-scale observations and satellite-derived products.

2. Genomics & Bioinformatics Pipelines

Building variant calling workflows? The Biology category contains 1000 Genomes, ENCODE, TCGA via Broad GDAC, UniProt, NCBI Taxonomy, and Sanger COSMIC. These aren't just links—they're the foundational references that power publications in Nature and Cell.

3. Network Science & Graph Analytics

Analyzing citation patterns or social networks? ComplexNetworks provides Stanford Large Network Dataset Collection (SNAP), AMiner Citation Networks, DBLP citations, and protein-protein interaction networks. These are the benchmark datasets that every graph neural network paper cites.

4. Cybersecurity Threat Detection

Training intrusion detection systems? The CyberSecurity section includes CCCS-CIC-AndMal-2020 (400K malware/benign samples) and cyber defense exercise traffic logs. These production-scale datasets are typically locked behind institutional firewalls—here, they're openly accessible.

5. Energy Consumption Forecasting

Developing non-intrusive load monitoring? Access AMPds (Almanac of Minutely Power), UK-DALE, REDD, ECO, and PUDL (Public Utility Data Liberation Project). These span residential, commercial, and grid-scale measurements across multiple countries.

6. Economic Policy Analysis

Modeling trade flows or productivity trends? Economics offers Penn World Table, World Input-Output Database, UN Comtrade, Maddison Project, and Atlas of Economic Complexity. These are the datasets behind World Bank reports and IMF forecasts.

Step-by-Step: Installation, Setup & Contribution Workflow

While awesome-public-datasets is primarily a reference repository (not a software package), integrating it into your workflow—and contributing back—requires understanding its architecture.

Step 1: Clone and Navigate

# Clone the main repository for browsing
git clone https://github.com/awesomedata/awesome-public-datasets.git
cd awesome-public-datasets

# Or clone the core engine for contributions
git clone https://github.com/awesomedata/apd-core.git
cd apd-core

Step 2: Understanding the Directory Structure

apd-core/
├── core/                          # Main metadata directory
│   ├── Agriculture/               # Category folders
│   │   ├── Global-dataset-of-historical-yields-for-major-crops.yml
│   │   └── Lemon-Dataset.yml
│   ├── Biology/
│   ├── Climate+Weather/
│   └── ... (40+ categories)
├── deploy/                        # Build assets (icons, etc.)
├── CONTRIBUTING.md                # Contribution guidelines
└── README.md                      # Local development docs

Step 3: Local Preview (For Contributors)

# Install dependencies for the core generator
pip install -r requirements.txt  # if available

# Or use the provided generation scripts
python -m core.generate  # Renders README from YAML sources

Step 4: Contributing a New Dataset

CRITICAL: Never edit awesome-public-datasets/README.md directly. Instead:

# Fork apd-core, then create your metadata file
cat > core/EarthScience/my-new-dataset.yml << 'EOF'
---
title: "My New Satellite Dataset"
description: "High-resolution multispectral imagery for coastal monitoring"
url: "https://example.com/dataset"
category: EarthScience
status: OK
license: CC-BY-4.0
updated_at: 2024-01-15
EOF

# Submit via pull request following CONTRIBUTING.md guidelines

Step 5: Join the Community

# Get instant updates on new datasets via Slack
# Request invite at: https://join.slack.com/t/awesomedataworld/shared_invite/zt-dllew5xy-PJYi~mWUdY3hupohbmVZsA

REAL Code Examples: Working with Datasets from the Repository

Let's extract and explain actual patterns from the repository's structure and documentation, showing how to programmatically interact with these resources.

Example 1: Parsing the YAML Metadata for Automated Discovery

The repository's apd-core engine uses YAML files as the single source of truth. Here's how to build your own dataset discovery tool:

import yaml
import glob
from pathlib import Path

def load_dataset_metadata(core_path="apd-core/core"):
    """
    Load all dataset metadata from apd-core YAML files.
    This mirrors how the README generator builds the final list.
    """
    datasets = []
    
    # Recursively find all .yml files in category directories
    for yml_file in Path(core_path).rglob("*.yml"):
        with open(yml_file, 'r') as f:
            try:
                data = yaml.safe_load(f)
                if data:  # Skip empty files
                    # Extract category from directory structure
                    category = yml_file.parent.name
                    data['category'] = category
                    data['source_file'] = str(yml_file)
                    datasets.append(data)
            except yaml.YAMLError as e:
                print(f"Parse error in {yml_file}: {e}")
    
    return datasets

# Usage: Build a searchable index
all_datasets = load_dataset_metadata()
print(f"Loaded {len(all_datasets)} datasets across {len(set(d['category'] for d in all_datasets))} categories")

# Filter for OK-status climate datasets
climate_ok = [
    d for d in all_datasets 
    if d.get('category') == 'Climate+Weather' 
    and d.get('status') == 'OK'
]
print(f"Verified climate datasets: {len(climate_ok)}")

This pattern lets you bypass the README entirely and query the raw metadata for custom applications—building recommendation engines, automated data pipelines, or domain-specific search tools.

Example 2: Direct API Access to a Featured Dataset (Open-Meteo)

The repository lists Open-Meteo as a verified (|OK_ICON|) open-source weather API. Here's actual usage:

import requests

def fetch_weather_forecast(latitude=52.52, longitude=13.41):
    """
    Fetch weather data using Open-Meteo's free API.
    No API key required—ideal for rapid prototyping.
    """
    url = "https://api.open-meteo.com/v1/forecast"
    
    params = {
        "latitude": latitude,
        "longitude": longitude,
        "hourly": ["temperature_2m", "precipitation", "windspeed_10m"],
        "daily": ["temperature_2m_max", "temperature_2m_min", "sunrise", "sunset"],
        "timezone": "auto",
        "forecast_days": 7
    }
    
    response = requests.get(url, params=params, timeout=30)
    response.raise_for_status()  # Raise exception for bad status codes
    
    return response.json()

# Fetch Berlin weather
berlin_weather = fetch_weather_forecast(52.52, 13.41)
print(f"Elevation: {berlin_weather['elevation']}m")
print(f"Timezone: {berlin_weather['timezone']}")

# Extract tomorrow's max temperature
tomorrow_max = berlin_weather['daily']['temperature_2m_max'][1]
print(f"Tomorrow's high: {tomorrow_max}°C")

This exemplifies the repository's value: Open-Meteo is buried among hundreds of weather sources, but the curation process surfaced it as a genuinely free, no-key-required API—critical for reproducible research.

Example 3: Processing USDA Agricultural Data (JSON API)

The USDA NASS County Crop Yields entry provides a static JSON API. Here's production-ready extraction:

import requests
import pandas as pd

def fetch_usda_crop_yields(state="IA", year=2022):
    """
    Fetch county-level crop yield data from USDA NASS.
    Data served via GitHub Pages static JSON—no rate limits.
    """
    base_url = "https://productofamerica.github.io/usda-county-yields/"
    
    # Construct endpoint from repository metadata pattern
    endpoint = f"{base_url}data/{state}/{year}/corn.json"
    
    try:
        response = requests.get(endpoint, timeout=30)
        response.raise_for_status()
        data = response.json()
        
        # Normalize nested JSON to DataFrame
        records = []
        for county in data.get('counties', []):
            records.append({
                'county_fips': county['fips'],
                'county_name': county['name'],
                'yield_bushels_per_acre': county['yield'],
                'harvested_acres': county.get('harvested_acres'),
                'production_bushels': county.get('production'),
                'year': year,
                'state': state
            })
        
        return pd.DataFrame(records)
        
    except requests.exceptions.RequestException as e:
        print(f"Failed to fetch {endpoint}: {e}")
        return pd.DataFrame()

# Analyze Iowa corn yields
ia_corn = fetch_usda_crop_yields("IA", 2022)
print(f"Records loaded: {len(ia_corn)}")
print(f"Statewide average yield: {ia_corn['yield_bushels_per_acre'].mean():.1f} bu/acre")

# Identify top-performing counties
top_counties = ia_corn.nlargest(5, 'yield_bushels_per_acre')
print("\nTop 5 counties by yield:")
print(top_counties[['county_name', 'yield_bushels_per_acre']].to_string(index=False))

This pattern—static JSON APIs for large government datasets—is a hidden gem surfaced by the repository. No wrestling with USDA's SOAP APIs or bulk download portals.

Example 4: Validating Dataset Health with Repository Icons

The |OK_ICON| and |FIXME_ICON| system encodes operational status. Here's how to programmatically respect these indicators:

import re

def parse_readme_health_status(readme_path="awesome-public-datasets/README.rst"):
    """
    Parse the README to extract dataset health indicators.
    Useful for building trust scores in automated pipelines.
    """
    health_map = {}
    
    with open(readme_path, 'r') as f:
        content = f.read()
    
    # Match pattern: * |ICON| `Title <url>`_ [`Meta <...>`_]
    pattern = r'\*\s+(\|OK_ICON\||\|FIXME_ICON\|)\s+`([^<]+).*?<([^>]+)>'
    
    for match in re.finditer(pattern, content):
        icon, title, url = match.groups()
        status = 'OK' if 'OK_ICON' in icon else 'FIXME'
        
        health_map[title.strip()] = {
            'status': status,
            'url': url,
            'reliable': status == 'OK'
        }
    
    return health_map

# Build reliability-filtered dataset list
health = parse_readme_health_status()
verified_only = {k: v for k, v in health.items() if v['reliable']}
print(f"Verified datasets: {len(verified_only)} of {len(health)} total")

# Safety check before pipeline execution
def safe_dataset_url(dataset_name, health_map):
    """Only return URL if dataset is verified OK."""
    info = health_map.get(dataset_name)
    if not info:
        raise ValueError(f"Unknown dataset: {dataset_name}")
    if not info['reliable']:
        raise RuntimeError(f"Dataset {dataset_name} needs attention (FIXME status)")
    return info['url']

This defensive programming pattern—trusting but verifying via repository health indicators—prevents pipeline failures from link rot.

Advanced Usage & Best Practices

Pro Tip 1: Slack Integration for Real-Time Alerts

Join the Awesome Data World Slack and set up keyword notifications. New datasets in your domain get pushed to you—no more manual polling.

Pro Tip 2: GitHub Actions for Automated Health Monitoring

Build a workflow that weekly checks FIXME_ICON entries in your domain:

# .github/workflows/dataset-health.yml
name: Dataset Health Check
on:
  schedule:
    - cron: '0 9 * * 1'  # Weekly Monday 9am
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          python scripts/check_fixme_status.py --category "Climate+Weather"

Pro Tip 3: Cross-Reference with Academic Provenance

When evaluating datasets, click through to the Meta YAML links. These reveal:

• Original submitter and submission date
• Update frequency
• Known issues or deprecation notices
• Alternative access methods

Pro Tip 4: Domain-Specific Subsetting

Don't clone the entire repository. Use sparse checkout for your domain:

git clone --filter=blob:none --no-checkout https://github.com/awesomedata/apd-core.git
cd apd-core
git sparse-checkout init --cone
git sparse-checkout set core/Climate+Weather core/Energy
git checkout

Comparison with Alternatives

The Verdict: Use Kaggle for quick ML competitions and notebooks. Use Google Dataset Search for exploratory discovery with high noise tolerance. Use Data.gov for authoritative US policy data. But when you need curated, domain-organized, health-verified datasets with full provenance, awesome-public-datasets is unmatched.

FAQ: Developer & Researcher Concerns

Q1: Is awesome-public-datasets actually maintained, or is it abandoned like so many "awesome" lists?

A: Actively maintained. The repository uses automated generation via apd-core, with the Slack community providing real-time human oversight. The FIXME_ICON system creates explicit accountability—unlike static lists where broken links persist for years.

Q2: Can I use these datasets commercially?

A: Most are free, some are not—the repository explicitly notes this. Always verify the license on the source page. The YAML metadata increasingly includes license fields, but legal due diligence remains your responsibility.

Q3: How do I contribute a dataset I discovered?

A: Never edit the README directly. Fork apd-core, create a YAML file in the appropriate core/ subdirectory, and submit a PR following CONTRIBUTING.md.

Q4: What if a dataset link is broken?

A: Check if it's marked |FIXME_ICON|—the maintainers may already know. If not, file an issue on apd-core or submit a PR with updated metadata. The automated system will propagate fixes to the main README.

Q5: Is there an API or programmatic interface?

A: Not officially, but the YAML metadata in apd-core serves as a machine-readable API. Parse it directly (see Example 1 above) or use GitHub's API to query the repository contents.

Q6: How does this compare to paying for data providers like Bloomberg or Refinitiv?

A: For research, prototyping, and non-real-time analysis, open datasets often suffice. For low-latency trading data, exclusive surveys, or compliance-critical applications, commercial providers remain necessary. Use awesome-public-datasets to de-risk your methodology before purchasing.

Q7: Can I trust datasets marked |OK_ICON| to be permanently available?

A: No external link is permanent, but OK status indicates recent verification. Build defensive pipelines: cache downloads, version-pin datasets, and monitor FIXME transitions in your domain.

Conclusion: Your Data Discovery Problem Is Solved

The brutal truth? Dataset discovery is no longer your bottleneck. For years, researchers have wasted countless hours on fragmented searches, broken portals, and dead links. The awesome-public-datasets repository—born from rigorous academic research at Shanghai Jiao Tong University, refined through automated curation, and sustained by a global community—eliminates this friction entirely.

This isn't just a list. It's a provenance-aware, health-monitored, continuously updated infrastructure for open data discovery. Whether you're training climate models on NOAA datasets, analyzing protein interactions with STRING data, or forecasting energy consumption with UK-DALE, the repository surfaces verified sources faster than any alternative.

My recommendation? Star the repository now. Clone apd-core and build your domain-specific metadata parser. Join the Slack community for real-time updates. And most importantly—stop searching, start building.

The data you need is already curated. The only question is whether you'll use it.

→ Explore awesome-public-datasets on GitHub

→ Join the Awesome Data World Slack Community

→ Contribute via apd-core