Stop Scraping Blindly! This Prediction Market Framework Has 36GB of Ready Data

What if I told you that every prediction market researcher you admire is secretly wasting 80% of their time on something a single command could solve?

Here's the brutal truth: if you're analyzing Polymarket or Kalshi data in 2025, you're probably building yet another fragile scraper, fighting rate limits at 2 AM, or discovering that your "complete" dataset has gaping holes where critical trades should be. I've been there. You've been there. The cycle of collect, break, fix, repeat has claimed more promising research projects than peer review ever will.

But what if the largest publicly available dataset of prediction market data was already sitting there, waiting for you? What if you could skip the data engineering nightmare and jump straight to the analysis that actually matters?

Enter prediction-market-analysis — a framework so ruthlessly efficient that it makes traditional data collection look like carving spreadsheets with a stone axe. Created by Jon Becker, this isn't just another GitHub repo with promises. It's a battle-tested system with pre-collected datasets, intelligent indexers, and an extensible analysis engine that transforms raw market chaos into publication-ready figures and statistics. Whether you're a quantitative researcher, a DeFi analyst, or a data scientist hunting for alpha in crowd wisdom, this framework is about to become your secret weapon. And trust me — once you see what's inside that 36GB compressed archive, you'll never look at prediction market research the same way again.

What Is prediction-market-analysis?

prediction-market-analysis is a comprehensive Python framework designed for collecting, storing, and analyzing prediction market data from two of the most significant platforms in the space: Polymarket and Kalshi. Created by Jon Becker, this open-source project represents a paradigm shift in how researchers approach prediction market analysis — moving from fragmented, self-built tooling to a unified, extensible ecosystem.

The framework's crown jewel? The largest publicly available dataset of Polymarket and Kalshi market and trade data, weighing in at a substantial 36GB compressed (and significantly larger when extracted). This isn't a toy dataset or a sample — it's the real deal, collected through sophisticated indexers that pull from both APIs and blockchain sources.

But why is this trending now? The timing is everything. Prediction markets have exploded from niche crypto curiosities to mainstream financial instruments, with Polymarket alone processing billions in volume during high-profile events. Researchers, journalists, and quantitative traders are scrambling to understand crowd wisdom, market microstructure, and the very nature of information aggregation. Meanwhile, Kalshi has opened legally regulated prediction markets to US retail traders, creating entirely new datasets that barely existed five years ago.

The framework sits at this intersection of exploding data availability and desperate researcher demand. Instead of every PhD student and hedge fund analyst building parallel infrastructure, Becker's framework offers a communal foundation — one that's already being cited in serious academic research. The repository includes references to multiple 2026 papers leveraging this exact dataset, from microstructure analysis to crowd wisdom decomposition. This isn't hobbyist code; it's infrastructure for a growing field.

Key Features That Separate Amateurs from Pros

Let's dissect what makes this framework genuinely powerful — not just in marketing slides, but in the trenches of real research.

Pre-Collected Datasets at Scale The 36GB compressed dataset isn't merely large; it's strategically comprehensive. It encompasses market metadata, complete trade histories, and blockchain-derived records that would take months to accumulate independently. The Parquet-based storage format is a deliberate, professional choice — columnar compression means lightning-fast analytical queries without the bloat of raw CSVs or the complexity of a full database setup.

Dual-Source Data Collection The framework's indexers don't rely on a single point of failure. For Polymarket, it combines API access with direct blockchain indexing — critical because on-chain data is the ultimate source of truth for a decentralized platform. Kalshi integration leverages their official API with robust handling of rate limits and pagination. This dual approach means your research isn't hostage to a single API change or outage.

Automatic Progress Persistence Here's where experience shows. The indexers save progress automatically, enabling interruptible, resumable data collection. Anyone who's watched a week-long scrape fail at 99% knows this pain. This framework treats your time as valuable — because Becker clearly learned these lessons the hard way so you don't have to.

Extensible Analysis Architecture The src/analysis/ directory isn't a random collection of scripts. It's organized by platform (kalshi/, polymarket/) with a common interface layer, meaning you can write platform-specific analyses that still leverage shared utilities. Output generation supports PNG, PDF, CSV, and JSON — whatever your publication pipeline demands.

Production-Grade Tooling The use of uv for dependency management, make for task orchestration, and zstd compression for distribution reveals a maintainer who understands modern Python development. This isn't pip-install chaos; it's reproducible, fast, and maintainable.

Real-World Use Cases Where This Framework Dominates

Academic Research on Market Microstructure Becker's own 2026 paper, The Microstructure of Wealth Transfer in Prediction Markets, demonstrates the framework's capability for serious economic research. The granular trade data enables analysis of price impact, liquidity dynamics, and trader behavior at resolutions impossible with aggregated data. If you're studying how information incorporates into prices, this dataset is a goldmine.

Crowd Wisdom & Calibration Studies The referenced paper Decomposing Crowd Wisdom: Domain-Specific Calibration Dynamics shows how prediction market data can test whether crowds are actually wise — or just lucky. With complete market lifecycles and resolution outcomes, you can measure calibration curves, identify systematic biases, and test interventions. Political scientists, psychologists, and economists are all finding applications here.

Quantitative Trading Strategy Development For the pragmatically minded: prediction markets exhibit inefficiencies that systematic strategies can exploit. The complete trade history enables backtesting of market-making, momentum, and mean-reversion strategies. The blockchain data for Polymarket is particularly valuable — you can analyze gas optimization, MEV exposure, and on-chain flow that precedes price movements.

Journalistic & Policy Analysis When prediction markets become election forecasting tools, understanding their dynamics becomes civically important. The framework enables transparency analysis — who trades, when, and with what information? The Kalshi data, in particular, offers insights into legally regulated prediction markets that policymakers are actively debating.

Educational & Reproducible Research Perhaps most valuably, this framework solves the replication crisis in prediction market research. Instead of "we scraped data but can't share it," researchers can point to a common dataset with known provenance. Students can reproduce seminal findings without building infrastructure from scratch.

Step-by-Step Installation & Setup Guide

Ready to stop reading and start analyzing? Here's your complete setup path.

Prerequisites

You'll need Python 3.9+ and uv installed. uv is a modern Python package manager that's significantly faster than pip — the framework's choice of tooling signals its commitment to performance.

# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

Clone and Install

# Clone the repository
git clone https://github.com/Jon-Becker/prediction-market-analysis.git
cd prediction-market-analysis

# Sync dependencies with uv
uv sync

The uv sync command reads pyproject.toml and uv.lock to create a reproducible environment. No dependency hell, no "works on my machine."

Download the Dataset

This is where the magic happens. That 36GB of compressed data — the largest public prediction market dataset in existence:

make setup

This single command downloads data.tar.zst from Cloudflare R2 Storage and extracts it to data/. The extraction reveals the organized structure:

data/
├── kalshi/
│   ├── markets/      # Market metadata and descriptions
│   └── trades/       # Complete trade execution records
└── polymarket/
    ├── blocks/       # Blockchain-derived transaction data
    ├── markets/      # Market metadata and resolution states
    └── trades/       # On-chain and off-chain trade records

Pro tip: Ensure you have ~100GB free before extraction. The compressed 36GB expands significantly, and Parquet's efficiency is in query speed, not disk usage.

Verify Your Setup

# Check data directory structure
ls -la data/

# Verify Parquet files are readable
uv run python -c "import pyarrow.parquet as pq; print(pq.ParquetFile('data/polymarket/markets/[your-file].parquet').schema)"

REAL Code Examples from the Repository

Let's examine actual code patterns from the framework, with detailed explanations of how they work in practice.

Example 1: Running the Interactive Indexer

The data collection process is orchestrated through a clean Makefile interface:

make index

This command launches an interactive menu system that lets you select which indexer to execute. Behind the scenes, this is running Python scripts in src/indexers/ with platform-specific implementations. The beauty is in the abstraction — you don't need to remember which script collects what, or which API credentials are needed. The menu handles discovery and execution.

The indexer architecture separates concerns elegantly:

src/indexers/
├── kalshi/           # Kalshi API client with rate limit handling
│   ├── markets.py    # Market metadata collection
│   └── trades.py     # Trade history pagination and storage
└── polymarket/       # Dual-source collection
    ├── api/          # REST API fallback methods
    └── blockchain/   # Direct Polygon network indexing

Critical implementation detail: The Polymarket blockchain indexer is essential because Polymarket's smart contracts on Polygon contain trade data that may differ from API snapshots. By indexing both, the framework ensures data completeness and verifiability — a requirement for serious research.

Example 2: Executing Analysis Scripts

Once data is collected, analysis follows the same interactive pattern:

make analyze

This presents a menu of available analyses from src/analysis/. The framework's design enables two execution modes:

# Run all analyses (batch mode for publication pipelines)
make analyze-all

# Or interactively select specific analyses
make analyze
# → Select from menu:
#   1. kalshi/volume_distribution
#   2. kalshi/price_efficiency
#   3. polymarket/liquidity_curves
#   4. polymarket/trader_profitability
#   ...

Each analysis script inherits from a common base class defined in src/common/, ensuring consistent output formatting, error handling, and progress reporting. The output directory structure mirrors the analysis organization:

output/
├── kalshi/
│   ├── volume_distribution.png
│   ├── volume_distribution.pdf
│   └── summary_statistics.csv
└── polymarket/
    ├── liquidity_curves.png
    └── trader_pnl_analysis.json

Example 3: Data Packaging for Distribution

When you need to share your augmented dataset or archive completed work:

make package

This executes a zstd-compressed tar creation:

# Equivalent manual command for understanding:
tar --zstd -cf data.tar.zst data/
rm -rf data/  # Clean up after packaging

The zstd compression choice is technically significant. Compared to gzip, zstd offers:

• 3-5x faster compression at comparable ratios
• Significantly faster decompression (critical for large datasets)
• Dictionary support for even better ratios on similar files

For a 36GB dataset that researchers will download repeatedly, this matters enormously. Becker's tooling choices consistently reveal production experience.

Example 4: Custom Analysis Script Structure

Based on the documented architecture in docs/ANALYSIS.md, a custom analysis follows this pattern:

# src/analysis/custom/my_analysis.py
from common.analysis_base import AnalysisBase  # Shared infrastructure
import polars as pl  # Fast DataFrame library for large datasets

class MyAnalysis(AnalysisBase):
    """Custom analysis demonstrating the framework's extensibility."""
    
    def load_data(self):
        # Leverage Parquet's columnar efficiency
        self.markets = pl.read_parquet("data/polymarket/markets/*.parquet")
        self.trades = pl.read_parquet("data/polymarket/trades/*.parquet")
    
    def compute(self):
        # Your analysis logic here
        # Polars lazy evaluation handles large datasets efficiently
        self.results = self.trades.group_by("market_id").agg([
            pl.col("size").sum().alias("total_volume"),
            pl.col("price").mean().alias("avg_price")
        ])
    
    def visualize(self):
        # Automatic output to output/ directory
        self.save_figure(self.results.plot.bar())
        self.save_csv(self.results)

The AnalysisBase class handles output path generation, figure formatting, and metadata logging. You focus on the analysis, not the plumbing.

Advanced Usage & Best Practices

Incremental Collection Strategy Don't reindex everything daily. The progress-saving mechanism means you can run make index on a cron schedule, collecting only new data since interruption. For Polymarket blockchain data, this is essential — the Polygon chain grows continuously, and full reindexing is wasteful.

Memory Management for Large Analyses That 36GB dataset can overwhelm naive pandas operations. The framework's Parquet format enables out-of-core processing — use Polars lazy evaluation or PyArrow datasets to process chunks without loading everything into RAM. For truly massive operations, consider DuckDB for SQL queries directly on Parquet files.

Custom Indexer Development The src/common/ directory provides interfaces for extending to new platforms. If you're researching a smaller prediction market (think: Manifold, Metaculus, or emerging DeFi platforms), implement the IndexerBase interface and plug into the existing pipeline.

Version Control Your Analyses The framework separates data (data/, gitignored) from code (src/, versioned). Commit your analysis scripts, not your outputs. This enables reproducible research where others can rerun your exact analysis on the same dataset.

Citation & Collaboration The repository actively encourages academic use. If you publish with this data, reach out to Becker — the README's research section is updated with citations. This builds the field's collective knowledge and establishes provenance for your work.

Comparison with Alternatives

The verdict is stark: unless you have highly specialized data needs not covered by Polymarket/Kalshi, building from scratch is technical debt you don't need. The platform APIs are valuable for real-time applications, but historical research demands the completeness that only systematic indexing achieves.

FAQ: What Developers Actually Ask

Is the 36GB dataset really necessary, or can I start smaller? The make setup downloads the complete archive, but Parquet's columnar format means you can query subsets efficiently. Start with specific market categories or date ranges in your analysis code — you don't need to load everything into memory.

How current is the pre-collected data? The dataset is periodically updated, but for bleeding-edge research, run make index to collect the latest data. The framework's progress saving means you only fetch what's new.

Can I use this for commercial trading strategies? The dataset's licensing isn't explicitly restrictive, but verify the repository's LICENSE file. The research focus suggests academic use is primary; commercial applications should confirm compliance with platform terms of service.

What's the difference between Polymarket API and blockchain data? The API provides convenient, structured endpoints but may have gaps or delays. Blockchain data is the immutable source of truth but requires decoding contract events. The framework collects both for verification and completeness.

How do I cite this in my research? The README includes a research section with Becker's contact information. Academic papers using this dataset are actively collected and displayed — reach out via email or Twitter to be included.

Is uv mandatory, or can I use pip? While uv is recommended for speed and reproducibility, the pyproject.toml should work with standard pip. However, uv is genuinely faster for this scale of dependency resolution — worth adopting.

What about data privacy or PII concerns? Prediction market data is inherently public — trades occur on public blockchains or regulated exchanges. The framework collects only market and transaction data, not user identities beyond public addresses.

Conclusion: Your Research Deserves Better Infrastructure

Here's what I've learned after years in quantitative research: the quality of your questions matters infinitely more than your data collection prowess. Yet too many brilliant researchers burn their cognitive budget on infrastructure that should be commoditized.

prediction-market-analysis is that commoditization done right. It's not just code — it's collective infrastructure that lets researchers focus on what they do best: asking penetrating questions and finding surprising answers. The 36GB dataset, the dual-source indexers, the extensible analysis framework — these aren't features to impress in a demo. They're force multipliers for actual discovery.

I've seen too many promising projects die in the gap between "I have an idea" and "I have clean data." This framework bridges that gap. Whether you're studying market microstructure, testing theories of crowd wisdom, or hunting for predictive signals, you're standing on shoulders that didn't need to exist — except that Jon Becker built them anyway.

Stop building scrapers. Start doing research.

Clone the repository, run make setup, and join the growing community of researchers who've discovered what happens when infrastructure gets out of your way. The dataset is waiting. The analyses are waiting. What will you discover?

→ Get prediction-market-analysis on GitHub