Stop Building Knowledge Bases Manually: sift-kg Does It in 2 Minutes

You've been there. Staring at a folder stuffed with PDFs, research papers, court filings, or competitor reports — knowing the connections are in there somewhere, but drowning in the sheer volume. You've tried Notion. You've wrestled with Obsidian. You've spent weekends manually tagging, linking, and organizing. Two years later, your "knowledge base" is a graveyard of half-linked notes you'll never revisit.

What if I told you there's a tool that transforms that document chaos into a browsable, interactive knowledge graph — in minutes, not years? No coding. No database setup. No infrastructure headaches. Just a CLI, your documents, and the raw power of modern LLMs.

Meet sift-kg, the open-source tool that's making manual knowledge management look like a relic from the dial-up era. This isn't another note-taking app. It's a pipeline that extracts entities, maps relationships, and builds persistent structured memory that both you and your AI agents can actually use. Whether you're a researcher mapping theoretical landscapes, an investigator tracing financial flows, or a developer building AI systems that need contextual understanding — sift-kg is about to change how you work with information forever.

Curious? You should be. Let's pull back the curtain on why developers and researchers are abandoning their manual workflows and flocking to this tool.

What is sift-kg?

sift-kg is an open-source CLI tool created by Juan Ceresa that transforms any collection of documents into a structured knowledge graph. Born from the frustration of watching smart people waste months on manual knowledge organization, it leverages large language models to automate what used to be painstaking human labor — while keeping you firmly in control.

The tool sits at a fascinating intersection of natural language processing, graph theory, and human-computer collaboration. Unlike rigid extraction systems that force your data into predefined boxes, sift-kg uses a schema-free approach by default: one LLM call samples your documents and designs a custom schema tailored to your specific corpus. The result? A knowledge graph that actually reflects how your information connects, not how some engineer in 2019 thought it should.

But here's what makes it genuinely trend-worthy: sift-kg is infrastructureless. No Docker containers to wrestle. No PostgreSQL instances to provision. No cloud services to configure. It runs entirely on your machine, processes 75+ document formats locally, and outputs everything to standard formats you already know — JSON, GraphML, SQLite, CSV. Your documents never leave your system unless you explicitly choose cloud OCR.

The project has gained serious traction in research, investigative journalism, legal tech, and AI agent development circles. Its live demos showcase real graphs generated from academic papers and news articles — 425 entities from 12 foundational AI papers for about $0.72 in API costs, or 431 entities mapping the FTX collapse from just 9 articles. That's the kind of efficiency that makes traditional knowledge management approaches weep.

Key Features That Separate sift-kg from the Pack

Zero-Config Start with Deep Customization

Point sift-kg at a folder and run. That's it. The tool auto-discovers your document schema, extracts entities and relationships, and builds your graph. For power users, drop a sift.yaml in your project for persistent, version-controlled configuration. The flexibility scales from "I need this done in 5 minutes" to "I'm building a production knowledge pipeline."

Universal LLM Compatibility

Locked into OpenAI? Fine. Prefer Anthropic's Claude? Works perfectly. Running Ollama locally for complete privacy? Fully supported. sift-kg uses LiteLLM under the hood, meaning any compatible provider works — including Mistral, local models, and emerging services. Your API keys, your choice, your control.

Human-in-the-Loop Entity Resolution

This is where sift-kg gets seriously clever. The LLM proposes entity merges — "Steph Curry" and "Stephen Curry" are probably the same person — but nothing merges without your approval. An interactive terminal UI lets you review each proposal with confidence scores and reasoning. For high-stakes use cases like genealogy or legal review, edit the YAML directly and study every decision. The tool respects that you are the final authority on what constitutes identity in your domain.

Interactive Browser Viewer with Focus Mode

The built-in visualization isn't a static PNG — it's a force-directed, explorable graph with community regions, hover previews, and keyboard navigation. Double-click any entity to enter focus mode: isolate its neighborhood, step through connections with arrow keys, and trace your exploration via a persistent trail breadcrumb. It's the difference between looking at a map and actually walking the territory.

AI Agent-Ready Knowledge Base

Here's the secret weapon: the same graph that powers your visualizations serves as structured persistent memory for AI agents. Commands like sift topology, sift query, and sift search --json output structured data that agents can consume directly. The bundled skill at .agents/skills/sift-kg/SKILL.md teaches agents session orientation, entity exploration, and grounded reasoning. Your AI stops starting from zero every conversation.

Budget Controls and Local Execution

Set --max-cost to cap LLM spending. Run entirely offline with local OCR (Tesseract, EasyOCR, PaddleOCR) and local LLMs via Ollama. For the paranoid and the privacy-conscious, this is non-negotiable — and sift-kg delivers.

Real-World Use Cases Where sift-kg Dominates

Research & Education: Mapping the Intellectual Landscape

Feed sift-kg a corpus of academic papers and watch it construct a living map of how theories, methods, and findings interconnect. The academic domain distinguishes abstract concepts (THEORY, METHOD) from concrete artifacts (SYSTEM — GPT-2, BERT, GLUE), showing where ideas support, contradict, or extend each other. Literature reviews that took weeks now take hours. Students get concept maps that actually reflect knowledge structure, not just citation counts.

Investigative Journalism & OSINT: Following the Money

The osint domain ships with entity types for shell companies, financial accounts, and offshore jurisdictions — plus relation types tracing beneficial ownership and transaction flows. Drop in FOIA releases, court filings, or leaked documents. The graph reveals hidden connections that scattered reading misses. The FTX demo (431 entities from 9 articles) shows how quickly complex financial fraud structures emerge when visualized as relationships rather than linear text.

Legal Review: Connecting Entities Across Document Collections

Legal work demands precision, which is why sift-kg's human-in-the-loop merging is crucial. Extract parties, agreements, properties, and obligations across thousands of pages. Review every proposed merge manually. Export to formats compatible with existing legal tech stacks. The source provenance feature — every extraction links to exact document and passage — provides the audit trail courts require.

Genealogy: Tracing Family Relationships Across Vital Records

Family history research is uniquely sensitive to false merges. "John Smith" isn't one person — it's hundreds. sift-kg's embedding-based clustering catches semantic similarities across spelling variants, while the manual review workflow ensures you don't conflate distinct individuals. The iterative resolution process (resolve → review → apply-merges → repeat) handles the messy reality of historical records.

Business Intelligence: Competitive Landscape Mapping

Drop competitor whitepapers, market reports, and internal documents into sift-kg. The schema discovery identifies what entities matter in your industry — perhaps PRODUCTS, PARTNERSHIPS, MARKETS, REGULATIONS — and maps how they interconnect. Spot white space, identify partnership networks, track regulatory exposure across your competitive set.

Step-by-Step Installation & Setup Guide

Prerequisites

You'll need Python 3.11 or newer. Check your version:

python --version

Basic Installation

Install sift-kg from PyPI:

pip install sift-kg

That's the core package. For additional capabilities, install optional extras as needed.

OCR Support (Scanned PDFs & Images)

For local OCR via Tesseract (recommended, no cloud dependencies):

# macOS
brew install tesseract

# Ubuntu/Debian
sudo apt install tesseract-ocr

Then use the --ocr flag during extraction:

sift extract ./documents/ --ocr

For Google Cloud Vision as an alternative OCR backend:

pip install sift-kg[ocr]
# Requires GCV credentials configured
sift extract ./documents/ --ocr --ocr-backend gcv

Semantic Clustering for Entity Resolution (Optional)

Improves duplicate detection across spelling variants (~2GB download for PyTorch):

pip install sift-kg[embeddings]

Enable during resolution:

sift resolve --embeddings

Development Installation

To hack on sift-kg itself:

git clone https://github.com/juanceresa/sift-kg.git
cd sift-kg
pip install -e ".[dev]"

Project Initialization

Create a new project configuration:

sift init

This generates two files:

• sift.yaml — project configuration (domain, model, OCR settings)
• .env.example — template for API keys

Copy and configure your environment:

cp .env.example .env

Edit .env with your preferred provider:

# OpenAI
SIFT_OPENAI_API_KEY=sk-...

# Anthropic
SIFT_ANTHROPIC_API_KEY=sk-ant-...

# Mistral
SIFT_MISTRAL_API_KEY=...

# Ollama (local, no API key needed — just ensure Ollama is running)

Configuration Hierarchy

Settings apply in this priority (highest first):

1. CLI flags
2. Environment variables
3. .env file
4. sift.yaml
5. Built-in defaults

This means you can override anything in sift.yaml with a quick flag for one-off commands.

REAL Code Examples from the Repository

Let's walk through the actual commands and code patterns from sift-kg's documentation, with detailed explanations of what's happening under the hood.

Example 1: Complete CLI Pipeline — From Documents to Interactive Graph

This is the bread-and-butter workflow that transforms a folder of documents into an explorable knowledge graph:

# Step 1: Initialize project structure
sift init                           # create sift.yaml + .env.example

# Step 2: Extract entities and relationships from all documents
sift extract ./documents/           # extract entities & relations via LLM

# Step 3: Build the knowledge graph from extractions
sift build                          # construct NetworkX graph, auto-dedup trivial variants

# Step 4: Find potential duplicate entities
sift resolve                        # LLM proposes merges, saves to merge_proposals.yaml

# Step 5: Review proposed merges interactively
sift review                         # approve/reject merges in terminal UI

# Step 6: Apply your decisions to the graph
sift apply-merges                   # rewire graph with confirmed merges

# Step 7: Generate narrative summary
sift narrate                        # prose report with relationship chains and timelines

# Step 8: Launch interactive browser viewer
sift view                           # force-directed graph with focus mode and trail breadcrumbs

What's happening here? The pipeline decomposes a complex NLP task into discrete, inspectable stages. extract calls your LLM with batched document chunks, returning structured JSON. build assembles these into a NetworkX graph with deterministic pre-deduplication. resolve uses the LLM's semantic understanding to identify likely duplicates across the full corpus. review puts you in control. apply-merges performs surgical graph surgery — rewiring edges, combining source documents, removing absorbed nodes. The result is a clean, authoritative graph that reflects human judgment, not just statistical patterns.

Example 2: Domain-Specific Extraction with Bundled Schemas

sift-kg ships with specialized domains for common use cases. Here's how to leverage them:

# List available bundled domains
sift domains

# Extract using the OSINT domain (investigations, financial tracing)
sift extract ./foia-documents/ --domain-name osint

# Extract using the academic domain (papers, literature reviews)
sift extract ./research-papers/ --domain-name academic

Configure permanently in sift.yaml:

# sift.yaml — persistent project configuration
domain: academic                    # or osint, general, schema-free (default)
model: openai/gpt-4o-mini           # any LiteLLM-compatible model
ocr: true                           # enable OCR for scanned PDFs

# Fine-tune extraction behavior
extraction:
  backend: kreuzberg                # 75+ formats via Kreuzberg engine
  ocr_backend: tesseract            # local OCR: tesseract | easyocr | paddleocr | gcv
  ocr_language: eng                 # OCR language code

Why this matters: The academic domain distinguishes CONCEPT from THEORY from SYSTEM — crucial for mapping how abstract ideas relate to concrete implementations. The osint domain knows about SHELL_COMPANY and BENEFICIAL_OWNER_OF — entities and relations that generic NER would miss entirely. Using the right domain is like switching from a general-purpose screwdriver to a precision torque wrench: the same underlying mechanism, but calibrated for your specific task.

Example 3: Python API for Custom Pipelines and Agent Integration

For programmatic use — Jupyter notebooks, web applications, or AI agent backends — sift-kg exposes a clean Python API:

from sift_kg import (
    load_domain, run_extract, run_build, 
    run_resolve, run_narrate, run_export, run_view, run_pipeline
)
from sift_kg import KnowledgeGraph
from pathlib import Path

# Load domain configuration — bundled or custom YAML
domain = load_domain(bundled_name="academic")  # or load_domain(Path("my_domain.yaml"))

# Extract with full control over OCR, backend, and concurrency
results = run_extract(
    Path("./research-papers"),           # source documents
    "openai/gpt-4o-mini",                # model specification
    domain,                              # entity/relation schema
    Path("./output"),                    # output directory
    ocr=True,                            # enable OCR for scanned PDFs
    ocr_backend="tesseract",             # local OCR engine
    extractor="kreuzberg",               # document extraction backend
    concurrency=4,                       # parallel LLM calls
    chunk_size=10000,                    # tokens per LLM context window
)

# Build graph with automatic trivial deduplication
kg = run_build(Path("./output"), domain)
print(f"{kg.entity_count} entities, {kg.relation_count} relations")
# Output: 425 entities, 892 relations

# Resolve duplicates with semantic clustering (requires [embeddings] extra)
merges = run_resolve(
    Path("./output"), 
    "openai/gpt-4o-mini",
    domain=domain,
    use_embeddings=True                  # KMeans clustering on sentence embeddings
)

# Export to multiple formats for downstream analysis
run_export(Path("./output"), "sqlite")   # DuckDB, Datasette, SQL queries
run_export(Path("./output"), "graphml")  # Gephi, yEd, Cytoscape
run_export(Path("./output"), "csv")      # Spreadsheet analysis

# Generate narrative or just refresh community labels (cheap)
run_narrate(Path("./output"), "openai/gpt-4o-mini", communities_only=True)

# Launch viewer with pre-filters
run_view(Path("./output"))                                    # full graph
run_view(Path("./output"), neighborhood="person:alice", depth=2)  # ego network
run_view(Path("./output"), top_n=10)                          # hub entities + neighbors

# Or run the complete pipeline in one call
from sift_kg import run_pipeline
run_pipeline(
    Path("./research-papers"),
    "openai/gpt-4o-mini",
    domain,
    Path("./output")
)

The power here: Full programmatic control with sensible defaults. The KnowledgeGraph object exposes entity counts, relation statistics, and graph structure for custom analysis. The run_pipeline convenience function collapses the standard workflow for batch processing. And the export flexibility means your graph integrates with existing data science tools — no vendor lock-in, no proprietary formats.

Example 4: Custom Domain Definition for Controlled Taxonomies

When you need strict schema enforcement — say, tracking only specific departments in a corporate hierarchy:

# domain.yaml — custom domain with closed vocabularies
name: Corporate Structure
fallback_relation: RELATED_TO          # catch-all for undefined relations

entity_types:
  PERSON:
    description: Employees and contractors
    extraction_hints:
      - Look for full names with titles
      - Include email prefixes as aliases
  
  COMPANY:
    description: Business entities in portfolio
  
  DEPARTMENT:
    description: Named departments within companies
    canonical_names:              # CLOSED VOCABULARY — only these exact values
      - Engineering
      - Sales
      - Legal
      - Marketing
      - Research
    canonical_fallback_type: ORGANIZATION  # non-canonical names retyped

relation_types:
  EMPLOYED_BY:
    description: Employment relationship
    source_types: [PERSON]        # type constraint: only PERSON → COMPANY
    target_types: [COMPANY]
  
  OWNS:
    description: Ownership stake
    symmetric: false              # direction matters: A OWNS B ≠ B OWNS A
    review_required: true         # always flag for human review
  
  MANAGES:
    description: Management responsibility
    source_types: [PERSON]
    target_types: [DEPARTMENT, COMPANY]

Use your custom domain:

sift extract ./hr-documents/ --domain path/to/domain.yaml

The enforcement mechanism: The LLM is explicitly instructed to use only defined types. canonical_names are injected into the extraction prompt, ensuring exact matches. Any deviation gets retyped to canonical_fallback_type during graph building. This is schema-as-code for knowledge graphs — version-controlled, reviewable, and reproducible.

Advanced Usage & Best Practices

Pre-Filter Before Visualization

Dense graphs become unreadable. Use CLI flags to scope the viewer:

sift view --neighborhood "Palantir Technologies" --depth 2  # ego network
sift view --top 10                                          # hub entities only
sift view --community "Community 1"                         # thematic cluster
sift view --min-confidence 0.8                              # high-confidence subset

Iterative Entity Resolution

One pass rarely suffices. After merging, new duplicates emerge as the graph simplifies:

sift resolve && sift review && sift apply-merges
# ...inspect graph...
sift resolve && sift review && sift apply-merges  # catch newly apparent duplicates

Budget-Conscious Operation

sift extract ./docs/ --max-cost 5.00   # hard cap on LLM spend
sift narrate --communities-only        # $0.01 vs. full narrative cost

Local-First Privacy

# Zero cloud dependencies
sift extract ./docs/ --ocr --ocr-backend tesseract  # local OCR
# Configure Ollama in .env for local LLM inference

Agent Integration Pattern

# Build once, query repeatedly for agent context
sift topology          # structural JSON for agent orientation
sift query "topic"     # subgraph for focused reasoning
sift search "X" --json # entity lookup with relations

Comparison with Alternatives

When to choose what: Use sift-kg when you need fast, flexible, transparent knowledge extraction with human oversight and AI integration. Stick with manual tools for small, stable personal note collections where the act of organizing is the value. Choose enterprise platforms when you need compliance certifications, multi-user workflows with granular permissions, and have budget for dedicated KG engineering teams.

FAQ

What does sift-kg cost to run?

The tool itself is free (MIT license). LLM costs depend on your provider and corpus size. The transformers demo (12 papers, 425 entities) cost approximately $0.72 with GPT-4o-mini. Set --max-cost to cap spending. Local models via Ollama cost nothing but require capable hardware.

Can I use sift-kg without sending data to OpenAI?

Absolutely. Install Ollama, pull a capable local model (Llama 3, Mistral, etc.), and configure SIFT_OLLAMA_MODEL in your environment. For OCR, use Tesseract, EasyOCR, or PaddleOCR — all local, no cloud. Your documents never leave your machine.

How accurate is the entity extraction?

Accuracy depends on document quality, LLM choice, and domain specificity. The academic and osint domains include extraction hints that improve precision. All extractions include confidence scores. The human-in-the-loop review for merging adds a critical accuracy layer that fully automated systems lack.

What if the LLM hallucinates relationships?

Every extraction links to source document and passage — you can always verify. Low-confidence relations (default <0.7) get flagged for review. Relations marked review_required in your domain config are always human-verified. The system is designed for transparency, not blind trust.

Can I incrementally add documents to an existing graph?

Yes. Extract new documents into the same output directory and rebuild. The graph grows incrementally. Run sift resolve again to catch new duplicates. This persistence is core to the "AI second brain" use case — knowledge that compounds over time.

Is there a hosted version for teams?

The open-source CLI is self-hosted by design. For forensic legal analysis with analyst verification, multi-user review, and LaTeX dossier generation, see Civic Table — a platform built on the sift-kg pipeline with additional vetting layers.

How do I handle very large document collections (10,000+ pages)?

Use sift resolve --embeddings for better duplicate detection batching. Consider chunking into thematic subdirectories and building separate graphs that you later merge. The JSON format is straightforward for custom aggregation scripts.

Conclusion

We've reached an inflection point in how humans and AI systems manage knowledge. The old paradigm — read, highlight, file, forget — is collapsing under information volume. Manual note-taking tools, however beautiful, don't scale. Fully automated extraction, however fast, lacks accountability.

sift-kg threads this needle with rare precision. It automates the tedious (extraction, formatting, visualization) while preserving human judgment where it matters (entity identity, relationship verification). It produces outputs that serve both human exploration and machine consumption. And it does all this without infrastructure, lock-in, or months of setup.

Whether you're mapping research literature, tracing financial fraud, building legal cases, or constructing persistent memory for AI agents — sift-kg transforms document collections from passive archives into active, queryable, growing knowledge structures.

The live demos speak louder than any description. The FTX graph, the transformers concept map — these were generated in minutes from public documents, not curated by hand over months.

Stop organizing knowledge manually. Start building knowledge graphs that compound.

👉 Explore sift-kg on GitHub — install with pip install sift-kg, run sift init, and turn your document chaos into structured intelligence in the next 10 minutes.

Your future self — and your AI agents — will thank you.