Run 235B Models on a Single GPU: Krasis Exposes the Secret

Run 235B Models on a Single GPU: Krasis Exposes the Secret

What if I told you that the $4,000 RTX 5090 sitting in your PC can run language models that normally demand $50,000 worth of server hardware? Not a toy version. Not a stripped-down distill. The actual 235-billion-parameter behemoths that power the most demanding coding, reasoning, and research workflows.

Here's the gut punch: you've been told a lie about GPU memory. The AI industry has convinced developers that running large models requires either cloud APIs or enterprise clusters. Every time you hit a context limit on ChatGPT, every time your local llama.cpp crawl grinds to a halt at 5 tokens per second, every time you abandon a project because "my GPU only has 24GB"—that's not a hardware problem. That's a runtime problem.

Enter Krasis, the hybrid LLM runtime that top developers are quietly deploying to shatter the VRAM ceiling. Built in Rust, orchestrated by Python^{↗ Bright Coding Blog}, and engineered with a singular obsession—efficient execution of massive Mixture-of-Experts models on consumer-grade hardware—Krasis doesn't just "make do" with limited VRAM. It weaponizes your system RAM, PCIe bandwidth, and intelligent quantization into a streaming pipeline that delivers prefill speeds above 2,000 tokens per second on models that physically cannot fit in GPU memory.

If you're still renting A100s or praying that transformers won't OOM on your 4090, stop. This changes everything.

What Is Krasis? The Runtime That Makes Impossible Models Possible

Krasis is a Python-orchestrated Rust LLM runtime created by brontoguana with a laser-focused mission: run 200B+ parameter models on commodity hardware with full GPU prefill and decode. It is not a model downloader. It is not a chat wrapper. It is a systems engineering achievement that rethinks how Mixture-of-Experts (MoE) architectures interact with memory hierarchies.

The name itself signals ambition. "Krasis" derives from a Greek term meaning mixture or blending—fitting for a tool that harmonizes GPU VRAM, system RAM, and disk caches into a unified execution fabric.

Why Krasis Is Trending Now

Three converging forces have made Krasis explosive:

1. The MoE Proliferation: Models like Qwen3-235B-A22B and DeepSeek-V2 use sparse expert activation, where only a fraction of parameters fire per token. Krasis exploits this architectural property with surgical precision.

2. Consumer GPU Power Surge: The RTX 5090's 32GB VRAM and PCIe 5.0 bandwidth create a tantalizing gap—enough compute, not enough memory. Krasis bridges this exact chasm.

3. API Fatigue: Developers are exhausted by rate limits, pricing unpredictability, and data privacy concerns. Krasis offers true local inference at speeds that rival hosted services.

The project's GitHub repository (brontoguana/krasis) has become a nexus for developers who refuse to accept that local = slow. With a dedicated subreddit, active issue tracking, and companion tools like ktop for runtime monitoring, Krasis is maturing from experimental to essential.

Key Features: The Engineering That Makes Magic Real

Krasis isn't magic—it's meticulously optimized systems design. Here's what separates it from every other "run big models locally" attempt:

Hybrid Memory Architecture with Intelligent Streaming

Krasis treats VRAM as a cache, not a container. Models stream through GPU memory during prefill, with algorithms specifically optimized for the attention-heavy prefill phase versus the autoregressive decode phase. This isn't naive offloading; it's phase-aware scheduling that maximizes throughput at each stage.

Automatic Quantization Pipeline

Feed Krasis a BF16 safetensors model from HuggingFace, and it automatically builds INT4 or INT8 caches on disk. The first run incurs compilation overhead, but subsequent launches reuse optimized weights. You trade disk space (roughly BF16 size + 2× quantized size) for the ability to run models that would otherwise demand impossible VRAM.

BF16 Attention with Selective Precision

Krasis preserves attention computation in BF16 while quantizing expert weights aggressively. This quality-preserving strategy maintains model capability where it matters (attention patterns, reasoning chains) while compressing parameter storage where MoE architectures allow it (sparse expert matrices).

Multi-GPU Decode Scaling

The latest pre-release adds multi-GPU support for decode acceleration. While prefill remains single-GPU bound by PCIe bandwidth, decode can distribute across matched GPUs for tangible speedups on multi-card workstations.

OpenAI-Compatible API Server

Krasis exposes http://localhost:8012/v1/chat/completions with SSE streaming. Drop it into Cursor, OpenCode, or any OpenAI SDK client. No refactoring. No wrapper scripts. Just change the base URL and keep building.

Production-Grade Quantization Validation

Perplexity benchmarks on WikiText-2 and C4 validation demonstrate that Krasis's Q4 configurations preserve quality. Qwen3-Coder-Next achieves 7.23 PPL at Q4—competitive with full-precision baselines and far superior to naive quantization approaches.

Use Cases: Where Krasis Destroys the Competition

1. Local Code Intelligence at Scale

Run Qwen3-Coder-Next (80B parameters, 159GB BF16) on a single RTX 5080 16GB with 1,801 tok/s prefill and 26.8 tok/s decode. That's enough throughput for real-time autocomplete, multi-file refactoring suggestions, and complex algorithm generation—without sending proprietary code to external APIs.

2. Research and Model Evaluation

Benchmark massive MoE architectures against private datasets without provisioning cloud instances. Krasis's benchmark suite (benchmarks/benchmark_suite.toml) automates matrix testing across quantization configs, generating markdown^{↗ Smart Converter} reports for reproducible research.

3. Privacy-Critical Enterprise Deployment

Financial services, healthcare, and legal organizations can deploy Qwen3.5-122B-A10B (122B parameters) on internal hardware with zero data exfiltration. The SSPL-1.0 license permits modification and internal use; commercial hosting requires contact.

4. Edge AI Prototyping

Developers prototyping agentic systems or RAG pipelines can iterate locally with production-scale models, then migrate to cloud only for final deployment. Krasis eliminates the "it works on my 7B model but fails at scale" surprise.

Step-by-Step Installation & Setup Guide

Krasis targets Linux (Ubuntu 24.04+) or WSL2 on Windows. macOS is unsupported—this is CUDA-native engineering.

Prerequisites

Step 1: One-Line Installation

# Download and execute the installer—no sudo required
curl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash

This creates ~/.krasis/venv, installs Krasis, symlinks commands to ~/.local/bin, and updates your PATH for the current session.

Step 2: CUDA Dependency Setup

krasis-setup

Installs the CUDA toolkit (may prompt for sudo), PyTorch, and sgl-kernel. Execute once per system.

Step 3: Model Acquisition

# Install HuggingFace CLI if needed
pip install huggingface-hub

# Download DeepSeek-V2-Lite (starter model)
huggingface-cli download deepseek-ai/DeepSeek-V2-Lite \
    --local-dir ~/.krasis/models/DeepSeek-V2-Lite

# Or the flagship Qwen3-Coder-Next (80B, ~148GB download)
huggingface-cli download Qwen/Qwen3-Coder-Next \
    --local-dir ~/.krasis/models/Qwen3-Coder-Next

Critical: Krasis requires BF16 safetensors, not GGUF or AWQ. Download the native HuggingFace checkpoint.

Step 4: Launch

krasis

The TUI guides you through model selection, CPU expert quantization (INT4/INT8), GPU selection, and live VRAM budgeting. First run builds caches; subsequent launches are dramatically faster.

WSL2 Memory Configuration

Windows users must expand WSL's default 50% RAM allocation. Create C:\Users\<You>\.wslconfig:

[wsl2]
memory=120GB

Adjust to leave ~8GB for Windows. Restart with wsl --shutdown from PowerShell.

Maintenance

# Upgrade in-place
curl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash

# Uninstall (preserves models)
curl -sSf https://raw.githubusercontent.com/brontoguana/krasis/main/install.sh | bash -s -- --uninstall

REAL Code Examples: From the Repository

Let's dissect actual patterns from Krasis's documentation, with detailed commentary on what each achieves.

Example 1: Source Installation for Developers

# Install Rust toolchain (required for building from source)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Clone and enter repository
git clone https://github.com/brontoguana/krasis.git
cd krasis

# Create isolated Python environment
python3 -m venv .venv && source .venv/bin/activate

# Editable install: changes to source reflect immediately
pip install -e .

# Install CUDA dependencies
krasis-setup

# Launch interactive TUI
krasis

Why this matters: The editable install (-e) is crucial for contributors modifying Rust extensions. The Rust toolchain compiles performance-critical kernels that Python alone cannot deliver—this hybrid architecture is Krasis's secret sauce.

Example 2: Non-Interactive Deployment

# Reuse configuration from previous TUI session—ideal for systemd services
krasis --non-interactive

# Override specific parameters for automated pipelines
krasis --non-interactive \
    --model-path /path/to/model \
    --num-gpus 2 \
    --benchmark

Why this matters: Production deployments require reproducibility without manual TUI interaction. The --benchmark flag validates performance before accepting traffic—essential for SLAs.

Example 3: Benchmark Suite Configuration

# benchmarks/benchmark_suite.toml
# Define hardware configurations to test
[[config]]
num_gpus = 1
gpu_expert_bits = 4      # INT4 for GPU-resident experts
cpu_expert_bits = 4      # INT4 for CPU-offloaded experts

[[config]]
num_gpus = 2
gpu_expert_bits = 4
cpu_expert_bits = 4

# Define models to evaluate
[[model]]
name = "DeepSeek-V2-Lite"

[[model]]
name = "Qwen3-235B-A22B"
# Pair native model with GGUF for CPU experts
gguf_name = "Qwen3-235B-A22B-GGUF"

Execute the matrix:

# Default suite configuration
krasis --benchmark-suite

# Custom configuration path
krasis --benchmark-suite /path/to/custom.toml

Why this matters: This TOML-driven approach enables systematic exploration of the quantization accuracy/speed tradeoff space. Each combination runs isolated with logs to benchmarks/suite_logs/ and aggregated markdown summaries—research-grade reproducibility with zero manual bookkeeping.

Example 4: Chat Client Integration

# Auto-discover local Krasis servers via ~/.krasis/servers/
krasis-chat

# Connect to specific instance
krasis-chat --port 8012

# Remote server or custom temperature
krasis-chat --url http://host:8012 --temperature 0.3

Interactive commands inside chat:

• /new — clear conversation history
• /system PROMPT — dynamically change system prompt
• /exit — terminate session

Why this matters: The auto-discovery mechanism (~/.krasis/servers/) enables multi-model workflows without port memorization. Temperature override (0.3 for deterministic coding, higher for creative tasks) provides rapid experimentation.

Example 5: OpenAI-Compatible API Consumption

# Any OpenAI SDK client works with base_url change
import openai

client = openai.OpenAI(
    base_url="http://localhost:8012/v1",  # Krasis server
    api_key="not-needed-for-local"        # placeholder, not validated
)

response = client.chat.completions.create(
    model="Qwen3-Coder-Next",  # model identifier from Krasis
    messages=[
        {"role": "system", "content": "You are an expert Rust developer."},
        {"role": "user", "content": "Explain lifetime elision rules."}
    ],
    stream=True  # SSE streaming supported
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Why this matters: Zero-friction migration from OpenAI/Anthropic APIs. The model parameter maps to Krasis's loaded model name. Streaming via Server-Sent Events ensures responsive UX even at 9.3 tok/s decode on 235B models.

Advanced Usage & Best Practices

Disk Cache Management

Krasis stores optimized weights in ~/.krasis/. Monitor growth:

du -sh ~/.krasis/
# Typical: 438GB (BF16) + 110GB (Q4) + 110GB (Q8 cache) = ~658GB for Qwen3-235B

Use fast NVMe SSDs. The cache is read-heavy during inference; SATA bottlenecks destroy prefill performance.

Quantization Strategy Matrix

Never use gpu_expert_bits = 16 in production. This validation mode exists solely for proving quantization correctness against full-precision baselines.

Multi-GPU Decode Tuning

Multi-GPU improves decode throughput, not prefill (PCIe bandwidth bound). Use matched GPUs—mismatched cards cause synchronization stalls. The TUI's live VRAM display prevents overallocation.

System RAM Headroom

Krasis needs 2× quantized size in system RAM. For Qwen3-235B at Q4 (110GB quantized): 220GB RAM minimum. Budget 8GB for OS overhead. ECC RAM recommended for multi-day inference sessions.

Comparison with Alternatives

Krasis wins when: Your model exceeds VRAM, you need MoE efficiency, you want minimal setup friction, or you refuse cloud dependency.

Alternatives win when: Your model fits GPU memory (vLLM/TensorRT are faster), you need maximum throughput on datacenter hardware, or you require permissive licensing for commercial SaaS.

FAQ

Is Krasis free for commercial use?

Krasis is SSPL-1.0 licensed: free to use, modify, and distribute internally. Commercial hosting or managed services require contacting the author.

Why does first launch take so long?

Krasis compiles optimized weight caches from BF16 safetensors to INT4/INT8. This one-time cost (minutes to hours depending on model size) amortizes across all future runs. Subsequent launches load cached weights directly.

Can I use my existing GGUF models?

Partially. Krasis requires BF16 safetensors for the native model path, but can use GGUF files as CPU expert sources via the gguf_name or gguf_path configuration. See benchmark suite TOML example above.

Does Krasis work on AMD or Apple Silicon?

No. Krasis is CUDA-native and optimized for NVIDIA GPUs. ROCm and Metal support are not on the current roadmap.

How does Krasis compare to quantization + llama.cpp?

Krasis's hybrid streaming architecture fundamentally differs from llama.cpp's layer-wise offloading. Krasis achieves 10-20× higher prefill throughput on oversized models by optimizing the GPU↔RAM streaming pattern for MoE's sparse activation patterns.

What models are confirmed working?

Verified: Qwen3.5-35B-A3B, Qwen3-Coder-Next (80B), Qwen3.5-122B-A10B, Qwen3-235B-A22B, DeepSeek-V2-Lite. Additional models may work; report successes/failures via GitHub issues.

Why is my RTX 4090 slower than benchmarked 5090?

PCIe bandwidth and VRAM capacity dominate. The 5090's 32GB enables larger GPU-resident expert caches, reducing streaming overhead. The 4090's 24GB works but with more aggressive CPU offloading.

Conclusion: The Local AI Revolution Is Here—Don't Miss It

Krasis represents a paradigm shift in what's possible with consumer hardware. For years, the narrative was clear: big models need big infrastructure. Rent cloud GPUs. Accept rate limits. Sacrifice privacy. Krasis demolishes that narrative.

With 2,124 tok/s prefill on 235B parameters—streaming through 32GB VRAM like water through a sieve—this runtime proves that architectural innovation trumps raw hardware expenditure. The Rust/Python hybrid delivers systems performance with Python ergonomics. The automatic quantization pipeline eliminates manual tuning torture. The OpenAI-compatible API means your existing tools just work.

Yes, there are constraints: NVIDIA-only, Linux/WSL2, substantial RAM and disk requirements. But these are known constraints you engineer around once, versus the perpetual constraints of cloud dependency—cost unpredictability, latency variance, data exposure, and creative limitation.

The developers already deploying Krasis aren't waiting for permission. They're running research-grade models on hardware they own, iterating at speeds that match or exceed hosted APIs, and building applications that were theoretically impossible six months ago.

Your move. Install Krasis today. Run that 80B model you've been avoiding. Discover what local inference at scale actually feels like. The repository is waiting: github.com/brontoguana/krasis. Star it, test it, break it, report it—then build something that shouldn't be possible on your desk.

The future of AI infrastructure isn't centralized. It's Krasis.