oobabooga/text-generation-webui: Local LLMs with Vision, Training & API

Developers running large language models locally face a consistent friction: juggling multiple backends, wrestling with CUDA dependencies, and stitching together separate tools for chat, vision, and fine-tuning. Cloud APIs solve the integration problem but introduce latency, cost, and data sovereignty concerns. oobabooga/text-generation-webui addresses this by packaging local LLM inference, multimodal vision, LoRA training, and an OpenAI-compatible API into a single open-source desktop application—no telemetry, no external calls.

This article examines what oobabooga/text-generation-webui actually delivers, how to deploy it, and where it fits in the increasingly crowded landscape of local AI tooling.

What is oobabooga/text-generation-webui?

oobabooga/text-generation-webui (also referred to as "TextGen" in recent branding) is an open-source desktop application for running large language models locally. Maintained by the developer oobabooga, the project has accumulated 47,447 GitHub stars and 5,983 forks as of its last commit on June 2, 2026. The codebase is primarily Python^{↗ Bright Coding Blog} and licensed under the GNU Affero General Public License v3.0.

The project emerged from the same design philosophy as AUTOMATIC1111/stable-diffusion-webui—prioritizing local execution, extensive hardware support, and a plugin-style extension ecosystem. Where it diverges is in its explicit focus on privacy: zero telemetry, no external resource loading, and no remote update requests. Every inference operation happens on the user's hardware.

TextGen supports five distinct inference backends: llama.cpp, ik_llama.cpp, Transformers, ExLlamaV3, and TensorRT-LLM. This backend flexibility matters because different model formats and hardware configurations favor different engines. GGUF models via llama.cpp work immediately with portable builds; ExLlamaV3 and Transformers require the full installation but unlock additional quantization and architecture support. Users can switch backends without restarting the application.

The project also functions as an API server, exposing OpenAI- and Anthropic-compatible endpoints for chat, completions, and messages—with tool-calling support. This allows existing applications built against commercial APIs to redirect traffic to a local instance with minimal code changes.

Key Features

Multimodal chat and generation. The core interface supports instruct, chat-instruct, and chat modes, with automatic prompt formatting via Jinja2 templates. Vision capabilities allow image attachment to messages for visual understanding. File attachments extend to text files, PDFs, and .docx documents. The interface includes message editing, version navigation, conversation branching, and a dedicated Notebook tab for free-form generation outside chat turns.

Backend abstraction and hot-swapping. The loader system autodetects model formats but allows manual override. llama.cpp handles GGUF models with GPU layer offloading, tensor splitting across multiple GPUs, and speculative decoding options including ngram-based draftless generation. Transformers and ExLlamaV3 support additional quantization schemes and attention implementations. TensorRT-LLM provides NVIDIA-optimized inference for supported architectures.

OpenAI/Anthropic-compatible API. The API server implements Chat, Completions, and Messages endpoints with tool-calling support. Custom functions—implemented as single .py files—can perform web search, page fetching, and mathematical operations. MCP (Model Context Protocol) servers are also supported for extended tool ecosystems.

Training and image generation. LoRA fine-tuning operates on multi-turn chat or raw text datasets with resume capability. A separate image generation tab supports diffusers models like Z-Image-Turbo with 4-bit/8-bit quantization, persistent gallery storage, and metadata preservation.

Privacy and customization. The application runs entirely offline. Theming includes dark/light modes, syntax highlighting for code blocks, and LaTeX rendering. Extensions—both built-in and community-contributed—add TTS, voice input, translation, and other capabilities.

Use Cases

Local development and prototyping. Teams building LLM-powered applications can run oobabooga/text-generation-webui as a local API replacement for OpenAI or Anthropic services. The compatible endpoints mean existing client libraries require only a base URL change. Tool-calling support allows testing agentic workflows without cloud costs or data exposure.

Private document analysis. Organizations handling sensitive materials—legal documents, medical records, internal research—can load models locally and attach PDF or .docx files for question-answering. No data leaves the hardware, satisfying compliance requirements that prohibit third-party API usage.

Multimodal research and evaluation. Vision model support enables testing of multimodal LLMs on proprietary image sets. Researchers can compare backend performance (llama.cpp vs. ExLlamaV3 vs. Transformers) on identical prompts without managing separate environments.

LoRA fine-tuning for specialized domains. The training tab supports domain adaptation on private datasets. Resume capability protects against hardware interruptions during long training runs. This suits organizations with insufficient data for full fine-tuning but enough for low-rank adaptation.

Offline deployment in restricted environments. Air-gapped or bandwidth-constrained environments can run the portable build with all dependencies included. No conda, pip, or network access required after initial download.

Installation & Setup

Portable Build (Fastest Path)

The maintainers provide prebuilt binaries for Linux, Windows, and macOS with CUDA, Vulkan, ROCm, and CPU-only variants. All dependencies are included.

1. Download from https://github.com/oobabooga/textgen/releases
2. Extract the archive
3. Double-click textgen (or run the executable)

The application window opens immediately. GGUF models placed in user_data/models/ are auto-detected.

Manual Portable Install with venv

For developers preferring source control or custom Python environments:

# Clone repository
git clone https://github.com/oobabooga/textgen
cd textgen

# Create virtual environment
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

# Install dependencies (choose appropriate file under requirements/portable for your hardware)
pip install -r requirements/portable/requirements.txt --upgrade

# Launch server (basic command)
python server.py --portable --api --auto-launch

# When done working, deactivate
deactivate

The --portable flag hides features unavailable in portable mode (training, certain extensions). The --api flag enables the OpenAI-compatible server. --auto-launch opens the interface in the default browser.

Full Installation (All Features)

For ExLlamaV3, Transformers backends, training, image generation, and extensions:

# One-click installer approach
git clone https://github.com/oobabooga/textgen
cd textgen
./start_linux.sh  # or start_windows.bat, start_macos.sh

When prompted, select your GPU vendor. After installation, access the web UI at http://127.0.0.1:7860.

For conda users, the README provides explicit PyTorch installation commands per platform and GPU type. For example, NVIDIA on Linux/WSL:

pip3 install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128

Then install the full requirements:

pip install -r requirements/full/requirements.txt

Real Code Examples

Automatic Model Loading on Startup

To skip manual model selection, create user_data/CMD_FLAGS.txt:

--model my-model.gguf

Replace my-model.gguf with any file in user_data/models/. Multiple flags go on separate lines:

--model my-model.gguf
--cache-type q8_0

The cache-type flag sets KV cache quantization—q8_0 reduces memory usage with minimal quality impact for llama.cpp backends.

Docker^{↗ Bright Coding Blog} Deployment (NVIDIA GPU)

# Symlink Docker configuration files
ln -s docker/{nvidia/Dockerfile,nvidia/docker-compose.yml,.dockerignore} .

# Copy and edit environment configuration
cp docker/.env.example .env

# Create required directories
mkdir -p user_data/logs user_data/cache

# Edit .env to set:
#   TORCH_CUDA_ARCH_LIST based on your GPU model
#   APP_RUNTIME_GID to your host group ID (run `id -g`)
#   BUILD_EXTENSIONS optionally for comma-separated extension list

# Configure runtime flags
echo "--listen --api" > user_data/CMD_FLAGS.txt

# Build and launch
docker compose up --build

Requires Docker Compose v2.17+. The TORCH_CUDA_ARCH_LIST must match your GPU's compute capability or PyTorch operations will fail with architecture mismatch errors.

API Server with Authentication

python server.py --api --api-key sk-local-123 --admin-key sk-admin-456 --listen --listen-port 5000

The --api-key secures standard endpoints; --admin-key restricts model loading/unloading operations. Without --admin-key, admin functions use the same key as standard API access. --listen exposes the server beyond localhost—use only in trusted network environments or behind a reverse proxy.

Advanced Usage & Best Practices

Backend selection strategy. For GGUF models on consumer GPUs, llama.cpp with --gpu-layers -1 (auto-offload) provides the simplest path. For FP16/BF16 models requiring full precision, Transformers with --load-in-8bit or --load-in-4bit reduces VRAM at latency cost. ExLlamaV3 excels for supported quantized formats with its optimized CUDA kernels. TensorRT-LLM requires upfront engine building but delivers highest throughput for fixed batch sizes.

Memory estimation. Before downloading models, use the GGUF Memory Calculator maintained by the project author. For recommended quantization levels, LocalBench provides empirical quality comparisons.

Speculative decoding. The ngram-mod speculative decoding type requires no draft model—instead, it extracts n-grams from the existing context. Configure via:

--spec-type ngram-mod
--spec-ngram-size-n 5
--spec-ngram-size-m 2

This trades minor generation quality for measurable speedup on long-context outputs.

Extension isolation. Extensions may introduce dependency conflicts. The update wizard's "Install/update extensions requirements" option reinstalls main project requirements afterward to enforce precedence. For production deployments, consider running extensions in separate virtual environments or containers.

Comparison with Alternatives

Trade-offs: llama.cpp offers maximum simplicity for pure text inference but lacks the integrated ecosystem. vLLM achieves superior throughput for concurrent API serving but requires more complex deployment and offers no training or desktop interface. oobabooga/text-generation-webui occupies the middle ground—more complex than llama.cpp alone, but consolidating capabilities that otherwise demand multiple tools.

FAQ

Does it require an internet connection? No. After initial download, operation is 100% offline with zero telemetry or external resource loading.

What Python version is required? Python 3.9+ for manual installation; portable builds include their own Python runtime.

Can I use AMD or Intel GPUs? Yes. Portable builds include Vulkan and ROCm variants. Full installation supports AMD via ROCm wheels and Intel via dedicated Docker configurations.

Is the OpenAI API fully compatible? The Chat, Completions, and Messages endpoints are implemented with tool-calling support. Some edge-case parameters may differ; verify against the wiki examples.

How do I update without losing settings? Run the platform-specific update wizard (update_wizard_linux.sh, etc.). User data in user_data/ persists across updates.

What's the license? GNU Affero General Public License v3.0. Commercial use is permitted under AGPL terms; SaaS deployments must provide source code to users.

Can multiple users share one instance? The --multi-user flag enables shared access without persistent chat histories—intended for small trusted teams, not production multi-tenancy.

Conclusion

oobabooga/text-generation-webui suits developers and organizations prioritizing local execution, privacy, and unified tooling over maximum inference throughput. Its strength is consolidation: one application handling chat, vision, API serving, training, and image generation, with sensible defaults for each. The 47,000+ star count reflects genuine utility for this use case, not marketing.

The project is best matched to: privacy-conscious teams prototyping LLM applications, researchers evaluating multiple backends, and users in restricted environments needing fully offline operation. It is less ideal for high-throughput production API serving, where vLLM or dedicated inference servers outperform it.

Download the portable build or explore the source at https://github.com/oobabooga/text-generation-webui. For detailed tutorials on multimodal setup, tool-calling, and training workflows, consult the project wiki.