lightningpixel/modly: Local AI Image-to-3D Generation on Your GPU

lightningpixel/modly: Local AI Image-to-3D Generation on Your GPU

Generating 3D assets from 2D images has traditionally meant uploading sensitive data to cloud APIs, paying per-generation fees, or wrestling with fragmented Python^{↗ Bright Coding Blog} pipelines. For developers building games, AR experiences, or prototyping industrial designs, this creates friction: latency, cost unpredictability, and dependency on services you don't control. lightningpixel/modly addresses this directly—a desktop application that runs open-source AI models entirely on your local GPU, converting photos into 3D meshes without network round-trips or subscription costs.

Created by Lightning Pixel, the project has accumulated 4,289 stars and 463 forks as of its last commit on July 13, 2026. Written primarily in TypeScript with a Python backend, Modly targets developers who want ownership of their generative pipeline. This article examines what Modly offers, how it works under the hood, and where it fits in the current landscape of 3D generative AI tools.

What is lightningpixel/modly?

lightningpixel/modly is an open-source desktop application for local, AI-powered image-to-3D mesh generation. It packages open-source computer vision models into an Electron-based interface that communicates with a Python FastAPI backend, keeping all inference on the user's hardware.

The project sits at the intersection of several active technical domains: generative AI for 3D content, edge/on-device inference, and developer tooling for creative pipelines. Its relevance in 2026 stems from a broader shift toward local AI execution—driven by GPU availability in consumer hardware, concerns about data sovereignty, and the maturation of open-weight models like Hunyuan3D and TripoSG that rival proprietary cloud alternatives.

Modly's architecture reflects pragmatic engineering choices. The TypeScript frontend handles UI state, workflow validation, and mesh visualization. The Python backend manages model loading, inference orchestration, and extension lifecycle. This split allows frontend developers to iterate on interface concerns while ML engineers can swap model implementations without touching renderer code.

The project explicitly targets Windows, Linux, and Apple Silicon macOS—notably excluding Intel Macs, which reflects the neural engine and GPU compute advantages of Apple's M-series chips for this workload. The live RAM indicator in the top bar and workflow validation before run execution suggest design priorities around transparency and error recovery rather than opaque "it just works" abstraction.

Key Features

Local GPU inference is the defining characteristic. All models run on-device; no images leave your machine. This has concrete implications for IP-sensitive workflows—fashion design, unreleased game assets, medical imaging—where cloud upload policies create legal review bottlenecks.

Extension system for models and processes. Modly doesn't hardcode model support. Extensions are GitHub repositories containing a manifest.json plus runtime entry files, installable via the Models page by pasting an HTTPS URL. The README lists five official extensions covering Hunyuan3D 2 Mini (three speed variants), TripoSG, and Trellis2 GGUF—indicating active maintenance of multiple architecture backends rather than betting on a single model family.

In-app mesh optimization. Imported meshes can be smoothed and decimated without external tools; optimized results write back into the workspace. This reduces toolchain fragmentation for users who would otherwise bounce between Blender, MeshLab, or command-line utilities.

Workflow validation with non-destructive error handling. Invalid graphs stay in place and surface inline or toast warnings rather than dropping the current mesh view. For iterative creative work, this preserves state and reduces rework.

Platform-native UI conventions. macOS uses native window controls; Windows and Linux retain custom controls. This isn't mere aesthetic preference—it affects accessibility expectations and OS integration depth for keyboard shortcuts and window management.

CLI for automation. A stdlib-only Python CLI (tools/modly-cli/agent.py) exposes health checks, model listing, workflow run management, and a convenience generate command that handles polling and export. The CLI outputs machine-readable JSON, making it scriptable for batch processing or CI pipelines.

Use Cases

Game asset prototyping. Concept artists can photograph physical maquettes or sketch scans and generate base meshes for character or prop blocking. The local execution means iteration cycles aren't gated by API rate limits or credit balances.

E-commerce product visualization. Retailers with large SKU catalogs can experiment with automated 3D preview generation from product photography. The extension system allows swapping models as the field evolves—today's Hunyuan3D variant may be tomorrow's different architecture without application rewrites.

Architectural and interior design. Site photos can seed 3D context models for spatial planning tools. The mesh decimation feature helps produce LOD-appropriate geometry for real-time walkthrough engines.

Research and education. ML practitioners studying image-to-3D architectures gain a runnable reference implementation with visible wiring between frontend state management and backend inference. The CLI's separation of canonical vs. experimental commands models good API design practice.

Privacy-critical industrial design. Aerospace, automotive, and medical device firms with strict data handling requirements can evaluate generative 3D workflows without vendor security reviews or air-gapped cloud exceptions.

Installation & Setup

Modly offers two paths: prebuilt installers from the Releases page, or building from source.

Prebuilt Installers

Download the latest release for your platform (Windows, Linux, or Apple Silicon macOS). The README doesn't specify package formats—check the release assets for .exe, .AppImage, .dmg, or equivalent.

Building from Source

Step 1: Clone and install JavaScript^{↗ Bright Coding Blog} dependencies

npm install

This installs the Electron frontend and build toolchain.

Step 2: Set up the Python backend

cd api
python -m venv .venv
.venv\Scripts\activate     # Windows
source .venv/bin/activate  # Linux / macOS
pip install -r requirements.txt

The backend uses a virtual environment to isolate Python dependencies. The requirements.txt isn't shown in the README, but standard practice would include FastAPI, PyTorch or ONNX Runtime, and model-specific packages.

Step 3: Run in development mode

npm run dev

This likely starts both the Electron renderer and the Python API server in watch mode.

Step 4: Verify with tests and type checking

npm test
./node_modules/.bin/tsc --noEmit -p tsconfig.node.json
npm run build

The TypeScript compiler check (tsc --noEmit) validates type correctness without emitting files. npm run build produces production assets.

Platform-specific packaging:

npm run package:mac  # Apple Silicon macOS only

Windows and Linux packaging commands aren't shown in the README; check package.json scripts or documentation for equivalents.

Real Code Examples

The README provides limited but concrete CLI usage. Below are the documented commands with context.

Health Check and Model Discovery

# Verify the running Modly instance is responsive
python tools/modly-cli/agent.py health

# List available models and their variants
python tools/modly-cli/agent.py model list

These use the canonical command surface. The CLI requires a running Modly desktop app—it doesn't bootstrap the backend itself.

Image-to-3D Generation with Polling

# Generate a 3D mesh from an image, with built-in polling and export
python tools/modly-cli/agent.py generate --image ./input.png --output ./export.glb

This convenience command wraps multiple operations: it starts a workflow run via POST /workflow-runs/from-image, polls completion status, and exports the final mesh. The JSON response includes recovery metadata—specifically workflow-run status <run_id> and workflow-run cancel <run_id> commands—so scripts can resume or abort interrupted jobs.

Workflow Run Inspection

# Check status of an existing run
python tools/modly-cli/agent.py workflow-run status <run_id>

For automation pipelines, this enables idempotent status checks without re-triggering expensive inference.

The README explicitly notes that dev serve-api and dev ensure-server start only the FastAPI backend without proving Electron/Desktop bridge readiness—useful for backend development^{↗ Bright Coding Blog}, not integration testing. The legacy, experimental, and hidden helper aliases exist for compatibility but aren't presented as stable contracts.

Advanced Usage & Best Practices

Extension management strategy. The GitHub-based extension system means version pinning isn't built in. For reproducible pipelines, consider forking extension repositories and installing from your fork, or vendor the extension code into your deployment. The manifest format isn't fully documented in the README—inspect existing official extensions for schema expectations.

Resource monitoring. The live RAM indicator surfaces memory pressure during inference. For batch operations via CLI, consider wrapping calls with systemd-run --scope (Linux) or equivalent to set memory limits and prevent OOM cascades.

Workflow design for validation. The README notes invalid graphs stay in place with warnings rather than auto-correcting. Design workflows defensively: validate node connectivity before long-running inference stages, and use the toast warning patterns to surface issues to operators.

Mesh optimization trade-offs. In-app smoothing and decimation are convenient but may not match domain-specific tools. For production assets, export to .glb and refine in specialized software; use Modly's optimization for preview-quality output or rapid iteration.

CLI scripting patterns. The generate command's polling is convenient but adds latency. For high-throughput scenarios, manage workflow runs explicitly with workflow-run status polling at your own interval, and parallelize across multiple Modly instances if GPU memory permits.

Comparison with Alternatives

Modly's position is clear: it trades the zero-setup experience of cloud APIs for control and privacy. Compared to raw ComfyUI workflows, it adds application structure—windowing, mesh viewing, workflow validation—at the cost of some flexibility. The extension system attempts to preserve model choice without the configuration burden.

FAQ

What GPU do I need? The README doesn't specify minimum requirements. Given the models listed (Hunyuan3D, TripoSG), expect 8GB+ VRAM for comfortable operation; Apple Silicon unified memory behaves differently.

Does it work on Intel Macs? No. macOS support targets Apple Silicon exclusively.

Is there a Windows ARM build? Not mentioned. The README specifies Windows generally, without architecture clarification.

What's the license? MIT License, with an explicit attribution requirement: forks must credit Modly and Lightning Pixel in UI or documentation.

Can I use Modly without the desktop UI? Partially. dev serve-api starts the backend alone, but the CLI requires a running desktop app for full functionality. The canonical agent contract assumes bridge readiness.

How do I add a custom model? Implement an extension repository with manifest.json and runtime entry files, then install via Models → Install from GitHub. No further documentation is provided in the README.

Is there commercial support? Not mentioned. Community support occurs via Discord.

Conclusion

lightningpixel/modly fills a specific niche: developers and technical artists who need image-to-3D generation under their control, on their hardware, with transparent model wiring. The 4,289 stars and active extension ecosystem suggest genuine traction, though the project demands more setup investment than cloud alternatives.

It's best suited for: privacy-sensitive workflows, teams with GPU infrastructure, researchers comparing model architectures, and builders who want to extend rather than consume a black-box service. The TypeScript/Python split and CLI surface indicate engineering maturity; the extension system shows architectural foresight for a fast-moving model landscape.

If that matches your constraints, download a release or clone the repository at https://github.com/lightningpixel/modly and evaluate against your specific models and hardware. For related approaches to local generative workflows, see [INTERNAL_LINK: local-ai-development-tools].