SkyworkAI/Matrix-3D: Generate Explorable 3D Worlds from One Image or Text

SkyworkAI/Matrix-3D: Generate Explorable 3D Worlds from One Image or Text

Creating explorable 3D environments traditionally demands specialized 3D modeling skills, expensive software, and hours of manual work. For developers building virtual worlds, game prototypes, or immersive experiences, this bottleneck slows iteration and raises costs. SkyworkAI/Matrix-3D addresses this directly: an open-source Python^{↗ Bright Coding Blog} pipeline that generates large-scale, omnidirectional explorable 3D scenes with high-quality panorama videos from a single image or text prompt. With 753 GitHub stars and active development through late 2025, it represents a practical approach to automated 3D world generation for technically literate creators.

What is SkyworkAI/Matrix-3D?

SkyworkAI/Matrix-3D is an open-source project maintained by SkyworkAI that combines conditional video generation with panoramic 3D reconstruction to produce explorable 3D worlds. Released in August 2025 with ongoing updates through September 2025, the project is licensed under MIT and primarily written in Python.

The tool sits at the intersection of several active research areas: panoramic image synthesis, video generation, and 3D Gaussian splatting reconstruction. Its core innovation is using panoramic representation for wide-coverage scene generation—meaning users can generate complete 360-degree environments rather than limited perspective views. This distinguishes it from single-view 3D reconstruction methods that produce isolated objects or narrow scene fragments.

The project builds on established foundations including FLUX.1 for image generation, Wan2.1 for video generation, MoGe for depth estimation, and gaussian-splatting for 3D representation. This architectural choice—combining specialized components rather than training monolithic models—reflects practical engineering priorities: leverage proven tools where they excel, integrate where needed. The result is a modular pipeline with clear separation between panorama image generation, panoramic video synthesis, and final 3D scene extraction.

Key Features

Large-Scale Scene Generation with 360-Degree Exploration. Unlike approaches that generate constrained viewpoints, Matrix-3D produces broad, expansive scenes allowing complete omnidirectional navigation. The panoramic representation ensures coverage without the seams or distortions typical of stitched perspective renders.

Dual Conditioning: Text and Image Inputs. The pipeline accepts both text prompts and existing images as starting points. This flexibility matters for different workflows: concept artists may start from text descriptions, while photographers or designers can seed generation from existing visual assets.

Customizable Camera Trajectories. Beyond static panoramas, Matrix-3D supports configurable movement modes including Straight Travel, S-curve Travel, and Forward on the Right. Users can also supply custom camera trajectories via JSON files, enabling precise control over the generated video path through the scene.

Two Reconstruction Paths: Speed vs. Quality. The project explicitly addresses the speed-quality tradeoff with two panoramic 3D reconstruction methods. An optimization-based reconstruction (~10GB VRAM, slower but higher quality) and a feed-forward PanoLRM reconstruction (~80GB VRAM, faster inference). This dual-path design lets users choose based on hardware constraints and output requirements.

Low-VRAM Operation. Through successive updates, the team reduced hardware barriers significantly. The 5B parameter video generation model introduced in September 2025 runs in low-VRAM mode with only 12GB VRAM—down from the original 60GB requirement for 720p generation. A 19GB VRAM mode for the full 720p model is also available, broadening accessibility beyond datacenter GPUs.

Multi-GPU Scaling. Video generation supports distributed inference via torchrun, with generation time for 720p video approximately one hour on an A800 GPU that can be reduced through parallelization.

Use Cases

Rapid Game Environment Prototyping. Level designers can generate explorable 3D drafts from concept art or text descriptions, testing spatial layouts before committing to full production modeling. The .ply output integrates with standard game engines and 3D tools.

Virtual Production and Film Previsualization. Directors and VFX supervisors can block out camera moves through generated 3D spaces, iterating on composition and pacing without waiting for asset builds. The custom trajectory support enables precise shot planning.

Architectural Visualization and Real Estate. Starting from photographs or text descriptions of spaces, developers can create immersive walkthroughs. The image-to-scene path is particularly relevant for visualizing renovations or unbuilt structures from reference photos.

VR/AR Experience Development. The omnidirectional output format suits headset-based experiences natively. Creators can prototype spatial environments quickly, testing user navigation and scale perception before final asset production.

Synthetic Training Data Generation. ML practitioners can produce diverse 3D environments with controlled camera paths for training embodied AI, navigation models, or view-synthesis networks—addressing the persistent need for varied, annotated 3D data.

Installation & Setup

Matrix-3D is currently tested on Linux with NVIDIA GPUs. The setup follows standard Python environment practices with specific PyTorch and CUDA requirements.

Clone the repository with submodules and create the environment:

# Clone the repository with recursive submodules
git clone --recursive https://github.com/SkyworkAI/Matrix-3D.git
cd Matrix-3D

# Create and activate conda environment
conda create -n matrix3d python=3.10
conda activate matrix3d

# Install PyTorch with CUDA 12.4 support
pip install torch==2.7.0 torchvision==0.22.0

# Run installation script
chmod +x install.sh
./install.sh

The --recursive flag is critical: the project includes submodule dependencies that won't resolve otherwise. The pinned PyTorch version (2.7.0) and CUDA 12.4 target suggest careful compatibility testing; deviating from these versions may introduce subtle failures in the video generation or 3D reconstruction stages.

After installation, download pretrained models:

python code/download_checkpoints.py

This fetches the complete model suite from Hugging Face: Text2PanoImage LoRA weights, three PanoVideoGen variants (480p, 720p, 5B), and PanoLRM reconstruction weights.

Real Code Examples

One-Command Full Pipeline

For quick starts, the project provides a single-script execution path:

./generate.sh

This runs the complete workflow: panorama generation, video synthesis, and 3D extraction with default parameters. It's useful for verifying installation and understanding baseline output before customizing.

Text-to-3D World Generation

Generate a complete explorable scene from a text description:

# Step 1: Generate panorama image from text
python code/panoramic_image_generation.py \
    --mode=t2p \
    --prompt="a medieval village, half-timbered houses, cobblestone streets, lush greenery, clear blue sky, detailed textures, vibrant colors, high resolution" \
    --output_path="./output/example1"

# Step 2: Generate panoramic video (single GPU)
VISIBLE_GPU_NUM=1
torchrun --nproc_per_node ${VISIBLE_GPU_NUM} code/panoramic_image_to_video.py \
    --inout_dir="./output/example1" \
    --resolution=720

# Step 3: Extract 3D scene via optimization-based reconstruction
python code/panoramic_video_to_3DScene.py \
    --inout_dir="./output/example1" \
    --resolution=720

The --mode=t2p flag selects text-to-panorama generation. The torchrun wrapper enables distributed inference; for multi-GPU systems, increase VISIBLE_GPU_NUM proportionally. Resolution must match between video generation and 3D extraction steps.

Image-to-3D World Generation

Seed generation from an existing photograph:

# Step 1: Generate panorama from input image
python code/panoramic_image_generation.py \
    --mode=i2p \
    --input_image_path="./data/image1.jpg" \
    --output_path="./output/example1"

# Steps 2-3: identical to text-to-3D pipeline above

The generated panorama and its prompt are saved as pano_img.jpg and prompt.txt in the output directory. This structure also accepts externally generated panoramas—useful for incorporating outputs from tools like [INTERNAL_LINK: HunyuanWorld] or custom pipelines.

Low-VRAM 720p Generation

Enable memory-efficient execution for consumer GPUs:

VISIBLE_GPU_NUM=1
torchrun --nproc_per_node ${VISIBLE_GPU_NUM} code/panoramic_image_to_video.py \
    --inout_dir="./output/example1" \
    --resolution=720 \
    --enable_vram_management  # Reduces VRAM from ~60GB to ~19GB

For even lower requirements, the 5B model trades some quality for accessibility:

VISIBLE_GPU_NUM=1
torchrun --nproc_per_node ${VISIBLE_GPU_NUM} code/panoramic_image_to_video.py \
    --inout_dir="./output/example1" \
    --resolution=720 \
    --use_5b_model  # Requires only ~12GB VRAM in low-VRAM mode

Custom Camera Trajectory

Supply a JSON camera path for precise movement control:

VISIBLE_GPU_NUM=1
torchrun --nproc_per_node ${VISIBLE_GPU_NUM} code/panoramic_image_to_video.py \
    --inout_dir="./output/example1" \
    --resolution=720 \
    --json_path YOUR_TRAJECTORY_FILE.json

Camera matrices use world-to-camera format in OpenCV convention. The project provides code/generate_example_camera.py for trajectory generation and ./data/test_cameras/test_cam_front.json as reference format.

Advanced Usage & Best Practices

Memory Planning. The VRAM requirements vary dramatically by stage and model choice. Plan your pipeline accordingly: if GPU memory is limited, use the 5B model with --enable_vram_management for video generation, then switch to optimization-based reconstruction (~10GB) rather than PanoLRM (~80GB). The full pipeline can run on a single 24GB consumer card with careful model selection.

Resolution Consistency. The --resolution parameter must match between video generation and 3D extraction. Mixing 480p video with 720p reconstruction will fail or produce corrupted output. The project supports 480 (960×480) and 720 (1440×720) natively.

Multi-GPU Scaling Strategy. Video generation benefits linearly from additional GPUs; reconstruction stages do not appear to distribute similarly. Allocate GPUs to the bottleneck step—typically video synthesis—rather than spreading across stages.

External Panorama Integration. The documented directory structure (pano_img.jpg + prompt.txt) enables hybrid workflows. Generate panoramas with specialized tools, then leverage Matrix-3D's video and 3D reconstruction stages. This modularity is intentional and worth exploiting.

Gradio Demo for Exploration. The included web interface (python code/app_matrix3d.py --max_gpus=1) requires 62GB for single-GPU text-to-3D workflows, or multiple GPUs for image-conditioned generation. Use this for demonstration and parameter exploration before scaling to batch processing.

Comparison with Alternatives

Matrix-3D's distinguishing characteristic is the integrated pipeline: competitors typically excel at one stage (image generation, video synthesis, or 3D reconstruction) while Matrix-3D chains these into a cohesive workflow. The tradeoff is higher overall complexity and resource requirements. For users needing only panorama images, lighter alternatives exist; for those needing complete explorable worlds from minimal input, Matrix-3D's integration is the relevant advantage.

FAQ

What hardware do I need? Linux with NVIDIA GPU. Minimum 16GB VRAM for full pipeline; 12GB possible with 5B model low-VRAM mode.

Can I use my own panorama images? Yes. Place as pano_img.jpg with prompt.txt in the output directory, then run video and 3D stages directly.

How long does generation take? Approximately one hour for 720p video on A800 GPU. Scales with GPU count and resolution.

What's the license? MIT License. Commercial use permitted with attribution.

Is Windows or macOS supported? Not currently tested. Linux only per documentation.

Can I control camera movement precisely? Yes. Three built-in movement modes plus custom JSON trajectory support.

What's the output format? Panoramic video as .mp4; 3D scene as .ply (Gaussian splat format).

Conclusion

SkyworkAI/Matrix-3D offers a genuinely integrated path from minimal input—single image or text prompt—to explorable 3D environment. Its modular architecture, multiple quality-speed tradeoffs, and active development (including significant VRAM reductions in recent releases) make it a practical option for developers, researchers, and creators who need 3D worlds without traditional modeling pipelines.

The tool is best suited for: rapid prototyping where fidelity can iterate upward; scenarios requiring panoramic/omnidirectional output; and technically capable users comfortable with PyTorch-based tooling and GPU resource management. It's less ideal for casual users seeking one-click mobile generation or those without NVIDIA hardware.

Explore the code, models, and documentation at https://github.com/SkyworkAI/Matrix-3D. The project page and technical report provide additional architectural details for researchers, while the Gradio demo offers hands-on exploration before local installation.