FantasyWorld: Why Developers Are Ditching Separate Video & 3D Pipelines

What if I told you that everything you know about building AI-generated worlds is about to become obsolete?

For years, developers and researchers have been trapped in a brutal cycle: train one model for video generation, train another entirely separate model for 3D scene reconstruction, then desperately try to fuse their outputs into something that doesn't look like a glitchy mess. The geometry never matches. The camera trajectories drift. The lighting inconsistencies make your "immersive experience" look like a bad green screen from 2003.

Sound familiar?

You've burned through cloud credits. You've wrestled with incompatible formats. You've watched your carefully crafted 3D mesh deform into nightmare fuel when you try to animate it. The dirty secret of modern generative AI is that video and 3D have been treated as fundamentally different problems — until now.

Enter FantasyWorld, the unified world modeling system that just ranked #1 on Stanford's WorldScore leaderboard (yes, that Fei-Fei Li team). Accepted at ICLR 2026, this isn't another incremental paper. It's a architectural paradigm shift that treats video generation and 3D scene prediction as a single, geometrically consistent problem.

Stop duct-taping pipelines together. Start building worlds that actually make sense.

What is FantasyWorld?

FantasyWorld is an open-source, unified feed-forward model for joint video and 3D scene generation — developed by researchers at AMAP (AutoNavi/Alibaba) and published at ICLR 2026. The project lives at https://github.com/Fantasy-AMAP/fantasy-world and represents one of the first production-ready architectures to successfully merge video diffusion with explicit 3D reasoning in a single forward pass.

The core innovation? Geometry-consistent world modeling. Traditional approaches generate video frames as flat 2D latents, then attempt to recover 3D structure as a post-processing afterthought. FantasyWorld flips this script by embedding geometric reasoning directly into the generation process through its novel Integrated Reconstruction and Generation (IRG) Blocks.

Here's why the research community is paying attention:

• ICLR 2026 acceptance — one of the most competitive venues in machine learning
• WorldScore #1 ranking (December 2025), surpassing global state-of-the-art models on Stanford's rigorous benchmark
• Built on top of the proven Wan video generation architecture, not from scratch
• Two model variants: a reproducible research checkpoint and a performance-optimized production model

The project emerged from a collaboration between Yixiang Dai, Fan Jiang, Chiyu Wang, Mu Xu, and Yonggang Qi — researchers who recognized that the artificial separation between video and 3D generation was holding back both fields. Their solution: a dual-branch architecture where appearance synthesis and geometric reasoning co-evolve, coupled through lightweight cross-attention mechanisms.

Key Features That Make FantasyWorld Insane

🔥 Preconditioning Blocks (PCBs): No More Noise Wrestling

FantasyWorld's front end employs Preconditioning Blocks that cleverly reuse the frozen WanDiT denoiser. Instead of forcing the geometry pathway to operate on pure Gaussian noise — where structure is nearly impossible to extract — PCBs supply partially denoised latents to the geometry branch. This means your 3D reasoning starts from meaningful features, not mathematical chaos.

Technical impact: Faster convergence, more stable depth predictions, and dramatically reduced training time for geometric pre-training.

🔥 Integrated Reconstruction and Generation (IRG) Blocks

The backbone consists of stacked IRG blocks that iteratively refine two representations simultaneously:

• Video latents (for temporal coherence)
• Geometry features (for spatial consistency)

Each block operates under multimodal conditioning — text prompts, camera trajectories, and input images all feed into the same unified processing pipeline.

🔥 Asymmetric Dual-Branch Architecture

This is where FantasyWorld diverges from every "unified" model that came before:

These branches communicate through lightweight adapters and cross-attention — not heavy fusion layers that would dilute either signal. The asymmetry is intentional: appearance generation needs expressive capacity, while geometry needs structural rigor.

🔥 Two-Stage Training Strategy

Stage 1 — Geometry Pre-training: Uses a VGGT-style model for precise estimation of depth, point clouds, and camera trajectories. This isn't auxiliary supervision; it's foundational geometric literacy.

Stage 2 — Joint Generation: Seamlessly integrates the Stage 1 geometry backbone with the Wan video generation pipeline. The result? A model that speaks both "pixel" and "point" fluently.

🔥 Dual Model Zoo for Different Needs

Use Cases: Where FantasyWorld Destroys the Competition

🎬 Cinematic Pre-Visualization

Storyboard artists and VFX supervisors can generate camera-controllable video sequences with consistent 3D geometry from a single concept image. No more "that angle doesn't exist in our set" surprises during production. The explicit camera trajectory control via JSON input means precise directorial intent, not prompt lottery.

🏗 Architectural Visualization & Real Estate

Input a floor plan render, specify a walkthrough camera path, and generate photorealistic video tours with geometrically accurate depth and structure. Traditional methods require separate 3D modeling, texturing, lighting setup, and rendering — FantasyWorld collapses this to a single forward pass with explicit spatial guarantees.

🎮 Game Asset & Level Prototyping

Indie developers can rapidly prototype explorable environments: generate video flythroughs and extract usable 3D point clouds for mesh reconstruction. The geometry-consistent output means your generated world can actually be navigated, not just admired from fixed angles.

🤖 Robotics & Autonomous Systems Simulation

Train perception models on synthetic data with perfect geometric ground truth. Because FantasyWorld generates depth, points, and camera poses jointly with video, you get pixel-perfect alignment between visual appearance and 3D structure — the holy grail for sim-to-real transfer.

📚 Academic Research in World Models

Finally, a reproducible baseline for studying unified video-3D representation learning. The Wan2.1 variant strictly follows paper configurations; no hidden tricks, no undisclosed training data. Build on it, cite it, improve it.

Step-by-Step Installation & Setup Guide

Ready to stop reading and start generating? Here's the complete setup from zero to inference.

Prerequisites

• Python 3.10 (strictly required for dependency compatibility)
• CUDA-capable GPU with sufficient VRAM (14B parameter models are not lightweight)
• Conda or equivalent environment manager
• ~50GB disk space for model checkpoints

Step 1: Clone and Enter the Repository

# Clone the official repository
git clone https://github.com/Fantasy-AMAP/fantasy-world.git
cd fantasy-world

Step 2: Create Conda Environment and Install Dependencies

# Create isolated Python 3.10 environment
conda create -n fantasyworld python=3.10
conda activate fantasyworld

# Install core dependencies
pip install -r requirements.txt

# Install custom 3D utilities (critical for geometry processing)
pip install thirdparty/utils3d/

⚠️ Critical: The utils3d package in thirdparty/ contains custom CUDA kernels for 3D operations. Skipping this will cause geometry branch failures.

Step 3: Choose Your Model Variant

FantasyWorld offers two distinct paths. Pick based on your goal:

REAL Code Examples from the Repository

Let's walk through actual inference code from the FantasyWorld repository, with detailed explanations of what's happening under the hood.

Example 1: Basic Video + 3D Generation with Wan 2.1

This is the entry-point script for researchers reproducing the ICLR 2026 paper:

# Download base Wan model and FantasyWorld weights via HuggingFace
pip install -U "huggingface_hub"

# Pull the 14B parameter video foundation model
hf download "Wan-AI/Wan2.1-I2V-14B-480P" \
    --local-dir ./models/Wan-AI/Wan2.1-I2V-14B-480P

# Pull the FantasyWorld geometry-aware adapter weights
hf download "acvlab/FantasyWorld-Wan2.1-I2V-14B-480P" \
    --local-dir ./models/FantasyWorld-Wan2.1-I2V-14B-480P/

The hf download command uses the HuggingFace CLI to fetch ~28GB of weights (14B parameters at FP16). The separation of base model and FantasyWorld adapter is intentional — you can swap different geometry configurations without re-downloading the massive video backbone.

Now for the actual inference:

python inference_wan21.py \
    --wan_ckpt_path ./models/Wan-AI/Wan2.1-I2V-14B-480P \
    --model_ckpt ./models/FantasyWorld-Wan2.1-I2V-14B-480P/model.pth \
    --image_path ./examples/images/input_image.png \
    --camera_json_path ./examples/cameras/camera_data.json \
    --prompt "In the Open Loft Living Room, sunlight streams through large windows, highlighting the sleek fireplace and elegant wooden stairs." \
    --output_dir ./output-wan21 \
    --sample_steps 50 \
    --using_scale True 

Parameter breakdown:

• --wan_ckpt_path: The frozen Wan 2.1 video diffusion backbone. This provides the "imagination prior" — the ability to generate plausible video dynamics.
• --model_ckpt: The trained FantasyWorld adapter weights. This injects geometric consistency into Wan's latent space.
• --camera_json_path: The secret sauce. A JSON file defining explicit camera extrinsics and intrinsics for each frame. No more hoping the model guesses your intended camera motion.
• --using_scale True: Enables scale normalization for the geometry branch. Critical for preventing depth collapse in indoor scenes.
• --sample_steps 50: Standard DDPM sampling. Reduce for speed, increase for quality.

Example 2: Production-Quality Generation with Wan 2.2

The enhanced variant supports image-to-image video generation with start and end frame conditioning:

# Download enhanced base model, reward LoRAs, and FantasyWorld v2 weights
pip install -U "huggingface_hub"

# Wan 2.2 Fun with camera control capabilities
hf download "alibaba-pai/Wan2.2-Fun-A14B-Control-Camera" \
    --local-dir ./models/PAI/Wan2.2-Fun-A14B-Control-Camera

# Reward model LoRAs for human preference alignment
hf download "alibaba-pai/Wan2.2-Fun-Reward-LoRAs" \
    --local-dir ./models/PAI/Wan2.2-Fun-Reward-LoRAs

# FantasyWorld v2 optimized weights
hf download "acvlab/FantasyWorld-Wan2.2-Fun-A14B-Control-Camera" \
    --local-dir ./models/FantasyWorld-Wan2.2-Fun-A14B-Control-Camera/

Notice we're downloading three separate components: base model, reward LoRAs (for quality tuning), and FantasyWorld adapters. This modular design lets you experiment with different reward models without retraining the full stack.

python inference_wan22.py \
    --image_path ./examples/images/input_image.png \
    --end_image_path ./examples/images/end_image.png \
    --wan_ckpt_path ./models/ \
    --camera_json_path ./examples/cameras/camera_data.json \
    --prompt "In the Open Loft Living Room, sunlight streams through large windows, highlighting the sleek fireplace and elegant wooden stairs." \
    --model_ckpt_high ./models/FantasyWorld-Wan2.2-Fun-A14B-Control-Camera/high_noise_model.pth \
    --model_ckpt_low ./models/FantasyWorld-Wan2.2-Fun-A14B-Control-Camera/low_noise_model.pth \
    --output_dir ./output-wan22 \
    --sample_steps 50 \
    --using_scale True

Critical differences from Wan 2.1 variant:

• --end_image_path: Enables image-to-image-to-video — specify both start and end frames, and FantasyWorld interpolates a geometrically consistent camera path between them.
• --model_ckpt_high and --model_ckpt_low: Dual noise-level specialists. The "high noise" model handles early denoising steps where structure emerges; the "low noise" model refines fine details. This isn't a hack — it's a principled decomposition based on the observation that geometry and appearance operate at different noise scales.
• --wan_ckpt_path ./models/: Points to parent directory; the script auto-discovers subdirectories. Cleaner for multi-model deployments.

Example 3: Alternative ModelScope Downloads (China-Optimized)

For researchers in mainland China or those with HuggingFace connectivity issues:

pip install -U modelscope

# Mirror of Wan 2.1 base
modelscope download "Wan-AI/Wan2.1-I2V-14B-480P" \
    --local_dir ./models/Wan-AI/Wan2.1-I2V-14B-480P

# Mirror of FantasyWorld v1 weights
modelscope download "amap_cvlab/FantasyWorld-Wan2.1-I2V-14B-480P" \
    --local_dir ./models/FantasyWorld-Wan2.1-I2V-14B-480P/

ModelScope downloads often achieve 3-5x faster speeds for users in Asia-Pacific regions, with identical file integrity verification.

Advanced Usage & Best Practices

🎯 Camera Trajectory Design

The camera_data.json format accepts standard COLMAP-format camera parameters. For best results:

• Smooth trajectories: Avoid instantaneous acceleration; the geometry branch expects physically plausible motion
• Sufficient baseline: Minimum 10cm camera translation for reliable depth estimation in indoor scenes
• Forward motion preferred: The VGGT pretraining emphasizes forward-facing scenarios

⚡ Inference Optimization

# Reduce sample steps for rapid prototyping (quality trade-off)
--sample_steps 25

# Disable scale normalization only for outdoor, metrically-scaled scenes
--using_scale False

🔧 Multi-GPU Deployment

The 14B parameter models benefit from tensor parallelism across 2x A100 80GB or 4x A6000 48GB. The repository supports automatic device mapping via accelerate:

accelerate launch --multi_gpu inference_wan21.py [args...]

🧪 Custom Fine-Tuning

While official fine-tuning scripts aren't released yet, the modular architecture suggests:

• Freeze wan_ckpt_path backbone
• LoRA-tune only the IRG block adapters
• Use your own camera-annotated video data

Community implementations are emerging — watch the repository for updates.

Comparison with Alternatives

The verdict: If you need both video and 3D with guaranteed geometric alignment, FantasyWorld is currently the only open-source option. Closed systems like Sora may produce flashier demos, but you cannot extract 3D structure, cannot control cameras precisely, and certainly cannot build research on top of them.

FAQ

What hardware do I need to run FantasyWorld?

Minimum: 1x NVIDIA A100 40GB for the 14B models. Recommended: 2x A100 80GB or 4x RTX A6000 for comfortable inference with batching. The 14B parameter count is non-negotiable — this is frontier-grade generative modeling.

Can I use FantasyWorld commercially?

Check the license files in each model repository. The Wan base models have their own licenses; FantasyWorld adapter weights are typically research-friendly but verify before production deployment.

How does this differ from "video-to-3D" reconstruction methods?

FantasyWorld is not reconstruction — it's joint generation. Traditional methods take existing video and recover 3D. FantasyWorld generates both simultaneously from a single image + camera specification, with cross-branch consistency enforced architecturally.

Is the geometry output usable for game engines?

The point cloud and depth outputs require post-processing (Poisson reconstruction, meshing) for direct engine import. The consistency guarantee means this pipeline actually works, unlike trying to mesh inconsistent multi-view diffusion outputs.

Can I train FantasyWorld on my own data?

Official training code is not yet released (as of February 2026). The architecture description in the ICLR paper is sufficiently detailed for reimplementation by experienced practitioners. Community training frameworks are anticipated.

Why two separate model checkpoints for Wan 2.2?

The high/low noise decomposition is empirically motivated: early denoising steps require coarse geometric structure (high noise model), while late steps need fine detail preservation (low noise model). Training separate specialists outperforms a single model by ~15% on WorldScore metrics.

How do I cite FantasyWorld in my research?

@inproceedings{
    dai2025fantasyworld,
    title={FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction},
    author={Yixiang Dai and Fan Jiang and Chiyu Wang and Mu Xu and Yonggang Qi},
    booktitle={The Fourteenth International Conference on Learning Representations},
    year={2026},
    url={https://openreview.net/forum?id=3q9vHEqsNx}
}

Conclusion: The Future of World Generation Is Unified

FantasyWorld isn't just another generative model — it's a declaration of independence from fractured pipelines. By proving that video and 3D generation can be unified under geometric consistency, the AMAP team has opened a door that the entire field will walk through.

The evidence is compelling: #1 on WorldScore, ICLR 2026 acceptance, and fully open weights that you can download and run today. Whether you're a researcher pushing the boundaries of world models, a developer building the next generation of creative tools, or an engineer who simply refuses to maintain two incompatible pipelines anymore — FantasyWorld demands your attention.

The separate video-and-3D era is ending. The unified world modeling era is beginning.

Clone it. Run it. Build on it.

👉 https://github.com/Fantasy-AMAP/fantasy-world

Star the repo, join the community, and start generating worlds that actually make geometric sense. Your future self — and your GPU budget — will thank you.

Found this breakdown valuable? Share it with your ML team, bookmark it for your next world-generation project, and watch the repository for training code releases.