URLab-Sim/UnrealRoboticsLab: MuJoCo Physics Meets Unreal Engine 5 Rendering

URLab-Sim/UnrealRoboticsLab: MuJoCo Physics Meets Unreal Engine 5 Rendering

Robotics simulation has long forced a difficult trade-off: physics accuracy versus visual fidelity. MuJoCo delivers industry-standard contact dynamics and articulated body simulation, but its native renderer is built for analysis, not presentation. Unreal Engine 5 offers photorealistic lighting, materials, and camera systems, yet its built-in physics engines prioritize game performance over scientific precision. For researchers training vision-based policies, validating sim-to-real transfer, or generating synthetic datasets, this gap means maintaining separate pipelines or accepting compromise in one domain.

URLab-Sim/UnrealRoboticsLab bridges this divide directly. It is an Unreal Engine 5 plugin that embeds MuJoCo as the physics backend while preserving Unreal's rendering pipeline. The result: accurate contact dynamics, tendons, and actuators running on a dedicated physics thread, with RGB, depth, and segmentation output from Unreal's camera system. This article covers what URLab does, how it works, and how to get started.

What is URLab-Sim/UnrealRoboticsLab?

URLab (Unreal Robotics Lab) is an open-source C++ plugin for Unreal Engine 5.7, developed by Jonathan Embley-Riches and collaborators at UCL, with research contributions from Jianwei Liu, Simon Julier, and Dimitrios Kanoulas. The project is licensed under Apache 2.0 and has accumulated 483 GitHub stars and 50 forks as of its last commit on July 8, 2026.

The plugin's core architectural decision is embedding MuJoCo 3.9+ directly into Unreal's editor and runtime, rather than communicating with an external physics process. This co-location eliminates IPC overhead and synchronization complexity that plague multi-process simulators, while keeping physics execution on its own thread to avoid blocking the render loop.

URLab is purpose-built for robotics research and development—not game development. It targets scenarios where physics correctness matters: legged locomotion, manipulation with contact-rich interactions, tendon-driven systems, and muscle-actuated models. The ICRA 2026 paper (arXiv:2504.14135) provides the academic foundation, and the project maintains comprehensive documentation at urlab-sim.github.io/UnrealRoboticsLab.

The plugin's relevance now stems from two converging trends: the maturity of MuJoCo as an open-source physics engine (post-Google DeepMind stewardship), and Unreal Engine 5's Nanite and Lumen technologies making film-quality rendering real-time accessible. URLab sits at this intersection for a technically demanding audience that needs both.

Key Features

Native MuJoCo Integration. URLab imports MJCF .xml files directly through Unreal's Content Browser, automatically constructing Articulation Blueprints with joints, actuators, tendons, muscles, and flexcomp support. Static meshes can alternatively become physics bodies through a single component attachment.

Decoupled Physics-Render Threads. Physics simulation executes on a dedicated thread while Unreal renders asynchronously. This architecture preserves MuJoCo's deterministic stepping while maintaining frame rates suitable for interactive development and policy training.

Multi-Modal Control Interfaces. Actuators accept commands from Blueprints, an in-editor dashboard, or external Python^{↗ Bright Coding Blog} and ROS 2 clients via ZMQ. PD controllers with live gain tuning enable rapid controller iteration without recompilation.

Sensor Suite Integration. Beyond MuJoCo's native sensors, URLab exposes Unreal's camera pipeline with real camera intrinsics: RGB, depth, and segmentation rendering. This enables direct generation of training data for vision-based policies without additional synchronization layers.

Deterministic Recording and Replay. Full simulation episodes can be captured and replayed with identical outcomes—critical for debugging rare failures and benchmarking controller changes.

Visual Debugging Tools. Body-island visualization, segmentation overlays, tendon and muscle rendering, collision geometry display, and mouse-driven body perturbation provide immediate physical intuition without leaving the editor.

Use Cases

Sim-to-Real Policy Validation. Researchers training policies in simulation need confidence that visual observations and physical behaviors transfer. URLab's accurate physics plus photorealistic rendering reduces the reality gap from both dynamics and perception sides—contact dynamics match MuJoCo's validated models, while rendered images include realistic lighting and materials.

Synthetic Dataset Generation. Computer vision researchers can programmatically render RGB-D sequences with perfect ground-truth segmentation, depth, and camera poses. Real camera intrinsics ensure geometric consistency with physical hardware. The deterministic replay system enables identical scene generation with varied lighting or textures.

Tendon and Muscle-Driven Robotics. Few simulators natively support MuJoCo's full tendon and muscle actuator models. URLab exposes these through Unreal's visual environment, enabling biomechanics researchers to study biological motor systems with presentation-quality visualization.

Interactive Controller Development. The live gain tuning and multi-source actuation (Blueprint dashboard, Python, ROS 2) allow rapid iteration on walking controllers or manipulation strategies. Engineers can test parameter sweeps in real-time without rebuilding the simulation.

Educational Demonstrations. Robotics courses benefit from URLab's visual clarity—students observe physical principles through debug overlays while interacting with physically accurate models, bridging theoretical dynamics and intuitive understanding.

Installation & Setup

URLab requires a C++ Unreal project; Blueprint-only projects are insufficient. The supported platform is Unreal Engine 5.7 specifically—the maintainers explicitly warn against 5.8 due to Vulkan driver regressions on Linux.

Step 1: Clone with submodules into your project's Plugins directory.

cd "YourProject/Plugins"
git clone --recurse-submodules https://github.com/URLab-Sim/UnrealRoboticsLab.git

The --recurse-submodules flag is essential: MuJoCo, CoACD, and libzmq are bundled as submodules and built from source, not system dependencies.

Step 2: Build native dependencies.

cd UnrealRoboticsLab/third_party
./build_all.sh        # Linux
# or
.\build_all.ps1       # Windows

This script compiles MuJoCo, the convex decomposition library CoACD, and the ZMQ messaging library against UE's toolchain. CMake 3.24+ is required.

Step 3: Build your Unreal project.

After dependency compilation, build your project through Unreal's standard C++ compilation path: Visual Studio 2022/2025 on Windows, or UE's bundled clang on Linux x86_64.

Platform Requirements:

• Windows: Win64, Visual Studio 2022 or 2025
• Linux: x86_64, UE's bundled clang (avoid 5.8 due to Vulkan issues)
• CMake 3.24+ for dependency builds
• Python 3.11+ (optional, for urlab_bridge external control)

For detailed platform-specific troubleshooting, see the Installation guide and subsequent Quickstart.

Real Code Examples

The README provides one explicit command sequence for installation. Below is the precise reproduction with explanatory context.

Plugin Installation with Submodules:

# Navigate to your C++ project's Plugins directory
cd "YourProject/Plugins"

# Clone URLab including all physics and messaging dependencies
git clone --recurse-submodules https://github.com/URLab-Sim/UnrealRoboticsLab.git

# Build bundled third-party libraries from source
cd UnrealRoboticsLab/third_party
./build_all.sh        # Linux shell script
# .\build_all.ps1     # Windows PowerShell equivalent

The build_all.sh (or .ps1) script handles compilation of MuJoCo 3.9+, CoACD for convex decomposition, and libzmq for network communication. These are not system packages to avoid version conflicts with Unreal's specific toolchain requirements.

Citation for Research Use:

@inproceedings{embleyriches2026urlab,
  title     = {Unreal Robotics Lab: A High-Fidelity Robotics Simulator with Advanced Physics and Rendering},
  author    = {Embley-Riches, Jonathan and Liu, Jianwei and Julier, Simon and Kanoulas, Dimitrios},
  booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2504.14135}
}

Use this BibTeX entry when URLab supports published research. The ICRA 2026 paper provides the formal evaluation and architectural justification for the plugin's design decisions.

Note: The README does not contain additional inline code examples for runtime usage (Python API calls, Blueprint snippets, or ZMQ message formats). The maintainers direct users to the Python & External Control documentation and Guides section for these. This reflects URLab's current documentation structure rather than a gap in functionality.

Advanced Usage & Best Practices

Threading Awareness. Since physics runs on a dedicated thread, avoid direct Blueprint manipulation of MuJoCo state from render-thread events. Use URLab's provided synchronization primitives or the ZMQ bridge for external control to prevent race conditions.

Deterministic Replay Workflows. Record episodes during policy evaluation, then replay with varied rendering parameters (lighting, textures, camera positions) to generate augmented datasets from identical physical trajectories. This multiplies training data without additional simulation cost.

MJCF Import Validation. After dragging .xml files into the Content Browser, verify articulation topology in the debug visualization before controller attachment. Body-island coloring reveals unexpected constraint disconnections that static inspection misses.

Version Pinning. The explicit 5.7-only support means upgrading Unreal requires URLab maintainer validation. Plan project timelines around this constraint, and monitor the Roadmap for version expansion plans.

CI Limitations. Unreal projects cannot use standard CI infrastructure. Contributors should prepare local build verification and test results for PR submission, as noted in CONTRIBUTING.md. This friction is architectural, not policy-driven—factor it into contribution planning.

Comparison with Alternatives

Isaac Sim offers GPU-accelerated physics and massive parallelization for reinforcement learning, but requires NVIDIA hardware and lacks native MuJoCo compatibility. Its Omniverse renderer is capable but distinct from Unreal's ecosystem. Choose Isaac Sim for throughput; choose URLab for MuJoCo-specific dynamics and Unreal rendering integration.

Gazebo/Ignition remains the ROS-native default with broad community support, but its rendering and physics accuracy lag for contact-rich manipulation. URLab targets researchers specifically needing MuJoCo's contact models with superior visual output, accepting the smaller ecosystem and C++ project requirement.

FAQ

Is Blueprint-only project support planned? No. URLab is a C++ plugin requiring C++ project configuration. This is architectural, not temporary.

Can I use Unreal Engine 5.8? The maintainers explicitly advise against 5.8 due to Vulkan driver regressions on Linux. Only 5.7 is tested and supported.

What license applies to my simulations? URLab itself is Apache 2.0. Your simulation content is yours. Third-party components (MuJoCo: Apache 2.0, CoACD: MIT, libzmq: MPL 2.0) have their own terms—see ThirdPartyNotices.txt.

How do I control robots from Python? Use the separate urlab_bridge package via ZMQ. Documentation covers remote control, policy deployment, and ROS 2 bridging.

Is there CI for pull requests? No. Standard CI doesn't work for Unreal projects. PRs require local build verification and passing tests, documented in CONTRIBUTING.md.

Can I import URDF files? The README specifies MJCF .xml import. Convert URDF to MJCF via MuJoCo's utilities first.

Does it run on macOS or ARM64? Only Windows (Win64) and Linux (x86_64) are documented. No Apple Silicon or macOS support is mentioned.

Conclusion

URLab-Sim/UnrealRoboticsLab occupies a specific, valuable niche: researchers and engineers who need MuJoCo's validated physics accuracy combined with Unreal Engine 5's rendering capabilities, without maintaining separate simulation and visualization pipelines. It is not a general-purpose game tool, nor a drop-in replacement for Gazebo's ROS ecosystem. The C++ project requirement, 5.7 version lock, and manual CI process reflect its research-tool orientation.

For teams simulating contact-rich robots, generating synthetic vision data, or studying biomechanical systems, URLab eliminates a significant integration burden. The Apache 2.0 license, active documentation, and ICRA 2026 publication provide credible foundations for adoption.

Clone the repository at https://github.com/URLab-Sim/UnrealRoboticsLab, review the full documentation, and verify your project meets the C++ and 5.7 requirements before installation.