SJTU-DeepVisionLab

SJTU-DeepVisionLab / WorldAct

Public

Official implementation of WorldAct

19
0
89% credibility
Found May 20, 2026 at 19 stars -- GitGems finds repos before they trend. Get early access to the next one.
Sign Up Free
AI Analysis
AI Summary

WorldAct is a research project from Shanghai Jiao Tong University that transforms static 3D worlds into interactive, editable scenes. It works by analyzing a generated 3D environment, identifying individual objects that can be touched or moved, creating clean 3D models of each object, and filling in the background. The result is a scene where objects can be picked up, rearranged, edited, and interacted with while keeping the rest of the world coherent. This enables applications like video games, robot training simulations, and immersive virtual experiences.

How It Works

1
🔍 You discover WorldAct

You come across a research paper or website about transforming frozen 3D worlds into living, interactive scenes.

2
🎓 You learn about the problem

You understand that today's 3D world generators create beautiful scenes that look great but can't be touched or changed.

3
✨ You see the magic happen

WorldAct takes a static 3D world and automatically finds every object that can be picked up, moved, or played with.

4
🧩 Objects become separate and ready

Each object gets its own clean 3D model that can be edited on its own, while the rest of the world stays perfectly intact.

5
🎮 Your world comes alive

The transformed scene now supports moving objects around, editing them, and running tasks where characters interact with the environment.

6
You put your interactive world to use
🎮
Build interactive games

Create game environments where players can pick up, move, and interact with objects

🤖
Train robot simulations

Use the scenes to teach AI agents how to navigate and manipulate real-world objects

🎨
Design immersive experiences

Craft virtual worlds where users can freely explore and modify the environment

Sign up to see the full architecture

4 more

Sign Up Free

Star Growth

See how this repo grew from 19 to 19 stars Sign Up Free
Repurpose This Repo

Repurpose is a Pro feature

Generate ready-to-use prompts for X threads, LinkedIn posts, blog posts, YouTube scripts, and more -- with full repo context baked in.

Unlock Repurpose
AI-Generated Review

What is WorldAct?

WorldAct is a research framework that takes static 3D worlds generated by systems like Marble and transforms them into interactive, editable scenes. It uses a multimodal agent to decompose monolithic 3D environments, identify individual objects, reconstruct their meshes, and inpaint the background. The result is scenes that support object-level editing, collision-aware manipulation, and embodied task execution while keeping the overall scene coherent.

Why is it gaining traction?

The hook here is bridging the gap between generative 3D world creation and actual interactivity. Most 3D world models produce beautiful but static assets. WorldAct adds the layer of editability and physical interaction that immersive content creation and embodied AI simulation demand. The pipeline handles the full workflow from decomposition to mesh reconstruction to background restoration, which means developers get a complete path from static scene to interactive scene.

Who should use this?

This is for researchers and developers working on embodied AI, robotics simulation, and immersive content pipelines. If you're building systems that need to manipulate or interact with procedurally generated 3D environments, WorldAct addresses a real gap. Game developers using generated worlds who need editable objects would also benefit. However, the project is extremely early stage -- training code and datasets are still marked TBD.

Verdict

WorldAct solves a legitimate problem in 3D world modeling, but with only 19 stars and critical components still unreleased, treat this as interesting research to watch rather than production-ready tooling. The 0.9% credibility score reflects that this is a fresh academic release with no community validation yet. Follow the repository and check back when training code and datasets ship.

Sign up to read the full AI review Sign Up Free

Similar repos coming soon.