jwang771/3d-world-simulator

Project overview

This is research code for simulating 3D deformable-object dynamics with a hybrid particle-voxel representation. A two-stage pipeline is used: a hybrid particle-voxel autoencoder compresses each scene into a latent representation, and a 3D latent dynamics model predicts the next latent state from recent latents and actions. The README lists six task settings (cloth, box, pusht, plug, marker, rope), each with its own voxel grid, workspace bounds, and tactile-sensing configuration in tasks.py. The repository is aimed at researchers who want to train or extend the model on their own manipulation datasets.

Setup and data

Create the Conda environment from the repository's environment file:

conda env create -f environment.yml
conda activate 3dws

Datasets go under a single data root, for example data/bimanual_plug_insertion_v3/{train,val}. The expected file layout is described in docs/data_format.md. Task-specific dataset and parameter settings live in tasks.py.

Training workflow

Training has three steps. First, train the autoencoder for a task:

python scripts/train_vae.py --task plug --data-root data --out-dir runs/plug/vae

Second, export latents for the train and validation splits using the trained checkpoint:

python scripts/export_latents.py --vae-ckpt runs/plug/vae/checkpoints/latest.pth \
  --packed-dir data/bimanual_plug_insertion_v3/train --out-dir runs/plug/latents/train
python scripts/export_latents.py --vae-ckpt runs/plug/vae/checkpoints/latest.pth \
  --packed-dir data/bimanual_plug_insertion_v3/val --out-dir runs/plug/latents/val

Third, train the dynamics model on the precomputed latents:

L=runs/plug/latents
python scripts/train_dynamics.py --latent-dir $L/train --val-latent-dir $L/val --out-dir runs/plug/dynamics \
  --n-history 8 --ar-schedule 1 2 3 4 5 6 7 8 9 10 --ar-schedule-epochs 50 100 150 200 250 300 350 400 450 500

--n-history controls how many past latents are used to predict the next one; the autoregressive schedule specifies which history lengths are introduced at the given epochs.

Constraints

• No pre-trained checkpoints, datasets, or inference code have been published yet; the README says they will be uploaded later. Today only the training scripts and environment definition are described.
• The repository does not declare a license, so reuse and redistribution terms are unclear.
• The pipeline assumes prepared datasets in the documented packed format and a task configuration that already exists in tasks.py; a new task requires adding that configuration and converting data to the expected layout.
• The project website and paper are the main references for method details and results (project page, paper).

HotRepo's take

On November 3, 2026, the project appeared on HotRepo's new-project list for Python with 102 stars gained in a window starting the previous day, alongside 7 forks. No GitHub all-language daily rank has been recorded. These numbers reflect early attention to a paper-backed research repository, not a runnable release: without checkpoints, data, or inference code, the available material supports reproduction of the training pipeline only, and that pipeline has not been exercised here.