Problem Formulation
By autoregressively modeling action-sequence-conditioned future prediction, the world model predicts future observations conditioned on sequences of actions and enables closed-loop interaction with both human operators and downstream policies.
World Modeling With Visual Actions
We then define the world model as a model that predicts future observations based on visual action sequences and historical context.
Conditioning on visual action sequences enables the system to learn scene interactions driven directly by embodiment motion.
Efficient Closed-Loop Interaction
To enable closed-loop interaction with human operators and downstream policies, the world model predicts the next observation in response to input actions and autoregressively feeds generated observations back as historical context for subsequent rollouts.
Following the described approach, we cache the key-value pairs of preceding tokens as historical context while applying full self-attention within the newly generated token block at each prediction.
Experiments
In this section, we design comprehensive experiments to evaluate GeniWorld by addressing the following questions:
Generative Fidelity and Generalization
Can GeniWorld generate high-fidelity video predictions and generalize effectively across diverse, unseen scenarios?
Policy Evaluation Reliability
Can GeniWorld reliably evaluate downstream policies while maintaining a strong correlation with real-world performance under scene perturbations and out-of-domain settings?
Data Synthesis for Policy Improvement
Can GeniWorld synthesize effective training data from limited recorded trajectories to improve policy learning and performance without the cost of constructing new real-world setups?
Real-World Platform Setup
We construct our experimental platform using a dual-arm Xtrainer robotic system. Following hand–eye calibration, we integrate the robot’s URDF into Isaac Sim for motion control and replicate the physical camera setup to render visually consistent observations.
We design four representative manipulation tasks involving diverse objects and complex physical dynamics:
- Move Bowl
- Fold Towel
- Place Mug
- Open Drawer
Demonstrations are collected using the Xtrainer teleoperation system. During training-data collection, we maintain a clean tabletop environment and randomize only the poses of the manipulated objects across episodes.
World Model for Policy Evaluation
We investigate whether GeniWorld can serve as an effective offline evaluator for downstream policies. By executing the same policy checkpoints in both GeniWorld and real-world setups, we quantitatively measure the correlation between simulated rollouts and physical task success rates.
Policies are fine-tuned from a vision-language-action model using the collected real-world demonstrations. For each trial, the policy is initialized with the same first-frame visual observation in GeniWorld and the real-world environment.
Human evaluators assess binary task success, while a vision-language model serves as an automated judge of manipulation plausibility.
Data Synthesis for Policy Improvement
We next ask whether GeniWorld can leverage limited demonstrations from a narrow distribution to synthesize diverse manipulation data, thereby improving policy performance and generalization under complex conditions.
Conclusion
In this work, we introduce GeniWorld, a generalizable interactive world model conditioned on visual actions that enables spatially grounded and closed-loop robot interaction.
Despite training on limited fixed-scene demonstrations, GeniWorld achieves strong in-domain performance and reliable zero-shot generalization to unseen scenarios. It enables robust policy evaluation and generates scalable, highly diverse synthetic trajectories that improve downstream policy performance.
These results highlight the strong potential of generalizable world models to provide scalable imagination spaces for embodied robot learning.
Frequently Asked Questions
What is GeniWorld? GeniWorld is a generalizable interactive world model conditioned on visual actions for spatially grounded, closed-loop robot interaction.
What robotic platform is used in the experiments? The experiments use a dual-arm Xtrainer robotic system, with its URDF integrated into Isaac Sim after hand–eye calibration.
Which manipulation tasks are evaluated? The evaluated tasks are Move Bowl, Fold Towel, Place Mug, and Open Drawer.
How is GeniWorld used to improve policies? GeniWorld synthesizes diverse manipulation data from limited demonstrations to improve policy learning and generalization.