This work introduces a framework for reward-free continual adaptation using world models. The approach is demonstrated across three distinct space robotics domains: planetary traversal, orbital navigation, and precision assembly. In each case, severe morphological failures are simulated through actuator malfunctions or kinematic misalignments to introduce significant discrepancies between pre-trained and degraded dynamics. Under time and interaction constraints, the agent is evaluated against a zero-shot baseline, an agent with privileged reward information, and one retrained from scratch. This work represents a step towards enabling truly resilient autonomy for extreme environments, where robots can adapt to unforeseen challenges without human intervention.
Methodology
The framework for reward-free continual adaptation builds upon the DreamerV3 architecture, which leverages a latent-state world model to compress high-dimensional observations into compact representations. During pre-training, the standard procedure is followed: the world model is jointly optimized on observed trajectories while an actor-critic policy is simultaneously trained entirely within synthesized rollouts of the world model.
The core scientific exploration lies in the online adaptation phase. Only the transition dynamics of the world model are updated, while the encoder and other heads, including the reward predictor, remain frozen. With this minimal change, the agent can leverage the latent reward landscape learned during pre-training to guide adaptation to new environment dynamics.
The workflow is divided into two distinct phases:
- World model pre-training.
- Reward-free adaptation.
Reward-Free Adaptation
Upon deployment, any robot is expected eventually to encounter novel environmental conditions and hardware degradation that significantly alter the underlying transition dynamics. Because the actor-critic policy was trained entirely within imagination, its behavior inherently captures the transition dynamics encoded in the original world model. Any significant divergence in the true dynamics of the new environment would therefore lead to catastrophic failure of the zero-shot policy, because sampled actions would no longer yield the expected outcomes.
Online adaptation is critical for recovering performance. The key challenge is that rewards may be unobservable. This framework addresses that challenge by leveraging the pre-trained reward predictor, which captures the underlying reward structure within the latent space of the world model. By freezing this head during adaptation, the actor-critic is expected to continue receiving meaningful reward signals based on its updated latent representations, even as the transition dynamics change.
This allows the agent to adapt its policy through unsupervised rollouts that update the transition dynamics while maintaining a consistent reward landscape.
For fixed-horizon tasks, the continuity predictor can either remain frozen or be updated to reflect new episode termination conditions. It is frozen here because termination conditions remain unchanged across the pre-training and deployment environments. If degradation significantly alters the episode structure, updating this predictor may be necessary.
Experimental Results
A series of simulation-only experiments demonstrates key aspects of the adaptation framework.
Task Formulation and Failure Modes
Three distinct tasks were designed across planetary and orbital domains, each paired with an unmodeled morphological failure that induces a significant shift in the underlying transition dynamics. All tasks are implemented on top of NVIDIA Isaac Lab within the scope of the Space Robotics Bench.
Planetary Traversal
The planetary traversal task uses an experimental rover inspired by NASA JPL’s ERNEST prototype. The 12-actuator rover features a double-sided active gimbal suspension system for maintaining ground contact and stability across extreme terrain.
The rover is tasked with traversing a procedurally generated obstacle course while maintaining a target velocity vector. A failure mode is simulated by locking the steering and drive joint of the front-right wheel, which introduces significant asymmetric drag.
Experimental Protocol
Throughout both phases, the agent maintains access to proprioceptive observations alongside the state for relative target tracking in planetary traversal and orbital navigation, or the pose of the bolt in screwdriving assembly.
The action space remains unchanged even after morphological degradation renders certain actions obsolete because of actuator failure. For each task, the adaptation phase is repeated across three random seeds with identical pre-trained models to evaluate robustness.
Performance Evaluation
Normalized task-specific progress metrics are tracked for four agents:
- A zero-shot baseline that evaluates the pre-trained policy.
- An agent retrained from scratch on the new dynamics, serving as an asymptotic upper bound.
- An adaptive agent with access to privileged rewards.
- The reward-free adaptive agent.
Aside from the zero-shot baseline, the reward-free agent is the only adaptive baseline that lacks access to the true reward signal. It relies entirely on the latent reward landscape encoded within the world model, making it viable for real-world applications.
The zero-shot policies fail catastrophically across all domains because of the unmodeled dynamics shift. The retrained baseline confirms that the tasks remain solvable under hardware degradation, while also highlighting the extreme sample inefficiency of learning from scratch. The privileged agent demonstrates rapid recovery, achieving performance close to the retrained upper bound.
The reward-free agent also shows promising initial recovery, demonstrating that latent world models can guide adaptation without external rewards. However, its learning profiles reveal clear limitations. In all three domains, the agent consistently underperforms compared with the privileged baseline. After an initial performance gain, it exhibits significant volatility and decay, particularly in the orbital and assembly tasks.
These results suggest that the latent reward landscape is sufficient for short-term policy recovery but lacks the long-term stability of explicit rewards. Updating the transition dynamics on degraded morphologies likely causes the recurrent state-space model representation to drift away from the original latent space.
Discussion and Conclusion
The experimental results validate the central hypothesis that latent-state world models pre-trained across diverse simulations encode a reward landscape capable of guiding online adaptation without observing new rewards. By isolating the dynamics update from reward prediction, this framework addresses a critical bottleneck in space robotics, where computing online rewards is often impossible.
However, late-stage decay exposes a capacity limit within the recurrent state-space model. Because the reward head remains frozen, continuously updating the core transition dynamics on degraded morphologies eventually overwrites the generalized principles acquired during pre-training.
Despite its effectiveness, the framework has clear limitations. To prevent late-stage decay, future work will investigate localized latent-space adapters that strictly bound dynamics updates.
The simulation-only study also bypasses the peculiarities of the sim-to-real gap. Furthermore, the demanding optimization of the online adaptation phase greatly exceeds the strict power constraints of space-grade embedded compute modules, necessitating future breakthroughs in efficient in-situ learning.
Frequently Asked Questions
What is reward-free continual adaptation? It is an approach that adapts a robot to changed dynamics without observing new external rewards, using a pre-trained latent reward landscape.
Which parts of the world model are updated during adaptation? Only the transition dynamics are updated. The encoder, reward predictor, and continuity predictor remain frozen under the conditions described.
How does the reward-free agent compare with the other agents? It initially recovers performance but consistently underperforms the privileged-reward agent and later shows volatility and decay.
What causes late-stage performance decay? Continually updating transition dynamics on degraded morphologies likely causes the recurrent state-space model representation to drift from the original latent space.