Vision-Language-Action Models
Beyond general supervised fine-tuning strategies, recent studies have also begun to examine the interactions among functionally distinct VLA modules during adaptation. Knowledge insulation investigates gradient interference between the vision-language backbone and the action expert, isolating action-side gradients to reduce their interference with pretrained vision-language knowledge. However, it mainly focuses on gradient isolation during supervised training and does not consider separate RL optimization for different modules during online post-training or investigate the effects of relative update frequencies between semantic-side and action-side modules.
Asynchronous VLA Execution and Optimization
Overall, existing studies have explored VLA model design, RL post-training, and asynchronous inference and system optimization. However, the parameter-update frequencies of functionally distinct modules during RL post-training remain underexplored, particularly mechanisms that assign different update frequencies to semantic-side and action-side modules.
Semantic-Action Frequency-Decoupled Optimization
The omitted expressions represent the effective update frequencies of the action expert and the semantic projection layer, respectively.
The action-to-semantic update-frequency ratio is then defined as the ratio between the action expert’s effective update frequency and the semantic projection layer’s effective update frequency.
Experiments
We conduct experiments to answer the following questions:
- Does TEMPO improve long-horizon VLA performance over the pretrained state-of-the-art VLA policy and the RL post-training baseline?
- Do the semantic projection layer and action expert provide complementary benefits during RL post-training?
- How does the update-frequency ratio between the semantic projection layer and the action expert affect performance?
- How does the post-training task-set size affect performance?
Real-World Experiments
Experimental Setup
We conduct multi-stage language-conditioned manipulation experiments on a physical platform comprising a 6-DoF Synria Alicia-D robot arm and a single-DoF gripper. Visual observations are provided by two Intel RealSense cameras: a D435i captures a third-person RGB view of the workspace, while a wrist-mounted D405 records close-range observations around the gripper.
We evaluate two multi-stage manipulation tasks, both of which require the robot to satisfy a prerequisite before interacting with the target object. For each task, we collect 60 human demonstration episodes, resulting in 120 episodes in total. Each episode contains synchronized images, robot states, executed action commands, and language instructions. These demonstrations provide task-specific supervised data before online RL post-training.
Online RL Post-Training
Evaluation reward during online RL post-training is reported for each task. For each random seed, we evaluate 20 trials at every checkpoint and sum the binary task rewards, assigning 1 to success and 0 to failure.
Across both tasks, the evaluation rewards fluctuate during the early stage and then gradually improve and stabilize as post-training proceeds. Compared with FLOWER-RL, TEMPO reaches and maintains higher reward levels during the later stages of post-training.
Conclusion
This paper presented TEMPO, a semantic-action decoupled RL post-training framework that optimizes different VLA components through dedicated RL loops, adapting the semantic projection layer while keeping the pretrained vision-language backbone frozen, and optimizing the action expert to learn control- and execution-related behaviors from online feedback.
A module-specific update-frequency strategy coordinates their adaptation, updating the semantic projection layer less frequently to mitigate drift in the latent action space and the action expert more frequently to incorporate control feedback efficiently.
Evaluated on CALVIN and two real-world manipulation tasks, TEMPO achieves improved long-horizon performance over both state-of-the-art VLA models and the RL post-training baseline in simulation, as well as higher rewards in physical experiments.
Future work will explore learning a language-conditioned dense reward function to provide richer intermediate feedback than the sparse instruction-completion reward. Such progress-aware signals may improve credit assignment and sample efficiency during RL post-training, especially for long-horizon tasks.
Frequently Asked Questions
What is TEMPO? TEMPO is a semantic-action decoupled reinforcement learning post-training framework for vision-language-action models.
Which VLA components does TEMPO optimize? TEMPO adapts the semantic projection layer and optimizes the action expert while keeping the pretrained vision-language backbone frozen.
How does TEMPO coordinate module adaptation? It updates the semantic projection layer less frequently and the action expert more frequently.
Where was TEMPO evaluated? TEMPO was evaluated on CALVIN and two real-world manipulation tasks using a Synria Alicia-D robot arm.