Beyond Policy Alignment:
Closing the Planning–Learning Loop for Robot Control
with Learned World Models

Our algorithm closes the planning–learning loop.

Model predictive control plans robot actions using a learned world model. The model predicts action outcomes, while a critic estimates future return. The robot executes one action, then replans.

The policy proposes actions. Aligning it with the planner leaves three issues: delayed reward propagation, uncertain terminal values, and differences in episode quality.

Our algorithm changes three learning signals while keeping the world model and planning optimizer.

The baseline uses one-step targets. Our algorithm uses hybrid multi-step targets.

Experience updates the model, critic, and policy for future plans.

Hybrid multi-step temporal-difference targets use several rewards to teach earlier decisions. We take available rewards from replay, extend the sequence with the model, and then bootstrap from the critic.

Adaptive terminal estimates lower the online terminal value when target critics disagree. A larger disagreement means a larger penalty, reducing the influence of uncertain values on planning.

Return-weighted actor distillation gives higher-return episodes more influence on imitation. It uses realized episode returns. After a fixed warmup, the extra loss helps the policy learn from planner experience.

On balance-hard, the best seed reaches fifty point nine percent success; the baseline seeds reach zero. In these selected rollouts, our algorithm maintains balance while the baseline falls.

On hurdle, the best seed reaches fifty-eight percent success; the baseline seeds reach zero. These selected rollouts show our algorithm clearing successive obstacles while the baseline falls.

These examples show other humanoid skills. Across the broader benchmark, performance remains task dependent, with the largest gains on balance-hard and hurdle.

In simulation, these selected wrench–nut alignment trials compare our algorithm with the baseline across different starting configurations.

The world model learned in Isaac Gym is deployed with online planning on the real KUKA robot: zero-shot sim-to-real transfer. Contact requires closed-loop realignment.

Additional trials with our algorithm show different starting configurations, including an unseen object size. The world model transfers from simulation without real-world fine-tuning.

The same simulation-trained models also plan for two unseen object sizes. These comparisons report observed success over fifteen trials per method and size.

Closing the planning–learning loop.