Unlocking Long-Horizon Planning: How GRASP Makes World Models Practical for Control

Introduction: The Promise and Pitfalls of Learned World Models

Learned world models—neural networks that predict future states from current observations and actions—have advanced rapidly. They can now simulate long sequences of high-dimensional visual data, adapt to diverse tasks, and even serve as general-purpose simulators. Yet, despite their predictive power, using these models for robust planning and control over extended time horizons remains a stubborn challenge. Traditional gradient-based planners often become trapped in ill-conditioned optimization landscapes, encounter deceptive local minima, and struggle with high-dimensional latent spaces. This article explores why long-horizon planning is so brittle with modern world models and introduces GRASP, a new approach that overcomes these fundamental bottlenecks.

The Challenge of Planning at Scale

As world models grow in capacity, they capture increasingly complex dynamics—yet the very features that make them powerful also amplify planning difficulties. Key problems include:

• Ill-conditioned optimization: Gradient paths become highly curved, making standard descent methods unstable.
• Non-greedy structure: Actions that look suboptimal in the short term may be necessary for long-term success, creating misleading local minima.
• High-dimensional latent spaces: Vision-based models introduce fragile state-input gradients that corrupt learning.

These issues are especially pronounced when the planning horizon stretches beyond a few steps. Long sequences compound gradient errors, making it nearly impossible to find effective action sequences with conventional tools.

How GRASP Overcomes Fundamental Bottlenecks

GRASP (Gradient-based Adaptive Stochastic Planning) tackles these challenges through three clever innovations. It doesn't discard the power of modern world models but instead redesigns the planning loop to work harmoniously with them.

1. Lifting the Trajectory into Virtual States

Instead of optimizing actions sequentially over time, GRASP introduces a set of virtual states—auxiliary variables that parallelize the optimization across all time steps. This trick transforms a sequential problem into a batch one, enabling more stable gradient updates and reducing the risk of vanishing or exploding gradients. By treating each time step independently during optimization, the planner can evaluate and adjust multiple actions simultaneously.

2. Injecting Stochasticity for Exploration

To avoid getting stuck in poor local minima, GRASP adds controlled noise directly to the state iterates. This stochastic perturbation acts as a form of exploration, allowing the planner to escape suboptimal basins and discover better action sequences. The noise is calibrated to gradually diminish as planning converges, balancing exploration with precision.

3. Reshaping Gradients to Avoid Brittle Pathways

One of the most insidious problems in vision-based models is the reliance on brittle gradients that flow through high-dimensional image encoders. GRASP reshapes these gradients so that actions receive clean, actionable signals without being distorted by the visual processing pipeline. This is achieved by decoupling the gradient computation for actions from the state-input gradients, effectively bypassing the noise-prone parts of the model.

A Deeper Look: Why Long Horizons Break Traditional Planners

To appreciate GRASP's contribution, it helps to understand why long horizons are the real stress test. Consider a robot navigating a maze: a short-horizon planner might only see the immediate wall and fail to plan a detour. But a long-horizon planner must reason about a series of turns, obstacles, and goals. Gradient-based methods typically backpropagate through the entire trajectory, and as the horizon lengthens, the gradient signal becomes either vanishingly small or explosively large. Moreover, the loss landscape becomes riddled with saddle points and flat regions. GRASP's virtual states and stochasticity directly counteract these phenomena, while gradient reshaping ensures that the action updates are grounded in the dynamics model's predictions rather than being corrupted by visual artifacts.

GRASP in Action: Key Results and Takeaways

In experiments across simulated environments, GRASP consistently outperforms prior gradient-based planners, especially as the horizon exceeds 50 steps. It achieves higher cumulative rewards, requires fewer iterations to converge, and is more robust to model inaccuracies. The approach is also compatible with existing world model architectures, meaning it can be dropped into many current systems with minimal modification.

For researchers and engineers working on model-based reinforcement learning, GRASP offers a practical recipe for scaling planning to realistic time frames. The three components work synergistically: virtual states stabilize optimization, stochasticity provides exploration, and gradient reshaping preserves signal quality.

Conclusion: Toward Robust Model-Based Control

World models are becoming more powerful, but unlocking their potential for control requires rethinking how we plan within them. GRASP demonstrates that by addressing the fundamental bottlenecks of long-horizon optimization, we can harness these models effectively. This work paves the way for deploying learned dynamics in real-world systems where planning over extended horizons is essential—from robotics to autonomous driving. As the field continues to push toward general-purpose simulators, tools like GRASP will be critical to turn predictive power into actionable intelligence.