Comparison of driving world models over model scale (parameters), FVD (6s rollouts) and representation quality based on probing for semantic segmentation. FVD evaluated on Waymo and segmentation on Cityscapes.
Performance (FVD) of long-horizon rollouts on nuPlan-turns over consecutive 4s windows. Models with good initial video quality can quickly degrade over time. FVD-slope captures this progressive fidelity loss.
Abstract
Current world models typically operate at a single abstraction level, favoring perceptual fidelity but lacking the spatial and semantic reasoning needed for downstream driving tasks. We propose a hierarchical driving world model that separates prediction into a high-level long-horizon scene forecaster and a low-level detail generator conditioned on it. This design improves both visual fidelity and spatial-semantic representation quality. We also introduce a two-stage training strategy: diffusion-forcing pretraining for richer representations, followed by teacher-forcing fine-tuning for stable autoregressive rollouts. Our method achieves state-of-the-art results on standard driving world model benchmarks, including long-horizon fidelity, counterfactual steering responsiveness, and internal representation quality.
Method
Hierarchical driving world model. The abstract predictor operates over a long temporal context to forecast a future state in latent space, capturing abstract scene dynamics over long horizons, and enabling steering control. The detail predictor is conditioned on this abstract prediction and generates fine-grained short-horizon frames, enabling high-fidelity local prediction grounded in long-range temporal context.