OnlineWMCausality-Aware Active Online Learning
for Effective World Modeling

Yikun Miao Fangqi Zhu Quanxin Shou Xiaoyi Pang Zhengyang Yan

Junhao Li Haodong Wang Zicong Hong Song Guo^†

Department of Computer Science and Engineering
The Hong Kong University of Science and Technology
Hong Kong SAR, China

Paper Coming soon Code Coming soon BibTeX

^† Correspondence: songguo@cse.ust.hk

Offline training uses a static simulator dataset, while OnlineWM actively interacts with a simulator and updates the world model in a feedback loop. — Fig. 1. Static offline datasets cannot adapt to a world model’s changing prediction errors. OnlineWM closes the loop between simulator interaction and model learning, acquiring informative experiences and learning how actions change future states.

Abstract

Generative world models aim to predict future states conditioned on actions, where action controllability is fundamental for reliable dynamics modeling. While recent efforts leverage simulator-generated data to enhance this capability, existing training pipelines face two fundamental limitations. First, static offline data collection leads to a distribution misalignment between training sets and the model’s evolving error patterns, failing to resolve critical long-tail scenarios where dynamics predictions remain unreliable. Second, the standard objective of minimizing observational discrepancy often encourages the model to exploit spurious correlations instead of capturing the underlying action-effect causality. To address these limitations, we propose OnlineWM, an online training framework that continuously improves world modeling through active simulator interaction and causality-aware optimization. OnlineWM introduces two key innovations: (1) Active Online Learning: Instead of using fixed datasets, OnlineWM adaptively queries the simulator for new interaction sequences that target the model's current predictive weaknesses, ensuring high-utility data acquisition. (2) Causality-Aware Fine-Tuning: We propose a counterfactual learning strategy that contrasts the outcomes of different actions from identical states, forcing the model to attribute state transitions to specific actions rather than ambient environmental evolution, thereby grounding its predictions in reliable causal mechanisms. By integrating active data acquisition with causal optimization, OnlineWM establishes a closed-loop refinement process that ensures the model is both robust to diverse scenarios and precise in its causal attribution. Extensive experiments demonstrate that OnlineWM significantly enhances action controllability and generalizes effectively to unseen domains, suggesting the learning of physically-grounded causal dynamics rather than simple visual patterns.

Method

OnlineWM pipeline: parallel simulation, difficulty scoring, active sample selection, counterfactual pair construction, joint optimization, and EMA updates. — Fig. 2. OnlineWM combines parallel simulation, difficulty-balanced acquisition and counterfactual optimization in a continuous cycle of data collection and model learning.

Two Core Strategies

Active Online Learning (AOL)

AOL acquires informative online training data by balancing scene and action difficulty. It pairs harder scenes with relatively easier actions, and easier scenes with harder actions. This complementary selection adapts simulator interaction to the model’s evolving prediction errors, prioritizing trajectories that are both novel and learnable.

Causality-Aware Fine-Tuning (CFT)

CFT strengthens action control by contrasting outcomes of different actions from the same conditioning state. For each selected trajectory, the most visually similar alternative serves as a counterfactual negative. The paired supervision guides predictions toward the selected outcome and away from the counterfactual, emphasizing action-dependent differences.

Experiments

We evaluate OnlineWM in terms of visual quality and action controllability. Experiments use held-out Minecraft sequences and report SSIM, LPIPS, VBench, action-token accuracy and relative pose error. Comparisons with random acquisition and AOL-only training assess the benefits of CFT; additional ablations examine counterfactual selection and the acquisition strategy.

↑ higher is better ↓ lower is better

Action controllability

Tab. 2

Action-token accuracy and relative pose error across training variants. Best results are in bold.

Action controllability. Tab. 2 in the paper.
Method	Action Accuracy		Camera Control
Method	Combined↑	Fine-grained↑	RPE_rot↓	RPE_trans↓
Base Model	0.240	0.490	0.6906	0.0235
+ Random	0.356	0.569	0.6393	0.0199
+ AOL	0.393	0.610	0.5033	0.0166
Full Model (w/ CFT)	0.425	0.640	0.4072	0.0159

Active acquisition improves both action accuracies and reduces trajectory errors. Adding CFT yields the best results across all four metrics.

Visual generation quality

Tab. 1

Visual quality measured by SSIM, LPIPS and VBench. Best scores among comparable variants are in bold; gray base-model VBench scores are reference values.

Visual generation quality. Tab. 1 in the paper.
Method	SSIM↑	LPIPS↓	VBench
Method	SSIM↑	LPIPS↓	Subject↑	Background↑	Motion↑	Aesthetic↑	Imaging↑	Overall↑
Ground truth	1.000	0.000	0.901	0.961	0.974	0.545	0.695	0.815
Base Model	0.458	0.506	0.895	0.965	0.977	0.549	0.713	0.820
+ Random	0.531	0.453	0.876	0.967	0.981	0.537	0.639	0.800
+ AOL	0.538	0.419	0.890	0.970	0.982	0.541	0.644	0.805
Full Model (w/ CFT)	0.531	0.411	0.890	0.968	0.979	0.543	0.681	0.812

Visual quality remains consistent across variants. AOL improves SSIM and LPIPS over random acquisition; CFT further improves LPIPS and the VBench Imaging and Overall scores.

AOL, CFT, and negative selection

Tab. 7

Random + CFT uses random acquisition; the random-negative variant pairs AOL with a randomly selected counterfactual branch. Best results are in bold.

AOL, CFT, and negative selection. Tab. 7 in the paper.
Method	Action Accuracy		Camera Control
Method	Combined↑	Fine-grained↑	RPE_rot↓	RPE_trans↓
Base Model	0.240	0.490	0.6906	0.0235
+ Random	0.356	0.569	0.6393	0.0199
+ Random + CFT	0.369	0.596	0.5023	0.0172
+ AOL	0.393	0.610	0.5033	0.0166
+ AOL + CFT (random negative)	0.410	0.622	0.4670	0.0163
Full Model (w/ CFT)	0.425	0.640	0.4072	0.0159

CFT improves action controllability even under random acquisition. The full model performs best, with visually similar negatives outperforming random negatives.

Additional experiments

Scoring-noise sensitivity

Tab. 8

Ranking agreement as the scoring-noise level varies from 0.3 to 0.7, relative to the default σ* = 0.5 shown in gray.

Stability of acquisition rankings. Tab. 8 in the paper.
σ*	Kendall’s τ_b↑(p-value)	Rank difference ≤ 1↑	Rank difference ≤ 2↑
0.3	0.8867 (≈ 10⁻⁵)	86.72%	99.22%
0.4	0.9386 (≈ 10⁻⁵)	92.97%	100.00%
0.5	1.0000 (N.A.)	100.00%	100.00%
0.6	0.9263 (≈ 10⁻⁵)	92.97%	99.22%
0.7	0.8320 (≈ 10⁻⁵)	75.78%	91.41%

Acquisition rankings remain stable around the default scoring-noise level. Most changes occur among branches with very similar scores.

Hardest-fork acquisition

Tab. 9

Random and highest-scoring-fork acquisition compared using six visual-quality and action-control metrics.

Random versus hardest-fork acquisition. Tab. 9 in the paper.
Method	SSIM↑	LPIPS↓	Combined↑	Fine-grained↑	RPE_rot↓	RPE_trans↓
Random	0.485	0.480	0.486	0.693	1.3642	0.2191
Hardest	0.457	0.561	0.404	0.604	2.0324	0.3530

Random acquisition outperforms hardest-fork selection across all six metrics.

Qualitative results

Action control in Minecraft

The Minecraft comparisons show ground-truth rollouts alongside OnlineWM and variants without CFT or AOL. They include leaf occlusion and interactions with forest, sandy and grassy terrain, where nearby geometry and environmental collisions challenge action-conditioned prediction. These examples examine how active acquisition and counterfactual supervision affect consistency between executed actions, camera motion and local scene structure.

Action control under leaf occlusion. The agent moves into tree leaves, creating strong occlusion and a shift in visual appearance. OnlineWM preserves the relation between the camera, trunk, and foliage more consistently, while removing CFT or AOL weakens the predicted response to the collision.

Collisions in forest terrain. These Minecraft rollouts compare the full model with versions without CFT or AOL near uneven forest terrain. The highlighted regions show differences in the predicted interaction with the surrounding blocks.

Approaching a sandbank. The models are compared as the camera approaches a stepped sand formation. The highlighted frames show differences in how the rollouts preserve the nearby surface and respond to the approach.

Interactions with grassy terrain. This example compares action-conditioned rollouts around uneven grassy terrain. The highlighted regions make differences in local geometry and the predicted interaction visible across OnlineWM and its ablations.

Swipe to compare

Generalization beyond Minecraft

We evaluate the Minecraft-trained model on indoor, outdoor and street-scene images without additional training or adaptation. OnlineWM and the WorldCompass RL post-training checkpoint receive the same action sequences. Generated frames, WorldMirror reconstructions and recovered camera trajectories illustrate transferred action control, while rotational and translational relative pose errors provide quantitative comparisons.

Looking up in an indoor scene. With the same movement and upward-view commands, OnlineWM produces a more pronounced upward camera change than the baseline. The accompanying reconstruction reports lower rotational and translational trajectory errors for OnlineWM.

Looking up among outdoor structures. The upward-view command tests action control in an outdoor scene with a different visual appearance from Minecraft. OnlineWM follows the upward change more closely, with lower trajectory errors in the accompanying reconstruction.

Indoor forward and diagonal motion. The sequence combines forward and diagonal movement in a furnished indoor scene. The generated views and recovered camera trajectories compare transferred action control, with lower reported pose errors for OnlineWM.

Turning left in an outdoor scene. Forward motion is interrupted by a leftward-view command among outdoor structures. The reconstruction and recovered trajectory provide a second view of the comparison, with lower reported pose errors for OnlineWM.

Turning right along a street. This street scene combines forward, rightward-view, and diagonal-motion commands. The rollouts compare the baseline and OnlineWM under the same action sequence, and the trajectory measurements favor OnlineWM.

Turning left along a street. A leftward-view command between forward-motion segments tests control in a street scene. OnlineWM has lower reported rotational and translational errors in the recovered trajectory comparison.