arXiv Science⌕ Search

arXiv subjects

Keyao Xu

Publications and source records attributed to Keyao Xu.

2 recordsLinked to original sources

ThinkWAM: Rethinking Latent Futures for Action Refinement

World-action models (WAMs) couple action generation with predictions of future states. An initial action chunk may be flawed. A future prediction that is not conditioned on the chunk is identical for every proposal and thus carries no proposal-specific signal, limiting its usefulness for refinement before execution. We introduce ThinkWAM, a latent WAM that establishes bidirectional feedback before execution through a proposal-rethink-refine loop: the policy first proposes a complete action chunk, the proposal corrects multi-horizon future latents through a learned residual, and the corrected latents are injected into a second pass of the same action head, which generates the executable chunk. An affordance-guided observation-action memory and a near-fine, far-coarse multi-horizon prediction scheme, both motivated by predictive coding, support this loop, and all prediction stays in latent space. ThinkWAM achieves 98.70% success on LIBERO, 84.76% on LIBERO-Plus without OOD fine-tuning, 85.75%/65.85% on RoboTwin 2.0 Clean/Random, and 65.17% on RoboCasa24. ThinkWAM achieves 64.4% mean success across three real-world OOD conditions. The mean two-pass inference latency per action chunk is 153.48 ms on LIBERO, 164.61 ms on RoboTwin, and 101.2 ms on the Galaxea A1Z. Ablations show complementary gains from rethinking and affordance guidance. Diagnostics indicate that proposal-conditioned corrections change both predicted futures and action trajectories, showing that the corrected future representations improve the final action.

cs.RO↗

HaM-World: Soft-Hamiltonian World Models with Selective Memory for Planning

World models support model-based planning through learned latent dynamics, but imagined rollouts can become unstable as the planning horizon grows or the dynamics distribution shifts. We propose HaM-World, a structured world model that combines history-conditioned selective memory with a Soft-Hamiltonian latent dynamics prior. The latent state is decomposed into a canonical (q,p) subspace and a context subspace c. Mamba selective state-space memory summarizes past observations and actions and conditions the same latent transition used for prediction, reward and value estimation, imagined rollouts, and cross-entropy method planning. The (q,p) subspace follows an energy-derived Hamiltonian vector field augmented with learnable residual and control dynamics, while c represents semantic, dissipative, and other non-conservative factors. On six DeepMind Control Suite tasks, HaM-World ranks first on four tasks and second on two, achieving the highest average AUC on the four-task core suite (117.9, 9.5% above TD-MPC2). Within the short-to-medium horizons used by the planner, it reduces imagined-rollout error to 45% of a strong baseline and wins 11 of 12 rollout-MSE cells for horizons k in {3,5,7}. At longer open-loop horizons, its error grows faster and is overtaken between k=7 and k=20; we therefore do not claim uniform long-horizon stability. Under 12 out-of-distribution perturbations involving dynamics shifts, action delay, and observation masking, it achieves the highest absolute return in every condition, with average gains of 10.2% on Finger Spin and 13.6% on Reacher Easy. Ablations show that memory accounts for the larger share of the observed gains, while Soft-Hamiltonian geometry provides smaller but consistent complementary improvements.

cs.AI↗