arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2506.23164

Mode Collapse Happens: Evaluating Critical Interactions in Joint Trajectory Prediction Models

Maarten Hugenholtz·Anna Meszaros·Jens Kober·Zlatan Ajanovic
View original source ↗Read source document ↗Export metadata ↓

Abstract

Autonomous Vehicle decisions rely on multimodal prediction models that account for multiple route options and the inherent uncertainty in human behavior. However, models can suffer from mode collapse, where only the most likely mode is predicted, posing significant safety risks. While existing methods employ various strategies to generate diverse predictions, they often overlook the diversity in interaction modes among agents. Additionally, traditional metrics for evaluating prediction models are dataset-dependent and do not evaluate inter-agent interactions quantitatively. To our knowledge, none of the existing metrics explicitly evaluates mode collapse. In this paper, we propose a novel evaluation framework that assesses mode collapse in joint trajectory predictions, focusing on safety-critical interactions. We introduce metrics for mode collapse, mode correctness, and coverage, emphasizing the sequential dimension of predictions. By testing four multi-agent trajectory prediction models, we demonstrate that mode collapse indeed happens. When looking at the sequential dimension, although prediction accuracy improves closer to interaction events, there are still cases where the models are unable to predict the correct interaction mode, even just before the interaction mode becomes inevitable. We hope that our framework can help researchers gain new insights and advance the development of more consistent and accurate prediction models, thus enhancing the safety of autonomous driving systems.

Explore related subjects

cs.ROcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Maarten Hugenholtz, Anna Meszaros, Jens Kober, Zlatan Ajanovic. 2025-06-29. Mode Collapse Happens: Evaluating Critical Interactions in Joint Trajectory Prediction Models. https://arxiv.org/abs/2506.23164

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Freeze, Share, Shrink: Rethinking the Action Backbone in Diffusion Policies

Many recent Vision-Language-Action models employ diffusion or flow-matching backbones with hundreds of millions of parameters for action generation. However, unlike image synthesis where the output spans millions of diverse pixels, a manipulation policy generates only short sequences of low-dimensional, physically correlated action values, a far simpler target that may not require such capacity. We confirm this intuition and show that, in modulation-conditioned diffusion policies, task adaptation can be routed entirely through the conditioning pathway, leaving a frozen, observation-free backbone that serves as a reusable trajectory prior. To establish this, we introduce a decoupled training recipe: a general-purpose action head is first pretrained on observation-free forward-kinematics data, then frozen while only the conditioning pathway is trained for downstream tasks. Using Diffusion Policy as a testbed, we show that on both MimicGen and LIBERO, a single frozen backbone shared across all tasks matches normally trained counterparts. In our ablations, this succeeds with modulation-based conditioning, while attention-based conditioning is embedded in the backbone's own weights and collapses once they are frozen. Ablations show the pretraining signal (joint positions, end-effector poses, or none) has little effect, while a randomly initialized backbone fails entirely, indicating that pretraining need only supply a general trajectory prior. Finally, a 5M-parameter MLP backbone matches or exceeds both the 244M U-Net and the transformer on these benchmarks, suggesting the action backbone is over-parameterized and that architectures inherited from image and language generation are a poor inductive-bias fit for the low-dimensional action target, with implications for action-backbone design in VLA models and other policies with an action backbone.

cs.RO↗
arXiv2026

On-Demand Human Assistance for Task Continuation under Physical Action Failures in LLM-based Planning

While robot action planning based on large language models (LLMs) has advanced remarkably, continuing a task after a physical action failure during execution remains a key challenge. For example, after failing to grasp an object, the robot may not notice that it has fallen off the table and keep trying to detect and pick it up, stalling the task. Existing approaches either rely on closed-loop autonomous re-planning, which fails when errors exceed the robot's capabilities, or on human intervention without updating the LLM's plan, hindering subsequent planning. We present a system for LLM-based robot action planning that integrates remote human intervention with feedback-based replanning. When the LLM detects an action failure, the robot requests remote assistance; an operator resolves the failure through teleoperation and reports the outcome in natural language, which is fed back to the LLM to update its plan so that the task can continue. We evaluated the proposed system on a real-world trash-collection task, comparing it with a fully autonomous method and a teleoperation-only baseline. The proposed system improves task progress over the autonomous method in this case study. For easy-to-retrieve trash, the gap from the teleoperation reference was small, while a larger gap remained for harder-to-retrieve trash. These results suggest that, in this case study, combining remote human intervention with feedback-based replanning was associated with continued task execution even in the face of otherwise unrecoverable failures. The project website is https://emergentsystemlabstudent.github.io/REPAIR/.

cs.RO↗
arXiv2026

Generating Realistic Safety-Critical Scenarios for Vehicle-Pedestrian Interactions

Automated driving system deployment requires rigorous validation across safety-critical vehicle-pedestrian interactions, yet real-world datasets rarely capture high-risk scenarios while simulation platforms lack realistic behavior. In response, this study proposes a three-stage framework that combines real-world grounding with adaptive simulation to generate behaviorally realistic safety-critical scenarios at scale. Stage 1 pre-trains multi-agent state-space Transformer-enhanced DDPG (MA-SST-DDPG) agents on real-world safety-critical data to learn human-like interactive evasive behaviors through data-driven learning. Stage 2 deploys pre-trained multi-agents in CARLA for online reinforcement learning to generalize across diverse scenarios, integrating real-world knowledge with simulation experience to produce a refined MA-SST-DDPG model. Stage 3 uses CARLA with the refined model to generate over 198,000 high-resolution interaction episodes from eight intersection scenarios, culminating in the Vehicle-Pedestrian Safety-Critical Interaction (VPSCI) dataset. The Refined MA-SST-DDPG model outperformed baseline methods in reproducing realistic evasive behaviors, achieving the lowest trajectory errors (ADE = 0.072 m, FDE = 0.142 m). Statistical comparison confirmed distributional equivalence between the generated and real-world data in both conflict severity and behavioral response. A Turing test confirmed that the three-stage framework generated evasive behaviors were indistinguishable from real-world interactions. These results demonstrate the framework's effectiveness in producing high-fidelity safety-critical data, offering valuable sources for the development of ADS and simulation-based safety evaluations.

cs.RO↗