AdvNav black-box attack framework disrupts vision-language navigation agents, arXiv study finds

A new arXiv preprint introduces AdvNav, a behavior-guided black-box adversarial attack framework that disturbs an agent's first-person views during vision-and-language navigation. The authors write that despite progress in embodied AI, vision-and-language navigation systems remain vulnerable to adversarial visual disturbances. Most existing methods rely on white-box access to target model gradients, which the paper says is often unrealistic for real-world deployed systems and computationally exhaustive because of recursive backpropagation. Prior black-box methods, the abstract says, predominantly target single-step decision tasks and struggle with multistep temporal dependencies. AdvNav builds a dual-granularity behavior-based feedback signal from the agent's self-output behaviors, aggregating a trajectory-level performance score for overall navigation degradation, an action-level reward score for potential decision risk, and a deviation indicator. That feedback guides a hybrid optimization strategy that heuristically tunes perturbation strength via adaptive updates and evolves noise spatial structure genetically to discover disruptive noise configurations. Evaluated against Transformer-based HAMT and LLM-based MapGPT with two types of backbones on the R2R dataset, AdvNav achieves 49.70/65.96/87.30% attack success rate, according to the abstract. The authors say the results demonstrate effectiveness and generality, reveal perception vulnerabilities, and offer insights for designing more resilient vision-language navigation models.