M2A Anonymous Submission

1. Abstract

TL; DR.

Vision-Language Navigation models increasingly use end-to-end MLLM training on long-horizon instruction datasets. This paper shows that such models can still fail at fundamental commands like move to the desk, even when they perform well on conventional VLN benchmarks.

We propose a primitive-centered training paradigm. The work introduces VLMB, a controllable dataset and benchmark for the move-to primitive, and M2A, a navigation model with dual-scale memory and explicit spatiotemporal grounding.

Overview of motivation and M2A step-wise planning

2. Motivation

Primitive-Level Failures Hidden by End-to-End Evaluation

Unexpected Failure

SOTA VLN models trained for long-horizon navigation can fail on simple primitive tasks. The videos below show failures on R-Nav-style target navigation, where the instruction is easier but action grounding must be precise.

R-Nav instruction: the SOTA model fails to stop at the target.

R-Nav instruction: another primitive episode where the agent misses the target.

Blind Success

Reordered R2R instructions should break faithful instruction following. However, models can still reach high success by following dataset or visual biases. The switcher compares original R2R and reordered R2R* examples.

The agent does not stop at the correct location.

The agent does not execute navigation actions according to the instruction.

R2R* example with reordered instruction, exposing blind success.

R2R* example where success does not imply faithful instruction execution.

3. Dataset Construction

Vision-Language-Move-Base

VLMB data construction pipeline — VLMB data generation pipeline.

01

Goal-Oriented Object Sampling

Sample visible, reachable objects from Habitat scenes while filtering ambiguous or non-informative targets.

02

Instruction Enrichment

Use MLLM verification and spatial-semantic augmentation to convert raw move-to commands into diverse grounded instructions.

03

Multi-Step Composition

Chain verified primitive segments into longer instructions for controlled compositional evaluation.

4. Method

Move-to-Anything

M2A method framework — M2A framework with dual-scale memory and explicit spatiotemporal grounding.

Dual-Scale Memory

A short-term perceptual buffer keeps recent observations high fidelity, while a long-term object-aware history stores salient keyframes.

Object-Aware Keyframes

The episodic memory selects frames by semantic novelty, preserving landmarks and scene transitions without storing every observation.

Explicit Spatiotemporal Grounding

Temporal and segment embeddings tell the model whether each token comes from current perception or historical memory.

5. Main Experiments

M2A Learns Transferable Navigation Primitives

M2A improves primitive-level navigation on R-Nav and VLMB-Bench, then transfers to compositional multi-step navigation and LH-VLN without long-horizon supervision from VLMB.