From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation

Xiangyun Huang1, Xiangchen Wang2, Runfeng Lin1,3, Yihao Xu1,
Kangyu Huang4, Jiang Hengchen1,5, Xiwang Dong1, Lin Jiarong1,*
1Beihang University 2Southern University of Science and Technology 3Central South University
4Harbin Institute of Technology, Shenzhen 5Dalian University of Technology
23231088@buaa.edu.cn, zivlin@buaa.edu.cn
arXiv Video Code Models Data

TL;DR

Route2Step decouples route-level semantic progress tracking from local action execution in vision-and-language navigation. MIA identifies the active sub-instruction and execution state, while MAG generates local action chunks conditioned on that state.

Approach

Route2Step framework overview

Instruction Analysis Module (MIA)

MIA receives the full route instruction and visual history. It predicts which sub-instruction is currently active and whether the agent is in a normal or recovering execution state.

Action Generation Module (MAG)

MAG conditions on the explicit progress state and recent observations to generate local action chunks. This separates the question of which step to follow from how to execute that step.

E-SPA and level-separated correction

E-SPA aligns fine-grained sub-instructions with contiguous trajectory segments without additional manual temporal labels. State-level corrective supervision is used for semantic progress, while direct action supervision is reserved for repeated local execution failures.

Simulation

R2R-CE

Instruction: With the large window to your right, head forward and to your left until you reach the base of some stairs leading upwards. Continue past the stairs and towards some more stairs leading downwards. Pass to the right of those second stairs and down a narrow hallway with the banister on your left and some artwork on the wall towards your right. Enter the next room on your right when you reach it.
Instruction: Go straight pas the couches and the fireplace on the right. Go through the double doors. Turn right and pass the couches. Go into the room on the right. Wait near the round table and chairs with the white flower in the table.
Instruction: Walk towards the black couch, and walk between the couch and the counter. Go straight down the hallway and passed the sitting area on the left. Continue straight. Stop by the small table and large mirror hanging on the wall.

RxR-CE

Instruction: We're in a living room, and we're facing towards a corner of a wall, on the right there's a wide painting and on the left there's a console table with some decorations on top, turn around... You're now facing towards four glass doors, on the left there's a small fence that's attached to the wall, walk towards that, turn slightly to the left, and walk past the four glass doors on the right, and the double doors on the left, continue walking down the hallway, pass the console table on the right, sorry, pass the console table that's on the left, and the painting that's on the right... We're now facing towards another console table and on top of it there are decorations, turn towards the right, on your right there are four glass doors and on the left there's a piano and many pictures on the wall... Walk towards the dresser that's in front of you and once you reach the dresser, you'll be facing towards a window which is covered with some beautiful curtains, turn towards the left and enter the archway that's in front of you... Once you enter the arch way, you'll come across a bathroom and in front of you there's going to be a tub, walk towards that tub, pass the arch way that's in front of you, once you get into the bathroom and you're facing towards the tub, walk towards the cabinet that's in the middle and it's on your left, once you're facing towards a sink and a mirror, turn slightly to the right and walk past the small hallway that's in front of you... Once you pass that hallway, walk towards the wall that's in front of you and you've reached your destination.
Instruction: You are in a living area, turn towards your right and keep moving forward. In front, you find a section of a room, go pass through the section and you enter into a corridor. Keep moving forward near to the wall in front of you and turn towards your left. You find an other section of a corridor, keep moving forward in the corridor, till you reach a staircase on your left. Turn towards the stair case and climb. Climb up the stairs, all the way to the top floor. Turn towards your slight right and you find an other section of a room. Go pass through the section by turning towards your slight right and you enter into a living area. Turn towards your left and keep moving forward. In front, you find a kitchen area, go near that area and stand in front of a coffee maker. Now, turn towards your slight left, you find an other section of a room. Go pass through the section and turn towards your slight right again. In front, you find a glass dining table. Walk across the glass dining table and stand near the window. You reached your destination.
Instruction: You are standing in a circular room facing an open archway. You are going to walk through that open arch way and proceed down the hallway. You are going to go all the way down the hallway until you end up in front of a kitchen island. Once you get to that kitchen island you are going to turn left. And proceed down the next hallway of arches. You are going to go all the way down that hallway until the hallway ends and there is a white flower pot with some light pink flowers in it. You are going to go through the archway to the left. From there you are going to walk over to the bottom of the stairs. And then once you're at the bottom of the stairs you're going to go through the archway immediately to the left of the stairs. In this circular room, there's a lot of circles, you are going to see 2 archways and a large rectangular mirror. You are going to go through the archway on the left. It has what looks like a rough iron light fixture and its next to a partially stained glass window. Go through that archway. Then you are going to continue straight and step onto that first rug. When you are standing on that rug you will see a bunch of men on horses in a picture to the left. You are going to enter the doorway to the left right after that picture. That is a bathroom. You are going to step into that bathroom and then you have reached the end.

Citation

@misc{huang2026route2step,
  title         = {From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation},
  author        = {Xiangyun Huang and Xiangchen Wang and Runfeng Lin and Yihao Xu and Kangyu Huang and Jiang Hengchen and Xiwang Dong and Lin Jiarong},
  year          = {2026},
  eprint        = {2608.03143},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.03143}
}