We introduce HM3D-MFMON, comprising 927 three-goal episodes from 36 multi-floor HM3D scenes, including 288 Cross-Floor-Required episodes, together with a post-hoc evaluation protocol for multi-instance sequential navigation.
LifelongCrossNav unifies support-aware 3D voxel mapping, persistent vision-language memory, stair-aware navigation, and historical semantic retrieval in a closed-loop system.
Experiments demonstrate improved multi-object and cross-floor navigation over a persistent planar semantic-memory baseline, while History POIs improve path efficiency for later object goals.
Task Formulation
Within an episode, LifelongCrossNav retains the 3D geometric map, stair states, and goal-independent vision-language voxel features across object goals. When a new object goal is issued, its text embedding and query-conditioned similarity field are recomputed, while the POIs, navigation path, and goal-verification states associated with the previous object goal are reinitialized.
Support-Aware 3D Voxel Mapping
LifelongCrossNav represents the environment using a sparse 3D voxel map. In contrast to planar projection, this representation preserves height and distinguishes vertically overlapping rooms, corridors, landings, and staircases. Inspired by traversability-aware ray-casting approaches, the map combines RGB-D geometry, vertical support relationships, and stair-semantic evidence to represent executable connections across floors.
RGB-D Projection and Sparse Voxelization
Valid depth pixels are back-projected into the camera coordinate system, transformed into the world frame using the 6-DoF camera pose, and quantized into sparse voxels. Depth-ray endpoints provide observed surface evidence, whereas intermediate ray locations provide free-space evidence. Only observed voxels and locally inferred navigation states are stored, allowing the map to grow incrementally with the explored space.
Support-Aware Voxel Types
The navigation map abstracts the observed space into four functional voxel types: Occupied, Traversable, Stair, and Unsupported.
Occupied voxels represent observed surfaces, including walls, furniture, floors, and stair candidates that have not yet been confirmed. For each ray-observed free-space voxel, the system searches downward within a local vertical range. Free space with reliable support is classified as Traversable, whereas free space without observed support is classified as Unsupported and provides geometric evidence for potential downward transitions.
Stair voxels represent stair surfaces confirmed jointly by semantic and geometric observations. SegFormer-B2 extracts a pixel-level stair mask, which is lifted into 3D using depth and camera pose and verified through local height variation, step trends, spatial continuity, and support relationships. Stair voxels are updated only after the system enters Stair Explore, where they form an explicit traversable connection between floors.
Unified Navigation Policy
The unified policy selects navigation targets from Basic Frontiers, Stair Frontiers, History POIs, and Live POIs. These candidates support three navigation modes: Basic Explore, Stair Explore, and POI Navigation.
Candidate Navigation Points
Basic Frontiers denote the three planar candidate types used during ordinary exploration:
- Traversable Frontiers are formed at boundaries between Traversable voxels and unknown space.
- Descend Frontiers are extracted near boundaries between Traversable and Unsupported voxels and indicate potential downward transitions.
- Ascent Frontiers are generated from stair-semantic evidence, RGB-D step geometry, and multi-frame consistency, with their navigation positions remaining on the current exploration plane near a potential ascending entrance.
Stair Frontiers are direction-aware advancing boundaries extracted from the currently confirmed stair-voxel cluster. Their heights change with the observed staircase, allowing the agent to continue exploring upward or downward within the stair structure.
Hierarchical Mode Switching and 3D Planning
Reliable current target evidence and reachable POIs are prioritized over unexplored geometric candidates. When no valid semantic candidate is available, Basic Explore first visits Traversable Frontiers on the current floor. Descend and Ascent Frontiers are considered only after no reachable ordinary frontier remains.
Once a stair entrance has been verified, the policy activates Stair Explore and follows Stair Frontiers until a new landing is reached. An active stair session retains control so that stair traversal is not interrupted by ordinary frontiers or unrelated semantic candidates.
All modes use a common 3D A* planner but differ in admissible voxel types and graph connectivity. Basic Explore uses a strict 26-neighbor graph over the current traversable region. POI Navigation uses the regular graph together with previously established stair connectivity when the selected POI lies on another floor. Stair Explore admits confirmed Stair voxels and expands the endpoint neighborhood within a fixed physical radius to connect staircase observations that may be sparse in the voxel map.
Final Target-Object Approach and Verification
Once stable target-object evidence is obtained, the detected mask is projected into 3D and a nearby safe observation position is selected. During the approach, the system updates the detection and verifies the target object using detection confidence, mask quality, visibility, viewing direction, and 3D distance.
Reaching either a History POI or a Live POI does not by itself complete the subtask; a current target-object observation must still pass the verification criteria. After successful verification, the next object goal is activated while the geometric map, stair structure, and goal-independent semantic memory are retained. If verification fails, the system returns to candidate selection and continues exploration.
HM3D-MFMON Benchmark
We construct HM3D-MFMON from HM3D v0.2 and its semantic annotations. HM3D provides semantically annotated indoor scenes with realistic multi-floor layouts and navigable stair connections, while remaining compatible with established Habitat ObjectNav protocols.
Following the sequential task formulation of MultiON, each episode contains three sequentially issued object goals. The agent receives only the current object goal and is informed of the next one after successfully completing the current object-goal subtask.
HM3D-MFMON includes six target object categories:
- Chair
- Bed
- Toilet
- Plant
- Sofa
- TV monitor
All valid instances of the current target object category are retained, and reaching any instance that satisfies the success criterion completes the corresponding subtask.
We select 36 HM3D scenes with valid multi-floor structures and stair connectivity and generate 927 three-goal episodes. By verifying target-instance distributions and navigable connectivity, we identify a Cross-Floor-Required (CFR) subset of 288 episodes whose complete object-goal sequences cannot be completed without at least one floor transition.
Baselines and Evaluation Settings
Multi-Object Navigation
For sequential multi-object navigation, OneMap serves as the primary baseline because it maintains a persistent open-vocabulary 2D semantic map that can be reused across successive object goals. OneMap and LifelongCrossNav are evaluated under the same task and evaluation settings described above.
To isolate the effect of historical semantic retrieval, we additionally evaluate a variant denoted as w/o H-POI. This variant disables only History POI generation, while retaining the support-aware 3D voxel map, accumulated vision-language features, and all cross-floor navigation components.
Conclusion
LifelongCrossNav is introduced for sequential multi-object ObjectNav in unknown multi-floor environments. LifelongCrossNav combines support-aware 3D voxel mapping, persistent vision-language semantic memory across object goals, and a unified policy for planar exploration and stair navigation, enabling the agent to jointly exploit vertical structure, cross-floor connectivity, and historical semantic observations.
The HM3D-MFMON benchmark and a post-hoc stage-wise evaluation protocol based on the actual starting state of each object-goal subtask are also introduced. Experimental results show that LifelongCrossNav consistently outperforms the planar persistent semantic mapping baseline on both the full multi-object benchmark and the Cross-Floor-Required subset. Ablation results show that History POIs reduce repeated exploration and improve later-goal efficiency. Future work will study real-world deployment and more robust cross-floor navigation.
Technical Supplement
This supplement provides the configuration, voxel-state definitions, navigation-mode details, type-aware planning rules, and additional stage-wise evaluation results omitted from the main paper.
Support Classification
Accordingly, TRAVERSABLE represents supported free space above an ordinary or reversible surface, whereas TRAVERSABLE_STAIR denotes free space supported by confirmed stair geometry.
In contrast, TRAVERSABLE_FAKE indicates observed free space for which no valid support is found within the support-search depth. It should therefore not be confused with FAKE_STAIR, which is a reversible surface state rather than an unsupported-air state.
Stair-State Promotion
During Basic Explore, SegFormer masks and RGB-D geometry may generate ascent or descent evidence, but ordinary OCCUPIED voxels are not immediately rewritten as confirmed stairs. After the corresponding stair entrance is reached and verified, the controller enters Stair Explore, implemented as the StairSem mode.
Within StairSem, accepted surface voxels are promoted to STAIR; adjacent occupied cells may become STAIR_EXPANDED; rejected evidence is retained as FAKE_STAIR. Support-dependent air voxels are then reclassified locally.
This delayed promotion prevents uncertain stair observations collected during ordinary exploration from directly changing the cross-floor planning graph.
Candidate Priority
The controller considers four main candidate classes:
- Basic Frontiers
- Stair Frontiers
- History POIs
- Live POIs
Their default selection priority is determined by the controller’s candidate-selection policy.
Here, Basic Frontiers collectively refer to Traversable, Descend, and Ascent Frontiers used during ordinary exploration:
- Traversable Frontier: the boundary between supported TRAVERSABLE voxels and genuine unknown space.
- Descend Frontier: the boundary between supported traversable space and unsupported or fake-stair geometry.
- Ascent Frontier: a semantic-geometric entrance proposal for an upward stair.
The controller first explores reachable Traversable Frontiers on the current floor. Only after no reachable ordinary frontier remains does it consider Descend and Ascent Frontiers. Descend is attempted before ascent by default, although the order may be adapted according to the most recent successful floor-transition direction.
A selected candidate is committed for a short grace period so that incremental map updates do not cause frequent target switching. Commitment is therefore a stability mechanism rather than an additional candidate class. A committed candidate is replaced only when it becomes invalid, unreachable, or is superseded by a higher-priority POI.
Normal Graph
Basic Explore and POI Navigation use a 26-neighbor graph. The heuristic is the 3D Euclidean distance from the current voxel to the goal.
The heuristic estimates the straight-line metric distance between the current voxel and the goal.
The cost is determined by the destination voxel state.
Destination-state costs penalize less reliable or inflated voxel states while preserving access to confirmed stair geometry.
The penalties favor stable supported air while preserving access to confirmed stair geometry. FAKE_STAIR remains available only as a costly fallback and is never selected as a navigation endpoint.
Ordinary planning prioritizes non-inflated voxels. When a mode explicitly permits an inflated pathable endpoint, its transition cost is multiplied by two. If the agent is already inside an inflated pathable region, the planner may temporarily retain inflated states to allow the agent to leave that region; strict planning resumes afterward.
StairSem Graph
Sparse RGB-D observations may leave gaps larger than one voxel between successive stair surfaces. StairSem therefore replaces the fixed 26-neighbor relation with all existing pathable endpoints inside a 1.0 m physical sphere. This neighborhood is defined in metric 3D space rather than by a fixed voxel offset.
Inflated TRAVERSABLE and TRAVERSABLE_STAIR endpoints may be used when necessary, whereas inflated STAIR, STAIR_EXPANDED, and FAKE_STAIR endpoints are rejected.
If the final Stair Frontier is temporarily unreachable, intermediate session-local stair targets are tried in descending order of vertical progress.
The expanded graph is deliberately permissive for incomplete stair observations. It verifies the stored endpoints of each edge but does not claim continuous swept-volume collision certification along every intermediate sample of a long edge.
Benchmark Summary
HM3D-MFMON contains 927 three-goal episodes from 36 multi-floor HM3D scenes. The finalized Cross-Floor-Required (CFR) subset contains 288 episodes whose complete object-goal sequences require at least one floor transition.
The first mandatory transition occurs at the first, second, or third object-goal subtask in 102, 101, and 85 episodes, respectively.
Post-Hoc Stage-Wise Shortest Paths
For stage i of episode e, the shortest-path denominator is recomputed from the actual beginning of that trajectory segment.
The denominator is the shortest navigable path from the segment’s actual starting state to any valid viewpoint for an instance of the requested object category.
The valid viewpoints include the navigable viewpoints of all instances belonging to the requested object category.
Let Sᵢ denote stage success and let Lᵢ be the executed trajectory length.
Stage-wise SPL combines stage success with the ratio between the shortest valid path and the executed trajectory length.
This post-hoc protocol does not require the agent to know future object goals and avoids fixing later-stage shortest paths before the actual completion location of the preceding goal is known.
History-POI Ablation Boundary
The w/o H-POI variant disables only History-POI generation and selection. It retains the persistent SED feature map, cumulative feature fusion, Live POIs, support-aware voxel mapping, stair perception, and cross-floor planning.
The ablation therefore measures the contribution of explicit historical semantic retrieval rather than removing the complete semantic-memory representation.
Global Results on Cross-Floor-Required Episodes
The CFR results expose the effect of unresolved floor transitions. If OneMap reaches the third object goal, its first two goals have already been completed without requiring a floor transition.
Frequently Asked Questions
What is LifelongCrossNav designed to do? It performs sequential multi-object navigation in unknown multi-floor environments while retaining geometric, stair, and semantic information across object goals.
What is included in the HM3D-MFMON benchmark? The benchmark contains 927 three-goal episodes from 36 multi-floor HM3D scenes, including 288 episodes requiring at least one floor transition.
How does LifelongCrossNav represent multi-floor environments? It uses a sparse support-aware 3D voxel map that distinguishes occupied, traversable, stair, and unsupported regions.
What does the w/o H-POI ablation test? It disables History-POI generation and selection while retaining persistent semantic features, live POIs, stair perception, and cross-floor planning.