Vision-Language Models for Embodied Navigation
The rapid progress of Large Language Models (LLMs) and Vision-Language Models (VLMs) has significantly advanced embodied artificial intelligence by enabling robots to reason about complex environments using natural language instructions. Rather than relying solely on geometric representations, these models provide semantic understanding, open-vocabulary perception, and contextual reasoning, allowing agents to execute tasks beyond traditional navigation objectives.
Recent foundation models have demonstrated increasingly general embodied capabilities. HabitatGS extends AI Habitat with photorealistic Gaussian Splatting scenes populated by dynamic humans, providing a more realistic benchmark for embodied reasoning. Embodied foundation models such as EmbodiedFM are trained on millions of navigation demonstrations spanning multiple robotic domains—including household robots, drones, and autonomous vehicles—and demonstrate strong zero-shot generalization across diverse navigation tasks.
Several works have also investigated reasoning-driven navigation. OctoNav introduces a think-before-action paradigm in which navigation policies are trained from instruction-trajectory pairs using reinforcement learning. NavR1 further incorporates Group Relative Policy Optimization together with Chain-of-Thought supervision to jointly learn dialogue, planning, reasoning, and navigation. These methods demonstrate that explicit reasoning improves embodied decision making, although they generally require extensive training datasets and computationally expensive foundation models.
Unlike these approaches, OptiSight does not require end-to-end policy learning or large-scale navigation datasets. Instead, it leverages pretrained VLMs only for semantic perception and high-level reasoning, while delegating continuous motion generation to deterministic geometric control.
Semantic and Object-Goal Navigation
Object-goal navigation has emerged as a fundamental benchmark for embodied AI, requiring agents to locate objects specified by semantic categories rather than predefined coordinates. OVON significantly expanded this setting by introducing over 15,000 object instances and demonstrating open-vocabulary object navigation using semantic representations.
Subsequent work has focused on improving navigation efficiency through memory-augmented reasoning. EfficientNav combines zero-shot LLM planning with semantic memory retrieval and memory clustering to reduce repeated reasoning and inference latency. Hierarchical reasoning frameworks such as HiRobot decompose complex user instructions into hierarchical subgoals while incorporating user feedback to refine navigation plans.
Although these approaches improve semantic navigation, they typically rely on repeated LLM or VLM inference throughout execution, increasing computational overhead. In contrast, OptiSight invokes semantic reasoning only at critical decision points while maintaining continuous navigation through lightweight visual servoing, substantially reducing latency.
Semantic Grounding and Chain-of-Thought Reasoning
Beyond navigation, similar reasoning paradigms have also been explored for robotic manipulation. ALRM combines ReAct-style reasoning with executable code generation and tool-based planning to perform long-horizon manipulation tasks.
While these methods demonstrate the benefits of explicit reasoning, many continuously query large VLMs throughout task execution, resulting in substantial computational cost and inference latency. In contrast, OptiSight adopts a state-driven Chain-of-Thought architecture in which semantic reasoning is invoked only at key state transitions within a finite-state machine, while continuous low-level motion is handled by deterministic visual servoing. This separation between high-level reasoning and geometric control enables explainable navigation while maintaining real-time performance on resource-constrained robotic platforms.
Methodology
OptiSight is a closed-loop autonomous navigation framework that integrates Vision-Language Model reasoning, open-vocabulary object grounding, geometric projection, and deterministic motion control. In its current implementation, the system is designed to execute the high-level navigation instruction “Get out of the room.” To accomplish this task, the framework decomposes the instruction into a sequence of semantic reasoning and geometric navigation stages.
During the Navigate state, the camera maintains its downward orientation to continuously monitor the planned path while proportional steering commands drive the robot along a sequence of 3D waypoints. Waypoints are generated from the planned trajectory and are sequentially discarded once the robot approaches within the specified distance threshold of the current target waypoint, ensuring smooth trajectory tracking and stable motion execution. If the semantic target is lost or the planned path becomes invalid, the FSM transitions to the Recover state, where the VLM is invoked again to re-establish the navigation objective before resuming the execution pipeline.
The following subsections describe the operations performed in each state of the finite-state machine and the actual implementation of the proposed method.
Finite-State Machine
Find
Once the target has been detected, the framework transitions to the Find state, where Grounded-SAM performs open-vocabulary localization to obtain an accurate segmentation of the target. To improve localization stability, the segmentation is refined through up to three consecutive inference passes, producing a consistent low-noise bounding region. The resulting image coordinates are then projected into the 3D camera reference frame using the geometric projection model, yielding the spatial location of the navigation target. Upon successful projection, the system proceeds to the Scan state.
Dual-Isolated System Architecture
OptiSight adopts a dual-isolated execution architecture that separates semantic reasoning from geometric navigation into two independent runtime environments. The two environments communicate through a lightweight socket-based interface, allowing each component to operate within its own software stack while exchanging only the information required for navigation.
The first environment executes Habitat-Sim together with the geometric navigation pipeline, including the finite-state machine, camera control, geometric projection, path planning, waypoint generation, and motion control. The second environment hosts the Vision-Language Model and the Grounded-SAM inference pipeline responsible for semantic reasoning and open-vocabulary object localization.
Communication between both environments is asynchronous. Whenever semantic reasoning is required, the navigation module sends the current observation and task prompt to the VLM server, which performs inference independently and returns the resulting semantic prediction. Since the VLM is queried only at the semantic decision points defined by the finite-state machine, geometric navigation continues uninterrupted while reasoning is executed in the background. Once the response is received, the finite-state machine resumes execution using the inferred semantic information.
This architecture isolates the dependencies of the robotics and VLM frameworks, enabling the use of modern multimodal models together with legacy robotics software without compatibility issues. In addition, separating semantic inference from geometric control improves resource utilization and allows the complete system to operate within an 8 GB VRAM budget suitable for deployment on resource-constrained edge platforms such as NVIDIA Jetson.
Experiments
The selected scenarios are designed to evaluate the robustness of OptiSight under six representative navigation challenges:
- Single-obstacle avoidance.
- Dual-obstacle reasoning.
- Semantic disambiguation between structurally similar objects.
- Partial observability requiring active visual search.
- Extreme viewpoints producing geometric distortions.
- Perceptual ambiguities caused by reflective surfaces.
The first three scenario types primarily evaluate navigation and semantic grounding, whereas the latter three focus on perception and geometric ambiguities that can challenge VLM-based reasoning. This set of scenarios therefore provides a diverse evaluation of both the semantic and geometric components of the proposed framework.
We evaluate OptiSight using the following metrics:
- Mission Success: Binary indicator of whether the navigation objective is successfully completed. A scenario is considered successful if the mission is completed in at least half of the independent runs.
- Success Rate: Percentage of successful runs across the trials.
- Execution Time: Time required to complete the navigation task, measured in seconds.
- Total Distance: Cumulative distance traveled by the agent, measured in meters.
- Total Steps: Total number of recorded state transitions, actions, and movement steps during execution.
- Collisions: Number of physical collisions with environmental obstacles.
- Recoveries: Number of recovery or obstacle-avoidance procedures triggered during navigation.
- VLM Requests: Number of VLM inference calls performed during a navigation episode.
- Minimum Obstacle Distance: Minimum Euclidean distance, in meters, between the agent and any detected 3D obstacle during navigation.
Together, these metrics characterize both task-level performance and the internal behavior of the system. Success rate, execution time, and traveled distance measure navigation effectiveness, while collisions and minimum obstacle clearance quantify safety. VLM requests, parsing failures, and recovery actions provide additional measures of computational and operational robustness.
Conclusions
OptiSight is a hybrid autonomous navigation framework that combines selective Vision-Language Model reasoning with deterministic geometric navigation. By integrating a finite-state machine, Grounded-SAM-based target grounding, and visual-to-3D geometric projection, OptiSight translates high-level semantic instructions into executable navigation actions while avoiding continuous VLM inference and computationally expensive mapping pipelines.
Evaluation in AI Habitat across 24 indoor navigation scenarios demonstrates that OptiSight can successfully handle diverse navigation conditions, including obstacle avoidance, semantic ambiguity, partial observability, and challenging viewpoints, while requiring only a limited number of VLM queries. These results highlight the potential of combining semantic reasoning with lightweight geometric control for efficient embodied navigation.
Future work will focus on extending the framework to a broader range of navigation tasks and validating OptiSight on physical robotic platforms.
Frequently Asked Questions
What is OptiSight?
OptiSight is a closed-loop autonomous navigation framework that combines VLM-based semantic reasoning with deterministic geometric control.
How does OptiSight reduce VLM computational overhead?
It invokes semantic reasoning only at critical decision points while continuous navigation is handled by lightweight visual servoing.
What does the dual-isolated architecture separate?
It separates the Habitat-Sim geometric navigation pipeline from the VLM and Grounded-SAM semantic inference pipeline.
What navigation conditions were evaluated?
The evaluation included obstacle avoidance, semantic ambiguity, partial observability, extreme viewpoints, and reflective-surface ambiguities.