The contributions of this work are threefold:
- We introduce a pre-planning evidence-acquisition mechanism that prompts a VLM to plan information-gathering subgoals, revealing hidden regions and verifying goal-relevant objects.
- We propose a feasibility-gating mechanism that evaluates whether acquired evidence supports feasible task planning, requesting additional observations or halting when goal requirements remain unsupported.
- We design three cooking scenarios spanning explicit, underspecified, and missing-object goals, and empirically show that EAFG improves grounded completion and infeasibility detection.
Related Work
Task and Motion Planning combines discrete symbolic reasoning with continuous motion planning to solve long-horizon manipulation tasks under geometric, kinematic, collision, grasping, and placement constraints. PDDLStream extends PDDL with streams that interface with black-box procedures such as inverse-kinematics solvers, grasp and placement samplers, collision checkers, and motion planners. It coordinates finite PDDL search with stream evaluations to generate and verify the continuous parameters required by candidate manipulation plans.
Prior work on planning under partial observability and object search handles hidden objects and uncertain states with POMDPs, belief-space planning, and information-gathering actions, including revealing occlusions or inspecting containers in manipulation. LLM-based pre-execution plan verification detects logical or procedural flaws in proposed robot plans, but does not actively gather evidence about hidden or absent objects. In contrast, our method enables the robot to act autonomously to gather evidence and, based on that evidence, decide whether to plan, acquire additional evidence, or halt.
Overall, prior work provides geometric feasibility through TAMP, semantic task decomposition through foundation models, or information gathering through partial-observability planning. Our contribution is to place evidence acquisition before task planning and to use acquired observations as an explicit feasibility gate for grounded planning or halting.
Evidence Acquisition and Feasibility Gating
We propose Evidence Acquisition and Feasibility Gating (EAFG), which uses a VLM to actively acquire observational evidence for verifying goal-relevant objects and environmental conditions before task planning. EAFG generates evidence-acquisition subgoals that resolve missing or uncertain information, and uses the acquired evidence to determine whether planning for the goal should proceed. If the required evidence cannot be obtained after exploring the goal-relevant regions, EAFG halts execution to avoid generating a plan based on unsupported assumptions.
Overview
EAFG takes a natural-language goal as input and generates a plan for acquiring the environmental information needed to determine whether the goal can be achieved. The input is a natural-language task goal.
The VLM first proposes a sequence of evidence-acquisition subgoals, where each subgoal is an exploratory action such as opening doors or cabinets and temporarily lifting occluding objects. These subgoals are used to inspect parts of the environment that may contain goal-relevant objects, rather than to directly perform the target task.
Each generated evidence-acquisition subgoal is passed to a TAMP, which attempts to instantiate it as a geometrically and kinematically feasible action sequence. The robot then executes the resulting action sequences. After each subgoal execution, the resulting observation image is stored as visual evidence for subsequent VLM inference.
After each evidence-acquisition iteration, the VLM receives the execution history and observation collage, updates the evidence state, and outputs a gate status: ready_to_plan, need_more_evidence, or halt. The first status triggers evidence-conditioned task planning, the second triggers another evidence-acquisition iteration, and the third stops execution when the required evidence remains unsupported.
Feasibility Gating
If the VLM determines that the acquired evidence is sufficient to plan for achieving the goal, it outputs ready_to_plan. The goal represents the task the robot is expected to achieve. In this case, the system exits the evidence-acquisition loop and proceeds to evidence-conditioned planning.
Kitchen Environment and Robot Embodiment
We use a kitchen environment for making chicken soup. The task requires long-horizon planning over object search beyond the initially visible area, access to storage spaces, placement of ingredients and seasonings, and the use of water and a stove.
The environment consists of five movable objects, eight support surfaces such as counters, stove burners, a sink, and a pot, two storage spaces enclosed by doors or a drawer, and six articulated objects such as doors and knobs. Objects are added to the PDDL problem only after they appear in the robot’s observation images. This environment was used in VLM-TAMP as a benchmark for evaluating long-horizon task planning and geometric feasibility.
In our experiments, all doors are initially closed. The chicken leg is on the top refrigerator shelf, and the salt and pepper shakers are on the cabinet’s left and right sides, respectively, from the robot’s viewpoint.
The robot is restricted to using only its left arm, so object rearrangement, storage-space access, and articulated-object manipulation must be performed within a limited reachable workspace. The robot can manipulate movable objects through pick-and-place actions and operate articulated joints by pulling, pushing, and rotating its wrist.
As a result, successful execution requires jointly handling unobserved regions, closed storage spaces, and geometric constraints during cooking-related manipulation.
Results
We evaluate the effect of EAFG by comparing VLM-TAMP with and without EAFG across three settings.
EAFG provides the largest benefit in S2, where the goal is underspecified: Recipe CR increases from 0.05 to 0.40 for GPT-5.5 and from 0.00 to 0.20 for Gemini-3.5-Flash because EAFG discovers the unmentioned but task-relevant salt and pepper and incorporates them into the task plan.
In S3, EAFG improves infeasible-goal handling by increasing Halt SR and reducing unnecessary execution attempts toward a missing carrot.
In S1, where the instruction is explicit and all required objects are present, EAFG remains competitive with the baseline and improves Recipe CR for GPT-5.5.
Limitations
EAFG relies on the TAMP module to instantiate each evidence-acquisition subgoal as a geometrically and kinematically feasible low-level action sequence. However, EAFG does not explicitly handle recovery from manipulation failures that occur during evidence acquisition.
Therefore, when TAMP fails, the system may be unable to acquire that task-relevant evidence in the environment, causing feasibility gating to assess feasibility based on insufficient information.
Despite this limitation, EAFG is compatible with a wide range of methods that combine VLMs with structured planners, and its robustness is expected to improve over time as the underlying planning and feedback mechanisms advance.
Conclusion
We proposed EAFG, a framework that actively acquires visual evidence before task planning and uses a feasibility gate to decide whether to plan, explore further, or halt.
Experiments show that EAFG improves grounded task completion under partial observability by discovering task-relevant hidden objects before planning. EAFG also promotes appropriate halting when required objects cannot be confirmed, reducing unnecessary execution attempts toward absent objects.
Frequently Asked Questions
What is Evidence Acquisition and Feasibility Gating? EAFG actively acquires visual evidence before task planning and uses that evidence to determine whether planning should proceed.
What gate statuses can the VLM produce? The VLM can output ready_to_plan, need_more_evidence, or halt.
What cooking environment is used for evaluation? The experiments use a kitchen environment for making chicken soup, involving hidden objects, storage spaces, ingredients, water, and a stove.
What is a limitation of EAFG? EAFG does not explicitly recover from manipulation failures during evidence acquisition, so feasibility may be assessed using insufficient information.