System Overview
This structure sits between classic robot middleware and recent language-based robotics. LLM and Vision-Language-Action (VLA) systems demonstrate powerful task-level priors and generalization.
Object Poses and Operation Parameters
The third handoff converts perception into runtime-facing objects. A WorldObject contains a 6D pose, frame ID, confidence, roles, skills, and metadata. The FoundationPose service boundary is now a concrete HTTP/RGB-D protocol: the agent sends target queries, an output frame, camera intrinsics, RGB, depth, and optional masks; the GPU-side service returns object names, frames, confidence, skills, roles, poses, and mesh metadata. The service also exposes health, model, and object endpoints, with a mock mode for software integration before the GPU perception stack and masks are available.
The prototype then enriches WorldObject records with deterministic operation parameters. Given pose, computed size, roles, available skills, and runtime bindings, the operation-parameter planner generates approach, grasp, lift, place, insert, pour, dispense, tap, swipe, and twist parameters. These defaults are stored as perception_args, skill_args, and target_skill_args, then merged by the runtime when compiling a skill call.
This handoff serves as an interpretable bridge between perception outputs and skill templates, so that a dry-run trace can explain exactly why a particular pick pose, place pose, insertion depth, or pour clearance was chosen.
Goal and Task Representation
This stop-on-failure policy is deliberately conservative. It prevents a language planner from papering over missing perception or unsupported skills. It also makes the representation handoffs easy to debug: a failure can be localized to scene origin preparation, world grounding, planner validation, skill coverage, operation-parameter resolution, runtime precheck, safety validation, executor routing, or hardware execution.
Field Lessons
Lesson 1: 6D Pose Is Necessary but Insufficient
The outputs of FoundationPose are valuable, but a mobile manipulation system needs pose plus frame, confidence, object identity, geometry, skills, roles, and target-specific operation parameters. The downstream planner should see an actionable object, not a raw perception result.
Lesson 2: Profiles Are Representations, Not Merely Configurations
The usage, calibration, and development profiles encode three different forms of state: object priors, geometric deployment facts, and capability contracts. Treating these profiles as first-class representations made startup checks and dry-run traces more informative than ad hoc configuration files.
Lesson 3: Constrained Intermediate Forms Make LLM Planning Debuggable
The LLM only proposes registered skill calls under a skill bank contract. It cannot invent arbitrary ROS commands, bypass role requirements, use undeclared arguments, or execute unregistered recovery. This design sacrifices flexibility, but it makes unsupported capabilities visible as unmapped_requests rather than hidden model assumptions.
Lesson 4: Deployment Blockers Are Representation Blockers
The remaining real-robot work is not just hardware bringup. It is the completion of missing representations: measured camera, lidar, and slide transforms; table frame localization; object meshes; real RGB-D masks or detections; and calibrated workspace bounds.
The field report value is that these blockers appear at the representation interfaces, where they can be tested.
Conclusion
This report reframes an OpenArm-based embodied intelligence prototype as a set of representation handoffs across system integration. The central artifact is not a new perception model or controller, but an auditable path from laboratory instructions, maps, object poses, object priors, and skill bank contracts to validated skill calls, operation parameters, motion goals, and runtime feedback.
The current implementation provides software-level and dry-run evidence that these handoffs are executable, while also making clear what remains before real-scene visual manipulation can be claimed. The next step is to replace placeholder calibration and model assets with measured field data, align the FoundationPose service with the current usage-profile object registry, enable strict real-scene grounding, and evaluate task-level failures across pick, place, insert, pour, clean, and screen-interaction tasks.
Acknowledgments
This work is supported by the Shenzhen Fundamental Research Program under Grant No. JCYJ20200109141235597, National Natural Science Foundation of China under Grants No. 72401122 and 61761136008, Guangdong Basic and Applied Basic Research Foundation under Grants No. 2024A1515012241 and 2021A1515110024, Shenzhen Peacock Plan under Grant No. KQTD2016112514355531, Program for Guangdong Introducing Innovative and Entrepreneurial Teams under Grant No. 2017ZT07X386, and the Australian Research Council (ARC) under Discovery Grant DP210101093 and DP220100803.
Frequently Asked Questions
What does the third representation handoff convert? It converts perception into runtime-facing objects containing poses, frames, confidence, roles, skills, and metadata.
Why is the stop-on-failure policy conservative? It prevents language planners from masking missing perception or unsupported skills and makes failures easier to localize.
What information should the downstream planner receive? It should receive an actionable object with pose, frame, confidence, identity, geometry, skills, roles, and target-specific operation parameters.
What remains before real-scene visual manipulation can be claimed? Placeholder calibration and model assets must be replaced with measured field data, strict real-scene grounding must be enabled, and task-level failures must be evaluated.