Assembling parts together to obtain a final product is a trivial task for humans. However, achieving a similar level of skill for a robot remains a complex challenge. Research on robotic assembly spans multiple domains with diverse applications and variable constraints, including robotic manufacturing, service robots, medical robots and applications, space exploration, fractured-part reconstruction in archaeology, and repair and restoration. There is continuous demand for advanced assembly capabilities in robots despite the extensive amount of study. Although current research addresses these challenges through visual perception and reinforcement learning, it remains debatable whether these systems exhibit the behavior of an intelligent robot with assembly task awareness. Assembly task awareness and an end-to-end learning method are introduced to enable it for robotic manipulation.
Introduction
Computer vision tools such as object detection, classification, and semantic segmentation are paramount to endowing robots with perception. While these tools provide basic scene understanding, robotic manipulation requires 3D spatial understanding of the robot environment. Consequently, 6-DoF object pose estimation emerged as a critical component in robotic perception and has become pivotal in robotic grasping and object placement.
However, a multi-object assembly task imposes more requirements on context awareness. Spatial awareness is crucial for localizing pre-assembled and post-assembled poses of assembly objects. Assembling objects in an optimal sequence requires temporal awareness. Inter-part relational awareness determines the structural coherence of a final assembly. Moreover, awareness of stability under gravity is essential for physical feasibility.
These observations formulate the main components of task awareness required for a progressive assembly task. This awareness can be interpreted as the ability of an entity to perceive an assembly configuration and reason about its spatial, temporal, and relational dependencies. A module that can learn to produce assembly-aware actionable outputs is extremely beneficial for robotic assembly manipulation. This is critical for modern robotics because it enables manipulation capabilities beyond primitive pick-and-place operations toward task-aware manipulation.
To this end, PVRA, a pointwise keypoint voting framework for robotic assembly, is introduced. The main contributions are:
- A 3D keypoint-based modular learning framework to enable assembly task awareness for robotic entities.
- An augmented set of metrics to benchmark assembly-aware role segmentation and pose estimation.
- Implementation and evaluation of the proposed framework on a custom object assembly dataset.
- Comparative evaluation of the proposed model against object-centric baselines.
Assembly Pose Estimation
Given a set of parts or components, Assembly Pose Estimation (APE) entails inferring the target object’s 6-DoF pose in the assembled product while satisfying the assembly constraints. In the absence of prior knowledge about the semantics of the parts, APE can be defined as generalized part assembly.
Generalized part assembly is capable of assembling novel or unseen objects given a target assembly. GPAT addresses this problem through target segmentation, where the network explicitly segments the target blueprint and infers the assembly poses. However, GPAT assumes a global and single-step APE, which overlooks intermediate steps in a progressive assembly task. This creates an adverse effect on the feasibility and accuracy of performing a progressive assembly task because of the lack of progressive reasoning and incorporation of physical constraints.
Proposed Method
PVRA is a supervised RGB-D point-cloud model for progressive assembly under the specific assumptions described in the paper.
The method perceives an RGB-D input of the assembly scene and extracts features to learn a mapping that produces assembly-task-aware actionable outputs. These outputs include:
- Point-wise semantic roles: target, base, and background.
- The 6-DoF pose of the target object.
- The 6-DoF assembly pose.
The PVRA model does not represent a complete controller that executes an assembly task. Instead, it predicts task-aware actionable outputs that enable downstream tasks for an autonomous agent to execute an assembly action at the current step of a progressive assembly.
The model shares an RGB-D feature-fusion strategy with PVN3D but adopts prediction heads and supervision directed toward the progressive assembly task.
Experimental Setup
PVRA is trained and validated on the Nema17 gear-reducer progressive assembly dataset. The dataset contains simulated assembly scenes with ground-truth object poses, assembly poses, RGB-D frames at each assembly step, and CAD files of associated objects.
The dataset consists of 431 assembly instances containing five assembly objects and four assembly steps. This yields 8,620 instances, which are used to generate a 60% training split, a 20% validation split, and a 20% test split.
The network was trained on a high-performance computing cluster consisting of four NVIDIA Tesla V100 GPUs with 128 GB of VRAM. The source code is hosted in a GitHub repository.
Metrics
Step-Segmentation Accuracy
Step-Segmentation Accuracy (SSA) measures assembly step-level segmentation accuracy using the average Intersection-over-Union (IoU) of the target and base roles. It reflects PVRA’s ability to identify the active target and base objects with temporal and relational awareness.
Step-Localization Accuracy
Step-Localization Accuracy (SLA) measures pose accuracy using normalized Maximum Symmetry-Aware Surface Distance (MSSD). Continuous local symmetries are declared for all Nema17 objects so that equivalent rotations are not penalized.
The formula uses the target-object diameter to normalize the symmetry-aware surface-distance error.
Discussion
The evaluation uses a synthetic dataset of a progressive assembly. A real-world deployment may introduce additional challenges such as sensor noise, calibration errors, reflections, and lighting conditions.
While domain transferability remains a future research direction, the simulation-based baseline comparison contributes an important reproducible benchmark for perception-based progressive assembly. As evident from the literature study, the availability of real-world structured datasets that represent progressive assemblies is limited. This is due to the complexity of capturing synchronized RGB-D frames and annotated 6-DoF poses in a real-world setting.
Therefore, this controlled setting is essential for isolating the methodological question: can an assembly-context-aware model achieve comparable performance to existing object-centric pose-estimation methods?
Conclusion
This work presented a task-aware, keypoint-based framework for progressive robotic assembly and a quantitative evaluation compared with existing object-centric perception methods. The evaluation confirms the ability of the model to learn assembly-task-specific dependencies and produce accurate actionable outputs for robotic manipulation.
Furthermore, this work motivates extending assembly-specific awareness toward broader task-specific awareness learning, which can benefit a wider range of robotic manipulation applications beyond the scope of progressive assembly.
Project funding was received from the Helsinki Institute of Physics’ Technology Programme through the ROBOT project and the Research Council of Finland under Grant 369003 through the PERFORM project. The authors acknowledge CSC—IT Center for Science, Finland, for computational resources.
Frequently Asked Questions
What is PVRA?
PVRA is a supervised RGB-D point-cloud framework that uses 3D keypoint-based learning for progressive robotic assembly.
What outputs does PVRA predict?
PVRA predicts point-wise semantic roles, the target object’s 6-DoF pose, and its 6-DoF assembly pose.
What dataset is used to evaluate PVRA?
The framework is evaluated on the Nema17 gear-reducer progressive assembly dataset, which contains simulated scenes, RGB-D frames, object poses, assembly poses, and CAD files.
What challenges may affect real-world deployment?
Real-world deployment may introduce sensor noise, calibration errors, reflections, and changing lighting conditions.