A robot should be able to reuse a manipulation skill it learned on a new object that serves the same function, even when its geometry is substantially different: a mug with a different handle shape, a bottle with a different cap, a pair of pliers with a different jaw length.
Existing approaches typically transfer demonstrations by identifying semantically corresponding interaction regions or affordances. While sufficient for locating approximate contact regions, these methods often fail to capture the functional intent of manipulation skills: not just where to interact with an object, but how, in what sequence, and with what relationship to the object’s articulated structure.
Dense correspondence naturally addresses this limitation by preserving a dense mapping between object surfaces. For manipulation, semantic correspondence determines where a robot should interact, whereas geometrically coherent correspondence determines how that interaction should be executed through the recovery of consistent local geometric frames. Effective skill transfer therefore requires correspondences that are both semantically consistent and geometrically coherent, motivating a dense correspondence formulation.
We propose Semantic Anchored Correspondence (SemAnCorr), a training-free framework that establishes dense correspondence across object instances by anchoring semantically meaningful regions and propagating these constraints over the object surface using functional maps. By jointly preserving semantic consistency and geometric coherence, SemAnCorr recovers the local geometric structure required to transfer demonstrated manipulation skills across geometrically diverse objects.
We evaluate SemAnCorr on a dense correspondence benchmark constructed from PartNet-Mobility and validate its practical utility through real-world articulated manipulation. Experiments show that improved geometric correspondence translates directly into more reliable zero-shot manipulation skill transfer across previously unseen object instances.
In summary, this paper makes the following contributions:
- We introduce SemAnCorr, a training-free dense correspondence framework that combines semantic anchor selection with functional map propagation to produce correspondences that are both semantically consistent and geometrically coherent.
- We construct a dense correspondence benchmark on PartNet-Mobility that evaluates both semantic accuracy (where to interact) and geometric coherence (how to execute the interaction).
- We develop an object-centric manipulation skill transfer pipeline that leverages SemAnCorr to transfer demonstrated skills across object instances, enabling zero-shot articulated manipulation from a single demonstration.
Method: Semantic Anchored Correspondence
At a high level, the proposed method first identifies semantically meaningful parts on each object’s surface, then selects a small set of corresponding part pairs that are consistent in both meaning and geometry, and finally uses these pairs as constraints to smoothly propagate a dense correspondence across the object surface.
Empirical Evaluations
We evaluate SemAnCorr on both a manipulation dataset and real-world robotic settings to answer two questions:
- Can dense correspondence simultaneously preserve semantic consistency and geometric coherence across geometrically diverse object instances?
- Does improved correspondence quality translate into more reliable zero-shot manipulation skill transfer?
Throughout, correspondences are visualized by transferring a smooth color gradient from the source object to the target: smooth transitions indicate geometrically coherent mappings, while fragmented patterns indicate spatial inconsistencies.
Evaluation Metrics
The evaluation metrics measure correspondence quality with respect to target vertices that have valid semantic labels.
Baselines
We compare against 4 representative baselines spanning geometry- and semantics-driven correspondence approaches.
- FM-WKS estimates functional map correspondences using Wave Kernel Signature descriptors, operating purely on intrinsic geometric structure without any semantic information, serving as a lower bound on contribution of semantic features.
- Robo-ABC establishes pixel-level semantic correspondence in 2D image space using pretrained visual features. We render 8 viewpoints per mesh, take the highest-confidence 2D match, and lift it to 3D. This represents the class of 2D semantic correspondence methods applied to 3D.
- D3Fields constructs per-vertex 3D descriptors by lifting DINOv2 features from multi-view renders and performing nearest-neighbor matching in feature space. This baseline isolates the contribution of functional map propagation over nearest-neighbor given the same 3D feature lifting strategy.
- DenseMatcher combines SD-DINO features with a learned DiffusionNet refiner and functional maps. We evaluate its ability to generalize on our PartNet-Mobility categories.
Cross-Category Generalization
Many manipulation skills naturally transfer across semantically related object categories. We therefore evaluate whether SemAnCorr generalizes beyond category boundaries, and whether the pipeline is able to provide a lightweight signal of cross-category generalization compatibility.
We further investigate whether the semantic anchor confidence computed provides a lightweight signal of compatibility between objects, which may be useful for estimating whether a demonstrated skill is likely to transfer without needing an additional classifier. Since semantic anchor confidence is a novel metric introduced in our method, there is no directly comparable baseline for this analysis.
The average anchor confidence between object categories is highest along the diagonal, corresponding to within-category pairs, and remains relatively high among functionally related categories such as containers (bottle, kettle, bucket, trashcan), hand tools (knife, pliers, scissors), and appliances (microwave, washing machine). In contrast, category pairs with less apparent functional or geometric similarity tend to receive lower confidence scores.
These trends suggest that anchor confidence captures a notion of semantic and geometric compatibility rather than category identity alone. While this score is not a direct measure of manipulation success, it may serve as a lightweight prior for deciding generalization compatibility.
Conclusion and Future Work
We presented Semantic Anchored Correspondence (SemAnCorr), a framework for constructing dense correspondences that preserve both semantic consistency and local geometric coherence across objects. Through benchmarks and real-world manipulation experiments, we showed that both properties are necessary for reliable skill transfer across geometrically diverse instances: semantic consistency determines where to interact, while geometric coherence determines how.
Future work includes extending the correspondence formulation to bimanual and dexterous manipulation, where multiple contact points must remain geometrically consistent. Beyond zero-shot skill transfer, an implication of SemAnCorr is its potential to bootstrap data for learned manipulation policies. Successful executions could be transferred to previously unseen objects to generate additional demonstrations, which could serve as supervision for fine-tuning manipulation policies, reducing the need for repeated human data collection.
Frequently Asked Questions
What is SemAnCorr? SemAnCorr is a training-free framework that establishes dense correspondence across object instances by anchoring semantically meaningful regions and propagating these constraints over the object surface using functional maps.
Why is geometric coherence important for skill transfer? Semantic consistency determines where a robot should interact, while geometric coherence determines how that interaction should be executed through the recovery of consistent local geometric frames.
Does SemAnCorr work across object categories? Yes, anchor confidence is highest within categories but remains relatively high among functionally related categories such as containers, hand tools, and appliances.
What are the future directions for this work? Future work includes extending the correspondence formulation to bimanual and dexterous manipulation, and using SemAnCorr to bootstrap data for learned manipulation policies.