This work considers two different approaches to the design of an end-to-end pipeline, to process RGB-D inputs and feed a control logic with an affordance map. This task is supported by a small-size circuitry interfaced with a sensor, encapsulated within a portable framework. In both cases, the main challenge is to develop an end-to-end pipeline that meets accuracy requirements and runs on an embedded device that satisfies the constraints on energy consumption and real-time performances.
Hardware-Aware Neural Architecture Search
HW-NAS typically drives the design of tiny, hardware-efficient DNN models, since it allows to tailor the network architectures to the target hardware platforms. When adopting HW-NAS, one should define a search space, a selection criteria, and a research strategy.
The search space covers the set of admissible candidate networks. Several studies showed that the design of the search space is crucial to the eventual performance. Popular general-purpose search spaces have been established.
Designing Efficient Portable Sensing Systems Based on DNNs for RGB-D Data
Two different approaches are considered for the design of an end-to-end pipeline, to process RGB-D inputs and feed a control logic with an affordance map. This task is supported by a small-size circuitry interfaced with a sensor, encapsulated within a portable framework.
The main challenge in both cases is to develop an end-to-end pipeline that a) meets accuracy requirements and b) runs on an embedded device that satisfies the constraints on energy consumption and real-time performances.
Fine Tuning-Based Approach
HW-NAS may exhibit some drawbacks. Firstly, it may involve remarkable computational costs for training. Secondly, it requires large labeled datasets; this can limit its applicability, since obtaining labeled masks for affordance segmentation can be very time-consuming. Finally, the definition of affordance can vary depending on the context, and potentially require network retraining.
Therefore, an alternative design strategy that does not involve HW-NAS is envisioned. The main goal is to apply a fine-tuning process for including depth information in a pre-trained, highly optimized architecture for RGB inputs.
The additional layer integrates in the architecture, and its filter values are learned by conventional backpropagation. This allows to optimize the input mapping while preserving the pre-existing weight configuration. One might argue that altering the base layer of a deep network compromises fine-tuning, since retraining typically affects the topmost layers. In fact, depth maps share many structural similarities with RGB data — features optimized for RGB images are likely to work well for depth data, as well. This allows the optimization process to adjust low-level features and incorporate depth information, without significantly affecting the original network configuration.
Experimental Setup
The code was developed in Python using TensorFlow and Keras libraries. Two well-known datasets, UMD and IIT, provided the benchmarks. The University of Maryland (UMD) dataset includes 28,843 RGB-D images across 7 object categories, with multiple framing angles for each object, allowing an assessment of the models' ability to handle framing variations.
The Italian Institute of Technology (IIT) dataset includes 8,835 images under varying framing, lighting conditions, occlusion levels, and resolutions. Each dataset features distinct object sets, serving as separate benchmarks.
Validation sets were created from the training set using a standard holdout method, and the test patterns were never used for parameter or hyperparameter tuning.
In the approach based on fine tuning, MobileNetV3 was adopted as backbone. MobileNetV3 has relatively low computational requirements while maintaining satisfactory performance on standard computer vision tasks, such as ImageNet classification. The model checkpoint, pre-trained on the ImageNet dataset, was downloaded from the Keras application repository. The same segmentation head described in the original work was incorporated into the architecture. The settings used for the final architecture selected by NAS were also applied in the fine-tuning.
Network hardware requirements have been estimated using parameters and FLOPS. These measures provide an estimation independent of the target deployment devices. However, target-specific measurements, based on hardware in the loop, may exhibit different trends depending on the specific SDK.
Efficient Usage of Depth Information
The capability of the proposed approach to efficiently utilize depth information is evaluated by analyzing both generalization performance and hardware requirements. Figure 2 shows the generalization performance achieved with the two approaches on the UMD dataset based on the use of depth data. In both cases, the radar plot gives the accuracy obtained for the three classes and the average classification score; the end-to-end pipeline fed with RGB-D data is compared with the pipeline fed with RGB data.
For the approach involving the HW-NAS, in the experiments without depth, the search space was simplified by removing the depth backbone. The availability of depth information led to an increase in accuracy across all classes. The improvements were most pronounced for the 'Grasp' and 'Don't Grasp' classes with gains of 2.8% each, which are generally more challenging.
For the fine-tuning-based approach, the network using depth data outperforms the baseline for the 'Grasp' and 'Don't Grasp' classes by 2.6% and 3.6%, respectively. For the 'Background' class, a negligible deterioration of 0.3% is observed, which could be attributed to statistical fluctuations in the optimization process. Both tests confirmed the benefit of depth information in improving the model's generalization performance.
Impact of Input Size on Performance
FLOPS remain the best device-agnostic estimator for inference time. The reliability of this estimator increases as the number of cores in the computing unit decreases. For example, in microcontrollers, the correlation between FLOPS and inference time approaches 1.
Output Analysis
The actual impact of depth sensors on affordance segmentation may vary significantly across different objects. An analysis was conducted on a test set generated with Unreal Engine 5. This setup enabled the creation of many heterogeneous images through a cost-effective procedure, where test patterns composed of RGB-D images and affordance masks were generated using the simulator.
A test set of 2,000 images was created, with objects belonging to the same classes available in the UMD dataset. Models trained on UMD were then tested on this test set, showing an improvement of 11.5% in the average accuracy when the depth information was used. This gap was reduced to 4.6% when data augmentation was applied to mitigate cross-domain issues commonly associated with simulator-generated images. The results confirmed the advantage of using depth data to generalize to unseen test objects, thanks to the geometric information it provides.
Deployment on Embedded Accelerators
The two proposed models using RGB-D data have been deployed on a prototype consisting of a Jetson Nano interfaced with a RealSense RGB-D camera, one of the most widely used RGB-D cameras. The Jetson Nano was chosen among the available embedded accelerators because of its excellent balance between performance and power consumption.
The comparison involved four models: the plain TensorFlow implementation of NAS RGBD and Mob3 RGBD; the optimized versions (_opt) of NAS RGBD and Mob3 RGBD obtained using the TRT optimizer with float16 quantization.
All the data on latency and memory usage were obtained by completing 100 inferences on RGB-D images captured by the camera. Unless explicitly stated otherwise, the reported metrics represent average values. Memory usage was monitored by relying on the operating system's utilities. Power consumption was measured by recording the current drawn at the board's power input using a wattmeter. Framing conditions were not particularly relevant to the hardware requirements, as they are independent of the input image.
The first analysis evaluates the inference time of the models for two power configurations of the Jetson Nano: 5W and 10W. In both cases, the value corresponds to the estimated maximum peak power consumption. The table gives, for each tested model, the time needed to process a frame and the frame rate expressed in frames per second.
The results show that optimization had a significant impact on the inference time for both models. Both optimized versions enable inference under 120 ms, supporting relatively fast processing suitable for real-time operations intended for human users.
Conclusion
The paper analyzed two different design strategies for the deployment of affordance segmentation in wearable sensing systems. The results showed that by including RGB-D sensors in the sensing pipeline one can improve the generalization performance with respect to the setup involving only RGB cameras. Extensive experimental analysis on two well-known real-world benchmarks proved that the proposed strategies can generate architectures lying on the Pareto-optimal front for generalization performance and hardware requirements. The architecture generated with the proposed strategies can process images in real-time with acceptable power consumption when running on a prototype based on a Jetson Nano and a RealSense RGB-D camera.
The current version of the system has been evaluated using foreground images, leaving the correct framing to the end users, which could prove challenging. Future versions will incorporate object localization capabilities, enabling a simpler framing setup. Additionally, future experiments will apply the same techniques to transformer models by designing dedicated search spaces or adapting the fine-tuning approach for models with positional encoding. Clinical trials will also be required to confirm the user's perceived benefits in the semi-autonomous pipeline, where the proposed system will be included in the full control pipeline, allowing an in-depth analysis of how efficient affordance segmentation can improve user interaction.
Frequently Asked Questions
What are the two main design strategies explored in this paper for RGB-D affordance segmentation? The paper explores hardware-aware neural architecture search (HW-NAS) and a fine-tuning-based approach that adapts a pre-trained RGB model to incorporate depth information by adding a learned input layer.
How much accuracy improvement does depth information provide at the class level? Depth information improved 'Grasp' and 'Don't Grasp' class accuracy by up to 2.8% with HW-NAS and 2.6-3.6% with the fine-tuning approach, while 'Background' accuracy remained nearly unchanged.
Can the proposed models run in real-time on embedded hardware? Yes, both optimized models achieved inference times under 120 ms on a Jetson Nano with a RealSense RGB-D camera, supporting real-time operation for human users.
What future directions are planned for this system? Future work will add object localization for simpler framing, extend the techniques to transformer models, and conduct clinical trials to evaluate user benefits in a full semi-autonomous control pipeline.