The proposed contact-feature-augmented learning-optimisation hybrid algorithm framework achieves prominent generalisation capability across arbitrary robotic manipulators
A research team led by Professor Xu Qingsong in the Faculty of Engineering at the University of Macau (UM), has achieved a breakthrough in the field of universal robotic manipulation. The team has developed a comprehensive framework for universal robotic grasping that can be adapted to complex scenarios, enabling the rapid deployment of customised and commercial robotic hands. The framework also offers an innovative technical solution for advancing flexible automated production, medical assistive robots, and service robotics. The research findings have been published in Nature Machine Intelligence, a top-tier international journal in artificial intelligence.
As intelligent service robots are deployed in an increasing number of scenarios, two issues with existing manipulation algorithms are hindering their large-scale commercial application: insufficient adaptability across hardware platforms, and poor robustness in complex environments with limited sensory perception. Traditional deep learning models that rely on monocular depth vision struggle to accurately characterise the physical contact between robotic hands and objects. Moreover, these models exhibit significant limitations in terms of generalisation when performing grasping tasks with multi-fingered and multi-degree-of-freedom robotic manipulators of varying models.
To address these industrial challenges, the research team explored the fundamental mechanisms of cross-manipulator universal grasping. They innovatively decoupled the kinematic model of robotic hands from the interactive contact states during manipulation, thereby establishing robust and deployable contact state features that are independent of specific robotic hand models. Using existing paired grasping datasets, the team constructed a large-scale, device-agnostic database of decoupled contact features. This database was then used to train neural networks to restore contact states from visual scene observations. By integrating a contact feature neural network as the environmental perception module and a universal analytical optimisation grasping framework based on contact feature reconstruction, the developed system achieves zero-shot adaptation. This enables stable grasping operations for all two- to five-fingered hinge-type robotic hands in cluttered, unstructured, and densely stacked three-dimensional environments.
The study proposes the first learning-optimisation hybrid algorithm framework to enable zero-shot robust grasping generalisation across different robotic hands in highly cluttered scenarios. It achieves state-of-the-art grasping performance in three-dimensional space. Validations across extended application scenarios demonstrate that the hybrid algorithm architecture is highly flexible and adaptable for diverse manipulation tasks. Adopting a low-cost sensing scheme, the framework presents great potential for practical deployment. It can rapidly enable commercial and customised robotic hands to perform intelligent manipulation in dense and cluttered environments, providing a novel technical pathway for advancing flexible automated production, medical assistance robots, and civilian service robots.
The corresponding author of the study is Xu Qingsong. The first author is Wang Xianli, a postdoctoral fellow in the UM Faculty of Engineering, and Tam Lap Mou, professor in the UM Faculty of Engineering. The research was supported by the National Natural Science Foundation of China (Grant No.: 52175556), the Science and Technology Development Fund of the Macao SAR (Grant Nos.: 0004/2022/AKP, 0102/2022/A2, 0078/2023/RIB3), and the University of Macau (Grant Nos.: MYRG-GRG2024-00222-FST-UMDF, MYRG-GRG2025-00121-FST). The full article is available at: https://doi.org/10.1038/s42256-026-01292-y.
