TaskGraspNet: Object Detection and Task-Oriented Grasping in Cluttered Scenes

The ability for robotic systems to detect objects in a scene and grasp them appropriately is fundamental for performing a wide range of actions. While this has the potential to significantly impact various sectors of our society, it remains challenging due to the computational requirements and the complexity of the problem at hand. While many works have provided methods to learn how to approach and grasp objects optimizing metrics like stability, robustness and velocity, little has been done to provide models with the capability to reason about how to specialize grasping strategies depending on follow-up actions, similarly to how humans change their grasping strategy of a tool depending on the purpose of the grasp (using or handing-over). This thesis addresses the challenge of task-oriented grasping in scenes or environments full of objects, focusing on developing an efficient solution that is also deployable on edge devices. This is achieved with the development of TaskGraspNet, a novel large-scale dataset for task-oriented grasping, built upon the GraspNet-1B dataset. This augmented dataset includes task annotations for each grasp, providing a rich resource for training and evaluating task-aware grasping models. Additionally, a unified pipeline RGBTaskGrasp was deployed, capable of simultaneously predicting the location of objects of interest and generating feasible grasps to perform specific actions. This approach demonstrates how a single model, trained on RGB data and using a pretrained backbone, can reliably perform both object localization and task-specific grasp generation. Through extensive experiments with real-world environments, we validate the effectiveness of our method in generating valid grasps for various tasks with real-world objects. This work contributes to advancing the field of robotic manipulation by providing both a valuable dataset and an efficient, task-aware grasping model.