Panoptic Segmentation for Fruit Harvesting

Agriculture is one of the oldest human activities, dating back over 11,000 years, and has always evolved alongside technological progress. Innovations in tools and machinery have historically aimed to reduce the physical and mental fatigue of farmers. Nevertheless, many operations, such as pruning and fruit harvesting, are still predominantly carried out manually, either because they require high precision or because existing solutions are far from being effective. Current research increasingly focuses on robotic and automated solutions, whose effectiveness, however, depends strongly on the ability of machines to perceive and interpret complex natural environments. In the case of fruit harvesting, the focus of this thesis, state-of-the-art perception technologies typically focus on detecting fruits while neglecting the surrounding structures that are equally crucial for robotic navigation and manipulation. This work addresses this gap by investigating panoptic segmentation, a computer vision approach that unifies instance and semantic segmentation, to enhance perception in agricultural environments, with particular focus on apple orchards. To overcome the lack of suitable public datasets, two dedicated resources have been created: SPARTA (Synthetic Panoptic Apple oRchard Tree Annotations), a synthetic dataset designed for controlled variability and scalability, and ATHENS (Apple Tree Harvesting Environment with Natural Scenes), a real-world dataset capturing the complexity and heterogeneity of natural orchard conditions. The latter was acquired during two outdoor campaigns in Saluzzo (Cuneo, Italy) using multiple stero cameras. On the methodological side, the thesis studies, compares, and extends state-of-the-art panoptic segmentation architectures, including Panoptic-DeepLab and ESANet, introducing modifications tailored to the task. The goal is to assess the effectiveness of synthetic datasets supporting in real-world scenarios, analyzing how resolution, dataset size, and variability affect final segmentation performance. Experimental results show that a more variegated synthetic dataset tends to reduce performance on the synthetic benchmark itself but improves generalization to unseen real data. Moreover, incorporating depth information brings a slight gain in segmentation performance, although it comes at the cost of roughly doubling computational requirements during training and inference. Overall, this work provides contributions ranging from the creation of new datasets to methodological insights on segmentation architectures and synthetic-to-real transfer, thereby supporting future advances in robotic perception for agriculture.