Semantic Obstacle Avoidance of Robotic Arm for Fruit Harvesting

With the technological progress of robotics, Artificial Intelligence and vision systems, the projects and applications that regard the automation of simple and repetitive tasks are spreading, and, from a business point of view, business owners are more interested in automating activities for which it is difficult to find personnel. Moreover, Fruit Harvesting heavily relies on intensive human labor and sometimes it can be physically challenging. These considerations, together with the need to decrease the cost of the harvesting task (the most expensive regarding fruit agriculture), leads to the development of solutions that involve the help of automation. Despite the efforts of the actual research in robotics, the main issue of this kind of system is that machines and robotic systems need to reach a high level of complexity in order to behave and execute a series of tasks that a human would perform unconsciously and with a certain dexterity (for example the collision avoidance of the arm with the environment). The solution proposed in this work is an automated fruit harvesting system employing a manipulator placed on a mobile platform that moves through the orchard. The manipulator that has been used to develop this thesis is a 6-degree-of-freedom robot. The vision capabilities are guaranteed by an RGB-D camera that is used in common vision applications. To complete and enhance the capabilities of the vision system, it has been necessary to train a semantic segmentation model (based on Convolutional Neural Networks) to make sure that the environment is correctly mapped and to provide the right inputs to the system. The system relies on a middleware, by which it is possible to build a robust logic and to develop a reliable and working application. Moreover, the usage of a platform for robot control, has been fundamental for path/trajectory planning and, thanks to its planners and plugins, it has been possible to achieve a good result in terms of system reliability. The presence of built-in functions resulted to be fundamental for the collision avoidance of the manipulator with the so-called hard obstacles that can cause damage to the manipulator or the plant itself. This thesis aims to develop an automatic fruit harvesting system that can be comparable to human performances and it can work autonomously without colliding with potential sources of damage. This work intends to build a vision system that is capable of reliably recognizing the elements of the scene to allow a clear understanding of the environment. The results show that the system is highly reliable and it can successfully grasp a discrete amount of apples in relative short time. Furthermore, the system can perceive and recreate a high fidelity environment thanks to the high accuracy that characterize the semantic segmentation model.