Computer Vision-Based System for Perception and Control in Autonomous Grape-Harvesting Robot

The topic of this thesis represents a portion of a larger multidisciplinary research project focused towards the design and development of a completely autonomous grape-harvesting robot. The final goal of the project is the implementation of a system that can perform vineyard harvesting operations in an independent, efficient, and precise way and with the least possible human intervention. The robot has the capability to navigate in the three-dimensional task space and also rely on various sources of information, such as visionary information picked up by the video camera, and pressure sensors located on the robotic hand, so as to complete its tasks with high reliability and accuracy. In order to achieve this goal, it is necessary to engage a number of coworkers who each have responsibility over different subsystems like the robotic scissor and hand mechanical design, sensor integration, and actuation. My specific role within this project, as described in this thesis, is the computer vision subsystem and the control of the robot as a whole moving entity. The first part of the thesis deals with object detection and computer vision, which form the basis of robot autonomy and perception. Convolutional neural networks (CNNs) were employed as the central tool for detecting the right objects in the vineyard setting. After exploring different methods and studying past efforts, the YOLOv8n model was selected. This architecture, in combination with the Nvidia Jetson Nano, was found to be the most suitable for this job. It offers a reasonable balance of accuracy, computational cost, and real-time performance. The training data set was initially provided by my professor and augmented with additional images that I personally collected in order to expand both its size and diversity. For the purpose of maintaining uniformity, all the images were designed in JPG format with a pixel resolution of $640 \times 853$. Having prepared the images, the annotations were inserted manually into them in .txt format by using cvat.ai.With the dataset prepared, the training process was performed on an Nvidia Jetson Nano, a compact yet effective tool with a dedicated GPU that allowed simple experimentation and iteration. The trained model performed up to satisfactory levels, with high precision and the ability to recognize multiple objects in a stream and consistently predict them. This consistent model is a necessary building block for deployment in actual vineyard conditions. Building on the success of the computer vision module, the second part of the thesis explores the overall control structure of the robot. A state-machine architecture was used to manage the types of operations and task switching. The control system combines and integrates information from different sources: visual data from the RGB and depth cameras, force and pressure readings from sensors located inside the robotic hand, and feedback from the base body of the robot, the robotic manipulator, and the cutting tool (scissors). This integration enables the robot not only to perceive the world, but also to decide and move its subsystems in real time. The information flow among perception, decision, and actuation is essential to the robustness and reliability of the robot's operation. Closed loop controls and smaller state-machines were applied within every task of the robot to ensure success. Different tests were done in the lab in order to optimize and finalize our whole system.