Exploring a Photorealistic Simulator to Help Data Collection for Real-World Applications

The increasing demand for data to train Neural Networks and other AI models is driving the development of techniques that enable programmers to acquire new data in a faster, cheaper, and easier way. This thesis focuses on creating a synthetic dataset using NVIDIA Isaac Sim, an application built on NVIDIA Omniverse, which facilitates the creation of a photorealistic "digital twin" of a specific environment and the randomization of its elements. Synthetic data is a class of data artificially generated, coming from the digital world rather than real world, with the main advantage of having almost unlimited amount of data, already labelled, that can deliberately include rare but crucial corner cases, or, in other cases, overcome the problem of not having data for that specific task. The goal of the Company is to gain experience with NVIDIA Isaac Sim, trying to see if it could be a solution to use it in the future to reduce costs and increase the speed of dataset creation for their technologies. The case study taken into consideration is “MI.RA Depalletizer,” a robotic system developed by COMAU capable of handling and depalletizing Stock Keeping Units (SKUs) of various unknown shapes, materials, and textures using a combination of AI algorithms, 3D cameras, and image analysis. The real dataset used during the product's training phase serves as a benchmark for analysing the performance of a synthetic dataset compared to a real one, by doing different experiments (e.g., a mixture of the two datasets) and trying to define the way synthetic data could enhance the training process. The synthetic dataset comprises images of boxes with random sizes, textures, colours, and positions scattered on a pallet. The environment can vary, incorporating different lighting conditions, colours, ground textures or disturbances (e.g., lights, shadows, or unexpected objects in the scene). The results of this comparison are used to fine-tune the simulator parameters to consistently achieve better, clearer, and more photorealistic data. The final step will involve creating a dataset for another system developed by COMAU to compare the results with the actual dataset for that specific application