Multi-sensor mobile perception system for machine learning applied to human recognition

Computer vision techniques, such as machine learning implemented for pedestrian recognition, are frequently employed in safety contexts for autonomous or semi-autonomous driving systems. Since the data redundancy is crucial in the field of human safety, a multiple sensor system is usually used. The collected data is then combined in order to obtain the most accurate estimation of the proximity of the vehicle. The robotic model presented in this work is a mobile platform with three cameras and a lidar mounted on the top of an aluminium profile. Intrinsic and extrinsic parameters are the object of the calibration part of this project; a particular focus must be done on the calibration of the thermal camera, due to the fact that it can't detect colors and can't be set through canonical methodologies. A convolutional neural network is firstly trained with a subset of the collected data. Then the remaining part of the data is used to validate the parameters of the network, testing also the performances in different weather and light conditions. The errors and confusion matrix are then used to evaluate the detecting system and computed thanks to the testing and validation phase of the job. Finally, data from two different cameras are combined using a late-fusion technique and leading to a further improvement in the human detection goal. The final results are positive and can be used for further improvements, for instance the network could be used for real time detection. In this case, this work can be also seen as a possible starting point for the applications in the fields of assisted drive and autonomous vehicles.