Deep learning 3D facial reconstruction framework for prosthetic rehabilitation

Facial prosthetic rehabilitation aims to provide a patient affected by severe pathologies, or victim of accidents, with the restoration of their facial capability. Previous to the actual medical procedure, the process involves showing the patient what the result of rehabilitation would be. To achieve this goal, current techniques are based on the creation of wax models and other similar artifacts. However, the modern advance in computer graphic techniques as well as machine learning suggests the possibility of moving the process to a computerized domain, thus making the model derivation faster and the model itself easily editable. The ultimate goal of this thesis was to achieve an accurate 3D model of a human face that could act as a replacement to the current methods cited above and to be able to tune it according to specific parameters. To achieve this goal different steps were followed. A fundamental characteristic of the face reconstruction had to be the accuracy in terms of resemblance of the original face. It was determined that an accurate model could not be obtained starting from a single image, but a set of images containing the person's face from different angles were needed. To then be able to automatically modify the model in order to visualize the effects of rehabilitation, it became necessary to detect 3D landmarks on the model. 3D landmarks are in fact mandatory to perform the aesthetic measurements on which to base the tuning of the model. Based on these measurement, the reconstruction was then automatically edited to achieve an aesthetically pleasing result. The 3D model reconstruction of the person's face was achieved by improving an already existing framework, based on deep learning techniques, which allowed to derive a face model starting from a single image. Building on said network, the reconstruction of an accurate model from multiple images was achieved. The automatic detection of 3D landmarks on the model was developed proposing an original implementation based on different state-of-the-art works. Another framework was then developed to perform the needed measurements and to compare them with established beauty canons, based on geometric pleasing characteristics of the human face. Finally, the framework needed to tune the 3D model was developed as well. The latter is designed to either take as input the result of the automatic measurement or an input provided by the clinician to manually determine the adjustments to be performed on the model. The present thesis' work resulted in the accurate and robust reconstruction of a 3D model starting from multiple images of the same subject. On this model a fast and automated 3D landmark detection was performed, with an accuracy comparable to manual land marking. The 3D landmarks were then exploited to perform aesthetic measurements on the model, to be then used to generate a preview for the facial rehabilitation procedure.