Enhancing deep learning techniques for computational social media image analysis with out-of-distribution detection

Social networks are one of the most interesting and widespread phenomenons characterizing our time. Consequently, the involvement and attitude people show towards them are particularly relevant for researchers in several fields, including psychology, philosophy, semiotics and social studies. The ever-increasing volume of interactions carried out by users on this kind of platforms, however, prevents experts from being able to manually inspect a sufficient amount of data in order to grasp a complete picture, motivating the need for the automation of information extraction. This thesis describes the infrastructure of a system able to collect and process data coming from social networks, producing quantitative outputs which can be exploited by semioticians in the context of the research project FACETS (Face Aesthetics in Contemporary E-Technological Societies). The input data, consisting mostly of profile pictures, are treated by means of several Computer Vision algorithms capable of extracting a specific kind of information. Some of these tools, though, are only able to deal with a given type of data and tend to fail or produce meaningless results when faced with unexpected inputs. As any kind of image could theoretically be received by the system, it is important for each model to be capable of autonomously rejecting undesired pictures. The rest of this thesis proposes several techniques to monitor the behaviour of neural networks when dealing with this problem, and evaluates some well-established solutions on custom benchmarks. Indeed, although most of the research works in this field (out-of-distribution detection) focus on detecting inputs coming from datasets different from the one the model was trained on, it can be argued that this is not sufficient to deem them undesired. The two datasets may show some levels of overlapping, either obvious (common categories) or hidden (common content). The proposed benchmarks take this into account by mixing different datasets and assigning a new binary label (in or out-of-distribution), using both manual and automatic techniques. The idea behind the automatic solution is to measure the distance between the concepts described by the class label of an image and the ones belonging to the target distribution. This is implemented by leveraging tools from the field of Language Processing, in particular the WordNet database and some metrics able to quantify the similarity among concepts. Experimental results show how the proposed setting is more complex than the typical ones, mainly due to the characteristics of the considered images, leading to a significant drop in performance with respect to simpler problems. Additionally, it is clear that different labelling strategies further affect the ability of a method to detect outliers, as taking into account the semantic content of each image becomes necessary. Evaluation results are used in order to select an appropriate technique and threshold which are later implemented in the processing architecture of the project, leading to an improvement in its out-of-distribution detection abilities.