An efficient method to label social media images through deep clustering

In recent years, Computer Vision has been experiencing a strong development thanks to Convolutional Neural Networks (CNNs). Models trained in a supervised way on large datasets such as ImageNet have achieved state-of-the-art performance in several tasks. But, the potential of supervised training is limited by the need to have the images annotated within the datasets, which is a very expensive process. This problem is even more pronounced in scenarios where the type of images and thus the labels to be assigned to them are not known in advance. This thesis aims to find a way to annotate unlabelled datasets efficiently by using a cluster-based rather than an instance-based type of annotation. Recent self-supervised learning techniques have been exploited to train models without the need for annotation and thus clustering algorithms are applied on the embeddings obtained from such models to cluster images. The thesis work is part of the FACETS (Face Aesthetics in Contemporary E-Technological Societies) research project, which will involve the collection of images from users’ social profiles. As there is no information on these images, a tool capable of clustering them will be needed in order to choose labels and then annotate them. Since FACETS data collection is not completed, the dataset SocialProfilePictures (SPP) was created to simulate a possible data collection. The dataset is annotated and thus provides a way to validate the proposed procedure. In the experiments, the potential of self-supervised training was evaluated in comparison to the use of transfer learning. What emerged was that for self-supervised training it is necessary to have a large amount of data in order to compete with pre-trained models on large labelled datasets, so in small data scenarios, it was necessary to exploit transfer learning. As for the clustering part, it turned out that better performance is obtained with fewer clusters than the effective number of classes, so clustering is better if done by macro-classes. Finally, a methodology was proposed in order to carry out the tuning of clustering in a completely unsupervised manner and initial interactive use of clustering exploration was implemented, which can be exploited later on for cluster-based annotation.