On the importance of Pseudo-Labeling for Generalized Zero-Shot Semantic Segmentation

Deep convolutional neural networks are one of the most powerful algorithms for understanding image content. Among their achievements, these networks have shown to be very effective in addressing the semantic segmentation problem, the goal of which is to assign the correct semantic label to each pixel in the image. However, having a full pixel-wise annotation to train these models in a supervised way is unlikely in the reality, both because of the burden of the pixel-wise annotation process and the high variety of existing concepts that cannot be captured by a single training set. To sidestep these issues, researchers have explored the Generalized Zero-Shot Learning (GZSL) scenario, that is building models able to assign pixels to both training classes (seen) and classes with no labeled pixels available at training time (unseen), using side-information capturing seen and unseen classes relationships in a given semantic space. Previous works address the task as a visual to semantic embedding problem by learning how to map pixels of seen classes to the semantic space. This has two main drawbacks: first, such models tend to assign pixels to one of the classes they know better, i.e. the seen classes; then, they also ignore that unseen classes can co-occur with seen ones in the training images. In this thesis, we argue, and experimentally demonstrate, that it is crucial to model this co-occurrence in order to effectively address the GZSL problem. Our idea is to capture useful latent information on unseen classes by supervising the model with self-produced pseudo-labels for unlabeled pixels. Motivated by this, (a) several strategies of hard and soft pseudo-labeling are proposed; (b) the impact of an additional entropy regularization term computed on the pseudo labels is analyzed; (c) objectness refinement methods are integrated to refine the pseudo-labels. We test our strategy on Pascal-VOC12 and COCO-stuff datasets, where it largely surpasses the state of the art, without requiring additional penalties to reduce the bias on seen classes.