Enabling Unsupervised Domain Adaptation in Semantic Segmentation from Deep to Lightweight Model

Semantic segmentation is the task of detecting and classifying each pixel of an image with a semantic class (i.e. road, sidewalk, pedestrian, etc). It is a powerful tool for various applications, ranging from robotics to aerial image analysis, but its primary application is in autonomous driving, where recognizing the entire scene is critical to making the best driving decision. One of the major challenges with semantic segmentation is the shortage of large pixel-by-pixel annotated datasets, which are costly in terms of both human and financial resources. A common approach is to train models using synthetic datasets composed of simulated images and pixel-accurate ground truth labels, and then utilize Domain Adaptation (DA) approaches to bridge the gap between such artificial domains (referred to as source) and the real ones (to which we refer as target). Deep learning networks are nowadays built on the notion of "infinite resources", resulting in larger and more accurate deep models (in terms of learnable parameters), but at the expense of efficiency. This model complexity affects both the inference time and the hardware requirements. Furthermore, data can be confidential or secret, reason why some companies opt to supply only a pretrained model without data access. With this in mind, we suggest extending existing DA techniques in order to harness the fine-grained information of these complex and deep networks while training efficient and lighter models on new domains. We present a benchmark in which we examine the ability of three traditional DA settings to transfer knowledge from a Deep to a Lightweight model: (i) Adversarial, (ii) Self-Learning (SSL) and (iii) the combination of Adversarial+SSL. Given the promising results obtained in the Self-Learning setting, we focus on existing SSL-based DA techniques to further improve the performance of the lightweight network. We extend the Cross-Domain mixing technique proposed in the DAFormer paper by introducing: (i) a novel cross-domain mixing via instance selection, (ii) a dynamic weighted segmentation loss and (iii) a dynamic mixing based on the lightweight model's per-class confidence. All of the experiments were carried out by applying the two conventional synthetic-to-real protocols: (i) Synthia to Cityscapes and (ii) GTA5 to Cityscapes.