Feature aware image sampling for CNN-based image quality enhancement algorithms

In classical machine learning algorithms, for example those used for classification tasks, a common issue arises when dealing with an unbalanced training dataset. When certain classes are under-represented, it can result in poor performance not only on those classes but also on average. This issue also occurs in Image Enhancement algorithms, where some types of images may be under-represented in the training dataset. Consequently, the performance on these types of images tends to be lower. The most common solution to deal with an unbalance dataset in classical machine learn- ing is to resample the dataset, in order to have a balanced representation of all classes. In the case of Image Enhancement, however, we usually do not have classes that can be used to quantify how unbalanced the dataset is and to resample the training dataset. In this work, we try to solve this problem by using the in-house unsupervised clustering algorithm, CARDIE, which creates classes of images based on their luminance and hue distribution. Moreover, we perform an analysis on the relevance of the clusters in relation to the Image Enhancement algorithm we are using. Based on these classes, we experiment with different techniques of resampling and data augmentation and show how the performance changes depending on the composition of the training dataset. In our experiments, we focus on two different Image Enhancement tasks, Tone Mapping and Denoising, first studying the literature and then taking one state-of-the-art algorithm for each (respectively, HDRNet and NAFNet) which we use as baseline for the resampling tests. For what concerns the Denoising task, we propose an alternative to CARDIE based on the quantification of the texture in the images, and study how the performance changes with respect to the clusters based on this feature. We show improvements on the average performance on both tasks, which we can confirm both with the PSNR metric and visually, by looking at some sample images.