Deep learning method for prostate cancer segmentation and quantification in immunohistochemical staining

The prostate is formed of different types of cells, and each of them can mutate and become cancerous. Prostate cancer (PCa), the most common male neoplasia, is estimated to affect about 36,074 people in Italy every year, according to 2020’s statistics. Recent studies show how 1 fifty years old man out of 4 has PCa cells, whereas at the age of eighty this condition affects 1 out of 2. Although widespread, PCa does not always present as an aggressive form, but commonly manifests itself in a benign form, such as prostatic hyperplasia. Therefore, it is essential to estimate the correct tumor grade (i.e., Gleason Score) because it is directly related to the intervention strategy adopted which can range from radiotherapy and radical prostatectomy to active surveillance in a slow tumor growth case. Nowadays, the PCa diagnosis is performed manually by the pathologist through a histological analysis in H&E of prostate tissue removed through agobiopsy from the patient. Considering that the diagnosis is based on a purely morphological biopsy reading, this method of analysis is time-consuming and subject to inter- and intra-operator variability in the classification. This thesis work aims to introduce an immunohistochemical analysis of the tissue together with artificial intelligence techniques to create an automatic system for the identification and classification of the prostate glands. The purpose is to develop a support algorithm for the pathologist that allows them to speed up the diagnosis process and, at the same time, provide a first completely objective classification of the lesion. The dataset consists in 32 whole slide images (WSIs) stained with p63 and racemase markers, and the respective 36,894 glandular annotations, which was used to train deep learning networks. The results clearly demonstrate the power of deep learning approach in the semantic segmentation of prostate glands and the robustness of K-Net even in the presence of a high imbalance between classes. The algorithm achieved a Dice similarity coefficient of 95.8%, 77.1%, and 92.9% for the discrimination of benign glands, prostatic intraepithelial neoplasia (PIN), and invasive cancer respectively. A further development of the thesis aims to demonstrate the feasibility of transforming H&E images into their respective images in immunohistochemical staining, using a Pix2Pix model. The dataset consists in 31 paired WSIs, stained with H&E, and with p63 and racemase. The application of this generative adversarial network (GAN) was also possible with subsequent tissue slices, thus with paired non-overlapping slides, thanks to the introduction of a customized algorithm for image registration that uses a combination of rigid translations and an optimal flow technique for punctual deformation of the glands.