Instance Segmentation of Giemsa and Immunohistochemical Lymphoma Cells: Towards AI-Enabled Virtual Flow Cytometry

In lymphoma diagnostics, flow cytometry is widely used for cell counting and sorting, providing essential insights into individual cell characteristics; however, it is limited by its single-cell analysis approach, which lacks spatial and morphological context. On the other hand, the manual analysis of whole-slide images (WSIs) from scanned slides is highly time-consuming and prone to both intra- and inter-operator variability. This thesis project proposes a tool for the rapid and automated delineation of cell nuclei in WSIs, with the goal of not only minimising diagnostic time but also reducing variability through the repeatable nature of automated analysis. Traditionally, Haematoxylin and Eosin (H&E) staining is among the most used methods in this field, with extensive related instance segmentation research in literature. Although cytological stains like Giemsa and immunohistochemical (IHC) markers offer complementary information not visible in H&E, their automatic segmentation has been less explored in the literature. Giemsa staining, selected for its high-quality delineation of chromatin and nuclear membranes, proves advantageous in distinguishing various cell populations within the tissue. This is the first work to address instance segmentation in Giemsa-stained WSIs quantitively validated. Integrating the segmentation of nuclei stained with various IHC markers enables the identification and classification of nuclei as immunopositive or immunonegative, whose presence and combinations are subsequently evaluated by the clinician to make a diagnosis. To achieve precise instance segmentation and classification of cell nuclei in Giemsa- and IHC-stained WSIs from lymphoma patients, provided by the A.O.U. Città Della Salute e Della Scienza Hospital in Turin and publicly available images such as the Multi-Organ Nuclei Segmentation (MoNuSeg) dataset, this research employs and modifies HoVerNet, a robust deep learning model for nuclear instance segmentation. HoVerNet’s modification to incorporate border-class prediction enhances its ability to separate closely positioned or overlapping nuclei, a common challenge in instance segmentation tasks. A key strength of this work lies in using a single architecture – HoVerNet – modified and optimized to manage different classification requirements for diverse segmentation goals. A comparison was also conducted with another state-of-the-art instance segmentation architecture, the Centripetal Direction Network (CDNet). Multiple inference and post-processing strategies were explored to optimize segmentation performance, which was assessed quantitatively through pixel-based metrics (e.g., Dice Score) and object-based metrics (e.g., Aggregated Jaccard Index), as well as qualitatively through consultations with a pathologist. Accurate segmentation of cell nuclei enables cell-by-cell analysis of morphology, texture, and colour intensity to determine marker positivity, directly informing pathologists about the tissue’s pathological status. Validation against ground-truth annotations confirms that this framework not only surpasses current state-of-the-art deep learning models in accuracy but also offers a viable alternative to flow cytometry by enabling multiplexed analysis of differently stained cellular populations within their spatial context. These findings support the development of an AI-driven diagnostic tool that enhances the speed and accuracy of lymphoma diagnostics and streamlines the histopathological workflow.