Multimodal 3D photoacoustic and contrast-enhanced magnetic resonance breast image registration using coordinate-based neural network: a preliminary investigation

Breast cancer (BC) represents a significant health challenge, being the leading cause of cancer mortality among women in Europe [1]. Various imaging techniques include mammography, magnetic resonance imaging (MRI), and ultrasonography are used for early and accurate diagnosis of the disease. Mammography, though widely used in screening programmes, has high false positive rates, is less effective in women with dense breast tissue, and uses ionizing radiation [1]. MR offers high sensitivity in breast cancer detection but is limited by poor specificity and high cost [1]. Ultrasound serves as an additional tool next to x-ray imaging and often is the only tool in imaging for pregnant and breastfeeding women [1]. Recently, photoacustic (PA) imaging has shown to be a good candidate imaging technique for BC due to its low-cost, the absence of ionizing radiations and being contrast agent-free [2]. However, being a relatively new technology, one of the main problems is the difficult interpretation of the images. In this regard, comparison with conventional imaging techniques is fundamental to identify and explain PA imaging biomarkers. For this reason, the goal of this thesis is to investigate the feasibility of a 3D image registration framework to align PA and contrast-enhanced MR images of the breast. In this research, two fundamental aspects associated with the challenges of such a registration problem will be investigated: the multimodal nature of this registration problem, namely the different imaging contrasts between the two modalities, and the complex non-rigid deformation due to different imaging examination settings. A novel registration algorithm based on coordinate-based neural networks (CBNNs) was adopted in this study. CBNNs use neural networks to directly represent the displacement field between the images. The first chapter provides an overview of the clinical background, the image registration problem, and the use of CBNN. The second chapter investigates the use of CBNN on a simplified 2D dataset to develop and test a set of differentiable similarity metrics for the registration algorithm. Mean Square Error, Normalized Cross-Correlation, and Normalized Mutual Information (NMI) were implemented, and experiments show that NMI offers the best performance in the multimodal case. Subsequently, a unimodal analysis was conducted between PA volunteer images acquired with the protocol of the photoacoustic-ultrasound hybrid system Photoacoustic Mammoscope 3 (PAM3) using a breast cup and PA images acquired without cup. Results show that the use of manually annotated landmarks and an initial affine pre-registration via Elastix before the CBNN deformable registration improves the alignment of largely deformed images. The last chapter presents results from the initial attempts to align PA and contrast-enhanced MR images. A modified algorithm was proposed, embedding PA images and Speed of Sound (SOS) as reference images and pre- and post-contrast enhancement T1 as moving images. The algorithm was tested on images acquired via PAM3 for PA and SOS and on MRI of the same patient. Although final results show sub-optimal alignment between the two modalities, this study provides a solid basis for future studies. [1] JAFARI, Seyed Hamed, et al. Breast cancer diagnosis: Imaging techniques and biochemical markers. Journal of cellular physiology, 2018, 233.7: 5200-5213 [2] MANOHAR, Srirang; DANTUMA, Maura. Current and future trends in photoacoustic breast imaging. Photoacoustics, 2019, 16: 100134