Cross-Modality Image Synthesis through Representation Alignment in Deep Generative Models

Cross-modality synthesis of medical images involves learning a robust latent representation that captures shared information across different imaging modalities. This task is complex due to the differences between modalities, which requires a model capable of understanding and transferring information across these domains. For instance, variations in magnetic field strength, coil configurations, and acquisition protocols can result in significant differences in image quality, resolution, and signal characteristics. To address this challenge, a novel cross-modality conditional generative model is proposed, which facilitates the synthesis of new modalities from existing ones. The model enforces a shared latent space by introducing priors designed to maximize the expected log-density in relation to the cross-modal variational representation. This allows the model to learn a latent space that generalizes across different modalities. The decoder portion of the network reconstructs the target modality by leveraging the latent representation obtained during the encoding process of the other modalities. The performance of the proposed approach is demonstrated through different experiments on diverse datasets. The first dataset consists of different magnetic resonance images (MRI) scans of prostates obtained in different centers while the second one consists of uncorrected and CT-corrected whole-body positron emission tomography (PET) images with fluorodeoxyglucose tracer. These experiments showcase the model's ability to generalize across various image distributions and modalities. Furthermore, comparisons with state-of-the-art methods illustrate that the proposed model not only achieves competitive performance on established benchmarks but also creates an expressive and meaningful latent space. This contributes to improved domain-adaptive medical image reconstruction, highlighting the potential of the model for diverse medical imaging applications.