Image anonymization is a challenging task that aims at removing all personally identifiable information to ensure the anonymity of the user who published the image while preserving the same semantic contents. The method proposed in this work is based on Latent Diffusion Models and exploits auxiliary networks to enable conditioning from different image-based modalities, which provides additional spatial information, without requiring to re-train the diffusion model. The approach involves two phases: real image decomposition and synthetic image reconstruction. The decomposition phase aims at extracting some non-sensitive representations from the original image using a set of task-specific pre-trained models and image processing algorithms.