Sexism and misogyny detection in social media memes remains challenging due to the inherent complexity of multimodal sources and the possible disagreement among different annotators. Current state-of-the-art systems often perform unevenly across hard and soft evaluations, where hard settings involve binary sexism prediction whereas soft ones entail probabilistic estimation of the judgment of multiple annotators. We propose a unified approach that jointly models both label types by combining soft label learning under an ensemble strategy, in which two models are trained on distinct, class-unbalanced dataset partitions, with hard labels supervision on borderline cases. Our experiments demonstrate that the proposed approach outperforms state-of-the-art methods addressing both evaluation settings, underscoring the importance of integrating deterministic and probabilistic predictions in sexism and misogyny detection on multimodal data..