Speech, with its various elements like intonation and non-verbal vocalisations, is considered to be the earliest form of human language. However, existing systems for understanding spoken language mostly focus on the textual aspect, disregarding these additional components. Recent advancements in speech language modelling have enabled the development of speech-based language models called SpeechLMs. Nevertheless, text remains the primary mode of communication on the internet. Given this pretext, the objective of the thesis is to analyse the current state-of-the-art speech models and design a novel approach to combine the speech and text modalities, obtaining an architecture that is capable of leveraging the advantages of both.