Speech-Text Cross-Modal Learning through Self-Attention Mechanisms

Speech, with its various elements like intonation and non-verbal vocalisations, is considered to be the earliest form of human language. However, existing systems for understanding spoken language mostly focus on the textual aspect, disregarding these additional components. Recent advancements in speech language modelling have enabled the development of speech-based language models called SpeechLMs. Nevertheless, text remains the primary mode of communication on the internet. Given this pretext, the objective of the thesis is to analyse the current state-of-the-art speech models and design a novel approach to combine the speech and text modalities, obtaining an architecture that is capable of leveraging the advantages of both. To do so, we adapt VisualBERT’s approach—a previous work that introduces a simple and flexible framework to model a vast range of vision-and-text tasks—for the speech and text modalities. Visual tokens obtained from patches of an image are replaced with audio tokens obtained from patches of the audio spectrogram computed on a speech sample, effectively replacing vision with speech. The concatenation of text and speech tokens is fed into a series of transformer layers that implicitly align tokens from the two modalities via self-attention. We name our model SpectroBERT, and through experiments carried out on common speech-text multimodal tasks such as Audio Question Answering (AQA) and Speech Emotion Recognition (SER), we demonstrate that SpectroBERT is able to implicitly align text and speech features, while retaining the simple and flexible formulation of its predecessor.