Methods and Measures for bias detection in natural language processing: A study on word embeddings and masked models

The society in which we live is influenced by prejudices that discriminate against specific groups of the population. In recent years, the presence of these biases has been detected in the textual data used to train natural language processing algorithms. Thus, the tools based on these algorithms present biases that harm specific categories of people. In addition to causing harm to people affected by biases, these tools do not comply with the fundamental right to non-discrimination, which may result in legal action against the responsible companies and institutions that created them. To detect and characterize this type of bias in natural language processing tools, the scientific community has developed methods and metrics to detect and measure bias. In this thesis, we apply these methods to analyze two different types of tools used in natural language processing. The first tool is word embedding, which maps words to vector representations, obtained with a Word2Vec or GloVe model. The second tool is based on large BERT-type masked language models, trained with textual data. We apply bias detection methods to both word embedding and a model’s output sequences. We analyze their capabilities and limitations. We then propose novel evaluation to assess whether a metric measures other phenomena besides a possible bias. We propose a novel approach to assess whether a model of the BERT family can be effectively evaluated in terms of the biases it contains. In the analysis phase, we conducted experiments for bias detection and measurement using these two types of tools on Spanish texts. This study is of interest to the community since most existing assessments focus on evaluating bias in English texts due to the prevalence of models and resources in that language.