Unlocking the potential of Large Language Models in insurance domain: a case study

Large language models (LLMs) have emerged as powerful tools in natural language processing showing outstanding performance across a spectrum of tasks and garnering considerable attention in research. Their capacity to generate human-like language and potential to reshape science and technology have encouraged their exploration across various industries, including insurance. The insurance sector faces the challenge of managing vast volumes of data from both internal and external sources and therefore formulating effective strategies for data management becomes crucial for maintaining competitiveness in an evolving marketplace. This thesis investigates the application of LLMs in classification and information extraction tasks within the context of complaints management. Specifically, the study aims to develop a system capable of autonomously extracting and interpreting information from customer complaints received via email. Leveraging two pre-trained open-source language model, Zephyr-7b-beta and Llama-70b, the research employs two training approaches to align the model with domain-specific knowledge. The first approach utilizes in-context zero-shot learning, enabling the model to perform targeted behaviors, without extensive retraining, trough configuring prompt statements. The second approach relies on Parameter Efficient Fine-Tuning (PEFT) for which the training phase involves adjusting the parameters of the last layer through a supervised approach. Starting from raw data and after extensive data analysis and pre-processing, a series of experiments were conducted on a test set comprising 1321 data points, selected through a stratified split. The models and two methods were evaluated using the Semantic Answer Similarity score metric. The findings demonstrate the positive impact of implementing LLMs in the complaints process, with fine-tuned model outperforming zero-shot learning approach for both classification and information extraction tasks leading to model accuracy of 80%. Through these methodologies, this work seeks to enhance the efficiency and accuracy of complaints management processes in the insurance industry as it places the ground for a further exploration that can be used to assess the eventual benefits of implementing the large language models to other case studies.