Comparative Analysis of Vector Databases for Real-Time Similarity Retrieval: Enhancing Large Language Model Performance

With the emergence of AI and big data, the dimensionality of data has increased significantly, necessitating more efficient methods for data storage and retrieval. Traditional databases are not well-suited for handling the high-dimensional and unstructured data typical of modern applications. Vector databases (VecDBs) have become essential for managing these vector representations, providing an effective solution for tasks that require rapid and accurate similarity searches. This thesis presents an analysis of vector databases (VecDBs) and their role in improving the performance of Large Language Models (LLMs) through real-time similarity retrieval. We developed a Retrieval-Augmented Generation (RAG) pipeline, which integrates LLMs with VecDBs to respond to queries about external documents, specifically PDFs. Our RAG pipeline augments LLMs by utilizing external knowledge stored in VecDBs, addressing issues such as outdated knowledge and hallucinations often found in LLMs. By embedding and indexing document data into high-dimensional vector representations, the system retrieves relevant information to enhance the accuracy of chatbot interactions. We tested three vector databases: Milvus, pgvector, and LanceDB, comparing their features and characteristics. This study aims to identify the strengths and limitations of each VecDB to understand their practical applications. The results provide insights into how VecDBs can improve LLM performance in real-time data retrieval tasks, offering a foundation for future research into optimizing the integration of LLMs and VecDBs for better data handling and knowledge extraction.