Cloud-native Kubernetes application to efficiently and securely stream and collect real-time data

Properly collecting and distributing data has become more and more important in recent years. The vast majority of companies, universities, research groups, and other similar entities need to manage some kind of data. For this reason, it is really important to have a system capable of suitably performing this task. The objective of this thesis is to design, develop and test such a system. More specifically, the intent is to build a product that can be easily adapted to most of the applications in which is important to gather and distribute big amounts of data in real-time. Nevertheless, in order to study a real use case with actual requirements, it has been developed a system capable of efficiently and securely collecting and streaming real-time data representing electromagnetic field (EMF) measurements, which allows human users to study, both utilizing real-time and historical data, the cellular coverage in California. To tackle this problem, a Microservices architecture has been chosen. This architecture was selected because it is well suited for the goal of having a scalable, flexible, and easily extensible distributed system. Kubernetes has been utilized as the containers orchestrator and the entirety of the cluster has been hosted on Google Cloud. The data is streamed and stored using Apache Kafka and MongoDB, respectively, while the security of the system is managed with the use of Anthos Service Mesh. The final product is able to properly and flawlessly process the necessary requests to suitably analyze the cellular coverage of the whole of California. In addition, it does so without overpaying for unused resources that have been carefully allocated to specifically meet the requirements.