Breaking the Challenge of Smart Microservice Autoscaling through Coordination

In recent years, microservice-based architectures (MSA) have gathered con- siderable attention for their potential to revolutionize the design and deployment of large-scale applications. This model promotes flexible, loosely coupled, and finely engineered software, making them easier to manage and facilitating DevOps practices. A critical aspect of MSA is the efficient scaling of microservices while ef- fectively managing resource allocation in the midst of increasing load intensity. Scaling may be applied vertically, by adding more resources to individual mi- croservices, horizontally, by instantiating additional instances of congested mi- croservices, or both at the same time. Recent works have investigated machine learning techniques and, in particular, reinforcement learning (RL) [1] [2], [3] to enhance scaling mechanisms. Taking into account both basic metrics such as CPU and memory usage, as well as higher-level metrics such as end-to-end latencies, these approaches strive to reflect the execution dynamics of microservice applica- tions. However, they predominantly ignore microservice interactions and consider the scaling of a microservice in an isolated manner. In contrast, we propose to exploit information about microservice dependencies and to provide coordinated scaling within groups of microservices. The way we approach the problem is through the design and usage of a soft- ware, called "Grouped Scaler Operator", orchestrating scaling actions among mi- croservice groups, by telling each scaler in a group what to do at a given instant, like scaling in/out or up/down. Our research focuses on the technical feasibility of this task, delving into the world of Kubernetes and its native implementation of controllers, called Operators. One other objective we have is to investigate the applicability and to design a methodology of putting intelligence into coordination, mainly through reinforcement learning, since it can recognise and learn complex patterns present in microservices interactions and then optimise scaling looking at them in such dynamic environments. To achieve this, various tools have been used, namely Kubernetes [4] for clus- ters, Kopf Operator Framework [5] for the scaling controller and Autoscalers [6] for scaling. For the metrics, Prometheus [7] and Istio [8] have been used. For the Reinforcement Learning part, PyTorch and various OpenAI API libraries [9][10] have been used, as long as monitoring tools such as Tensorboard[11] and Weights and Biases [12]. The deployments and tests have been carried out on the Grid’5000 national testbed [13] and on a private cluster provided by Orange-EOLAS as a part of the SCALER project [14], in sight of which this project is carried out.