Deep Reinforcement Learning for Horizontal Autoscaling: a Proof of Concept

Horizontal Autoscaling has become a fundamental feature of modern container orchestration systems, allowing the hosted services to scale dynamically following the demand, servicing it in the respect of the Service Level Agreements, while avoiding wasting resources with fixed or too generous allocations. In the industry, threshold-based autoscalers are still the standard, given their simplicity, but they do not always guarantee optimal performances nor real ease of configuration: as they operate reactively, they can be unstable in high variance environments, and alternatively require very conservative thresholds or risk arriving late in case of rapid demand growths. Given the complex nature of the context, Deep Reinforcement Learning techniques can be used both to automatize the policy tuning, both to discover new policies having more expressivity and better performances than those possible with threshold-based systems. After the definition of an exemplificatory target environment and the formalization of the autoscaling problem as a Markov Decision Process, several possible approaches are hereby evaluated, comparing their performance with tuned threshold-based systems. The developed final methodologies outperform all the tested threshold-based configurations in the considered environments, presenting better SLA violations/used resources ratios and decreasing the severity of SLA violations. As the training procedure shows however considerable caveats over its real-world applicability, the discussed approach is expanded to include Meta Reinforcement Learning principles, showing promising preliminary results that could indicate the effective possibility of, upon further developments, industry-level applications.