Horizontal Autoscaling has become a fundamental feature of modern container orchestration systems, allowing the hosted services to scale dynamically following the demand, servicing it in the respect of the Service Level Agreements, while avoiding wasting resources with fixed or too generous allocations. In the industry, threshold-based autoscalers are still the standard, given their simplicity, but they do not always guarantee optimal performances nor real ease of configuration: as they operate reactively, they can be unstable in high variance environments, and alternatively require very conservative thresholds or risk arriving late in case of rapid demand growths. Given the complex nature of the context, Deep Reinforcement Learning techniques can be used both to automatize the policy tuning, both to discover new policies having more expressivity and better performances than those possible with threshold-based systems.