Building a honeypot to mitigate bad bot traffic

Nowadays, scraping information on the Internet is a practice used by different players for various goals. This automated collection of data is performed by automated scripts, called bots, which have become more and more advanced during the years. Providing information has a cost, thus, even if scraping is not illegal by itself, it can cause problems to websites owners. A company of the travel technology sector is facing an excess of bot traffic targeting its booking domains: as a matter of fact, legitimate traffic is only 10\% of the overall volume. Every booking search has a high cost because fares are calculated in real-time taking into account a large number of parameters. The company is forced to bill its customers for this excessive traffic and inevitably clients' profit is reduced. During the years, the company has taken advantage of the most up to date solutions to solve this situation. However, current mitigation techniques are not being effective because of the technological advancement of bots. They can adapt to countermeasures in short amounts of time and they are becoming increasingly efficient in bypassing detection systems. Therefore, fighting against such advanced threats involves manual work rather than relying on automated systems. In this work we address this issue introducing a novel approach. Adopting the concept of a honeypot, a bait system in which attacks are collected and studied, we have implemented an infrastructure to capture the bot traffic. We serve alternative but plausible content to bots. This breaks their feedback mechanism because they do not receive a direct confirmation they have been detected. We have tested the structure in a controlled and monitored environment, then we have performed experiments with real bot traffic. At the same time, bot requests have been collected and studied. From the 12,801 collected requests of one of the experiments, we were able to infer theories about the bot adaptation techniques and the distribution of the workload among different botnets.