Data Science for Information Security with Open Source technologies

In the context of the European Commission, a logging monitoring system based on open source technologies is designed and developed for two web applications, named PABS and SYSPER. The architecture is based on the ELK stack, which comprises Elasticsearch, Logstash and Kibana to parse, store, perform queries on the logs and visualize the results. In addition, two custom python-based modules are developed: a log retriever and a log analyzer. The former is used to pull the logs from the servers and to feed them into Logstash. The latter applies a number of anomaly detection techniques to monitor several metrics of the monitored applications. An LSTM based network and a regressive model analyze access logs to detect respectively short and long term anomalies regarding the system utilization. To facilitate the troubleshooting process, exceptions triggered within the application are clustered through DBSCAN; the number of exceptions raised from within the same group is analyzed for anomalies through Robust Z-Score. Finally, the same technique, the Robust Z-Score, is exploited to monitor access to sensitive data.