Design and engineering of system for large-scale Internet Traffic Visualisation

Nowdays, implement intrusion detection solutions in network environments is always more important. One way, is by using an anomaly detection approach, which relies on the concept of anomaly as deviation from the "normal" behaviour. Traditional anomaly detection systems aim to detect any deviation in the traffic, while in this work we want to focus only on the most relevant ones. Moreover, some anomaly detection techniques could fail in cases like ports with a highly day-night effect in the traffic trend. So the proposed framework aims, in an automated way, to extract, aggregate and visualize meaningful features useful to spot anomalies inside big amount of data. These features are traffic volume, number of unique source IP addresses and number of unique destination IP addresses. We start with traces of real tcp network traffic previously captured, they are processed exploiting the big data approach and, in this way, the network features are aggregated in time series per destination port. The time series are stored in a proper database and then retrived to be visualized. Also, they are analyzed with a forecasting tool, that allows us to spot anomalies as changes in the trend and as dots falling outside the forecasted uncertainty interval. By applying the full process to our dataset, anomalies in the trends are reported. Once an anomaly is detected, it can be investigated to identify the threat.