% testing

This section develops the testing phase of the implemented solution, it is focused on one of the most important and critical aspects regarding security services and, in general, any network services, the time.

The tests have been carried out in different scenarios to detect how the tool execution time change due to different parameters given in input. Scalability is a crucial characteristic to understand the usability of the framework. In particular tests' scope is focused on the evaluation of the optimal distribution model to highlight the best features and to understand the most critical parameter that effects the scalability. Moreover this model, as it is discussed in Section~\ref{sec:optimal_model}, has a great role in the inter-policy analysis, for this reason some comparison tests are done to compare the policy services exploited by the \textit{policytoollib} and the optimization model.

Four test cases with different metrics have been identified as critical and interesting for the scalability evaluation:

\begin{itemize}
	\item 
	number of entities hosted in the cloud environment;
	\item
	number of physical hosts which make the cloud infrastructure;
	\item
	number of virtual machines instantiated and running;
	\item
	number of rules to be enforced.
\end{itemize}

All the performance evaluations are done on a physical machines with the following software and hardware characteristics:

\begin{itemize}
	\item 
	\textbf{TYPE}: x64 architecture
	\item
	\textbf{CPU}: Intel(R) Core(TM) i7-4720HQ, Quad-core  @ 2.60GHz
	\item
	\textbf{RAM}: 2x8 GB, DDR3 @ 1600 MHz
	\item
	\textbf{OS}: Windows 10 
\end{itemize}

%\begin{figure}[H]
%	\centerline{\includegraphics[width=\textwidth]
%		{latex_files/pdf/iptables_rtt.pdf}}
%	\caption{Latency test report for Iptables}
%	\label{fig:iptables-rtt}
%\end{figure}

\section{Scalability test}

The purpose of the following tests is to progressively to increase one of the previous metrics, fixing the others and evaluate the execution time.

The tool tested has several components, from the input reading to the rule injection on the devices, the optimization process is the scope of these tests. This means the following results and measures refers to the execution time used to create the distribution model and to optimize it.

The goal is understand how the different inputs and scenarios, that the tool may face, affects on the scalability and usability of the system itself.
\clearpage

The scalability test is divided into three subsection:
\begin{itemize}
	\item 
	\textit{entities test}: it evaluates the first metric previous described;
	\item
	\textit{infrastructure test}: it evaluates the second and the third metrics, about physical host and virtual machines;
	\item
	\textit{rules test}: it measures the last metric, the one which evaluate the performances based on the number of filtering rules.
\end{itemize}



\subsection{Entities test}\label{subsubsec:entities_test}

This test evaluate the scalability of the optimization model when the number of entities increase from one entity to five hundred entities. 

The test is developed with three virtual machines instantiated for each entity, which defines five rules in its security policy and the cloud infrastructure is composed by three physical hosts.
The figure~\ref{fig:num_entities} shows that the execution time is almost linear and the documented values are very low, approximately from $\sim50ms$ to $\sim275ms$. 

\begin{figure}[H]
	\centerline{\includegraphics[width=1.1\textwidth]
		{latex_files/pdf/test/num_entities.pdf}}
	\caption{Scalability based on number of entities}
	\label{fig:num_entities}
\end{figure}

This test highlights excellent framework scalability as expressed in the number of organizations. This is an important result, because cloud infrastructure are extremely growing environment, in which each day new companies move their business in this solution. 
\pagebreak

\subsection{Infrastructure test}

The second test points to analyse the execution time under the cloud infrastructure point of view. 
The test wants to highlight:
\begin{itemize}
	\item 
	horizontal scalability, which relates to the number of physical hosts;
	\item
	vertical scalability, which refers to the number of virtual machines.
\end{itemize}

These two tests are developed fixing three entities, with three rules defined in their security policy.
The other two parameters expressed as number of virtual machines and number of physical hosts will change.\newline

\subsubsection{Horizontal scalability}

The first test is done thinking about horizontal scalability in which a datacenter presents from one server to five hundred of physical machines, which represents a middle size cluster. 

The trend displayed into the graph (Figure~\ref{fig:num_hosts}) presents a perfect scalability, because its trend is a straight line with a small slope. Performance degrades linearly with the increase of the number of physical hosts, starting from $\sim100ms$, it reaches $\sim600ms$ with a half thousand of machines.

\begin{figure}[hb]
	\centerline{\includegraphics[width=1.1\textwidth]
		{latex_files/pdf/test/num_hosts.pdf}}
	\caption{Scalability based on number of physical hosts}
	\label{fig:num_hosts}
\end{figure}

The trend of cloud clusters is to increase the number of physical servers that compose the cluster itself. The graph shows a ideal flexibility of the optimal distribution model which is feasible also of big datacenters with thousand and thousand of servers.

\subsubsection{Vertical scalability}

Otherwise the second test is about vertical scalability, that means the number of physical servers are fixed to three and the number of virtual instances for each entities increase from one to six hundred.
The picture depicted below (Figure~\ref{fig:num_vms}) displays that the scalability trend is a bit different from the previous ones. Its line it between a linear shape and an exponential shape, though it is not so evident.

\begin{figure}[H]
	\centerline{\includegraphics[width=1.1\textwidth]
		{latex_files/pdf/test/num_vm.pdf}}
	\caption{Scalability based on number of virtual machines}
	\label{fig:num_vms}
\end{figure}

Cloud solutions as OpenStack, the one used as testbed, are strongly based on the dynamism of the environment.
This means that to create and to destruct virtual machines are high frequent operations performed everyday.

\hfill

The figure shows a great efficiency if the number of virtual machines grow.
This result is meaningful for the scalability of the framework, because the test highlights that in a use-case scenario the optimization model is able to analyse in a limited amount of time and to locate rules also when the number of destination in the infrastructure is huge.


\clearpage

\subsection{Rules test}\label{subsubsec:rule_test}

The last scalability test wants to analyse the tool scalability based on the number of rules enforced.

It is developed enforcing an increasing number of rules, from one filtering rule to one thousand and five hundred rules. They are defined in such a way that the intra-policy and inter-policy analysis cannot delete any of those rules. Each filtering rule is a subset of the previous one, but it performs the opposite action, in this way they fall under the generalization anomaly.

The rules are injected in a cloud deployment made of three entities, three virtual machines for each entity and three physical hosts for the infrastructure.

The evaluation shown by the picture (Figure~\ref{fig:num_rules}) highlights that the intrinsic computational cost of the MaxSMT problem is exponential in function of the number of rules to be injected. The computational is not negligible, because at a hundred and five thousand rules, remembering that the rules are divided among three entities, the computational time is $\sim 625s$.
 
\begin{figure}[H]
	\centerline{\includegraphics[width=1.1\textwidth]
		{latex_files/pdf/test/num_rules.pdf}}
	\caption{Scalability based on number of rules}
	\label{fig:num_rules}
\end{figure}

This result is fundamental, because it highlights what is the most critical scalability aspect. The optimization model has increased its execution time of one order of magnitude compared to the computational time due to the previous tests.

It is also necessary to underline that in a real use case it unusual to have that huge number of rules for each entities, which cannot be optimized by the policy analysis.
\pagebreak

\section{Comparison test}

This section regards the comparison tests between the inter-policy and intra-policy analysis, done by the \textit{Policy Tool Library} and the distribution process developed by this tool.

First of all scalability tests are done on the inter-policy and intra-policy conflicts analysis process, the former present three entities and for each entity the number of rules change as in the previous ``Rules test''~\ref{subsubsec:rule_test}. The latter is following the structure of the ``Entities test''~\ref{subsubsec:entities_test} on which all the parameters are fixed and the number of entities change.

\begin{figure}[H]
	\centerline{\includegraphics[width=0.82\textwidth]
		{latex_files/pdf/test/num_rules_p.pdf}}
	\caption{Scalability based on number of rules}
	\label{fig:num_rules_policy}
\end{figure}

\begin{figure}[H]
	\centerline{\includegraphics[width=0.82\textwidth]
		{latex_files/pdf/test/num_entities_p.pdf}}
	\caption{Scalability based on number of entities}
	\label{fig:num_entities_policy}
\end{figure}


These two tests are developed to highlight the scalability characteristics of first, the intra-policy analysis and second, the inter-policy analysis.

The first graph (Figure~\ref{fig:num_rules_policy}) points out that a huge number of filtering rules in the same policy can be challenging for the inter-policy analysis process, because the exponential shape expresses the same trend of the ``Rules test'' developed for the optimization process.

Otherwise the second test shows that the inter-policy conflicts analysis has a great scalability, because it seem to be a logarithmic shape in function of the number of entities.

This result is significant in a cloud context, because there is more dynamism regarding the organizations than the number of rules which are defined by them.

\subsubsection{Ratio test}

This last chart has the goal to compare the execution time of the optimization process related to how the policy conflicts analysis impacts on the overall execution.

As the picture depicted below (Figure~\ref{fig:num_rules_comp}) displays if the number of rules is low, the policy services exploited from the \textit{policy tool library} has a significant weight on the computation cost, which it reduces its impact with the increase of the number of filtering rules.

\begin{figure}[H]
	\centerline{\includegraphics[width=1.1\textwidth]
		{latex_files/pdf/test/num_rules_comp.pdf}}
	\caption{Scalability based on number of rules}
	\label{fig:num_rules_comp}
\end{figure}

This result leads to consider that for a limited number of rules ($< 300$), which corresponds to $< 2s$ of execution time, the external library is useless, because it burdens $\sim 20 \%$ on the computation time. 
The same actions could be done by the optimization process. Otherwise when the number of rules increase until they reach $> 1200$ rules, it is displayed that the actions performed by the policy analysis are extremely efficient respect to the downstream process. 

The previous graph (Figure~\ref{fig:num_rules_policy}) shows that the external library takes $\sim 14s$ to compute one hundred and five thousand rules, which is a better result than the one get by the optimization process, which spends $\sim 600s$ for the same task.

\section{Use-case test}

The previous test deeply analyses the scalability of the framework and highlights what parameters are critical for the computational time.
One of the goal of this thesis was to be a practical work and not only has a theoretical approach, for this purpose this test is performed to show a\break real-case use of this tool to understand its feasibility and usability in a production environment.
The evaluation is developed following with the guide lines described at the beginning of this chapter, which means to developed a stress test based on different parameters that affect the complexity of the problem.

A little time slot was spent to retrieve informations about the big cloud organizations, their infrastructure and their policies.
The following test-bad was define for this performance test:
\begin{itemize}
	\item 
	\textit{number of entities}: from 50 to 150 organizations;
	\item
	\textit{number of physical servers}: from 10,000 to 20,000 units;
	\item
	\textit{number of virtual machines}: from 500 to 30,000 instances;
	\item
	\textit{number of security rules}: from 250 to 15,000 filtering rules.	
\end{itemize}
 
\begin{figure}[H]
	\centerline{\includegraphics[width=1\textwidth]
		{latex_files/pdf/test/real_case.pdf}}
	\caption{Scalability based on real case scenario}
	\label{fig:real_case}
\end{figure}

\section{Test conclusions}

The tests development highlights how the optimization process scales based on different inputs. The external integration of the policy library with the distribution framework gets excellent results, in term of performance and optimization. 

Both of them, the optimization process and the policy tool library, are tested independently and in particular the first optimization performed by the ``policy tool library'' is necessary each time a huge number of rules are defined in a single policy. This avoids the optimization model to explode in the definition of condition clauses for the MaxSMT problem, which gets a lot of time to be solved.

Then the framework and in particular the optimization model have great scale features in a flexible environment as represented by a cloud infrastructure. 
Linear scalability with a small slope is a perfect characteristic for those environment, in which the dynamism is the main pillar.

Finally the test phase marks the number of rules as the main parameter on which both the external library and the optimization model scale worst. In this context future analysis and improvements can be done.