Performance prediction for secure software: analysing the impact of obfuscation

A frequent requirement for countering Man-At-The-End (MATE) attacks is protecting software that runs on end-user hardware controlled by attackers. Software obfuscation can safeguard intellectual property and prevent malicious product tampering, but it comes at a performance cost. This thesis analyses the critical challenges of predicting the performance overhead associated with obfuscation techniques commonly used in real world software, based on well-established literature. As patterns associated with these techniques are known, the performance impact implied in their usage is heavily dependent on the specifics of the protected software. An informed decision must be taken by the company or the developer aiming at protecting their own software, considering the changes in user experience associated with increased hardware resource utilization. This thesis proposes an automated approach to obfuscation, characterization, and performance data collection based on Tigress, an open-source C obfuscator. It is utilised to construct a dataset of real-world overhead data associated with each selected obfuscation technique and the program on which it was applied. It addresses the challenges of performance profiling on modern hardware to obtain high quality data, which are then used as input of a machine learning algorithm. On limited hardware resources, it prioritizes a closer-to-hardware profiling technique over tracing efforts, resulting in higher performances and higher-quality data. As program characterization is essential for the success of the proposed approach, static software metrics are collected on both source code and the compiler’s assembled output, keeping track of real-world compiler optimization and how source code is effectively translated into assembly code. At runtime, the collection of dynamic metrics is proven to be crucial for real characterization of the tested application and associated obfuscation technique. The predicted behaviour of certain obfuscation techniques is correlated with their practical behaviour on modern hardware, and this heavily influences the prediction efforts and the necessary descriptive metrics selection. The obtained dataset is utilised to train and validate a machine learning model, which has been demonstrated to predict the overhead of tested techniques with sufficient accuracy. Furthermore, this work presents a positive outlook on the development of increasingly accurate performance prediction techniques, and can thus serve as a fundamental starting point for future works.