System Architecture for Analyzing Application Logs in Automation Testing Environments

Technology keeps improving everyday. New functionalities are added, the ones already implemented are optimized. Devices are becoming more and more complex, not only in the design and development phases, but especially in the testing one. These trends are followed by multiple categories of products. The topics discussed in this thesis are related to software systems. There are entire teams whose job it is to verify that the product meets the requirements in terms of functionality and compliance with regulations. After the testing and validation phase, the product is really close to being placed on the market. Bugs and problems found after these phases can become really problematic for the company, in terms of money and resources. Moreover, depending on the application, they could be dangerous for people. That is why companies should not scrimp on the testing phase. Testing teams put lots of effort into the creation and the scheduling of test cases and, even more, in the analysis of the logs generated by the testing environment of the SUT (System Under Test). This thesis has been carried on within a larger project, with the purpose of reducing human effort in charge of the testing team. The goal is to provide a modular and flexible software application able to perform an analysis of the logs produced by the testing environment, combining them with data from the specifications and manuals of the system. A real industrial case of a complex system has been analyzed and a general and reusable architecture has been developed. Huge effort in scheduling test cases and analyzing the logs is required. The system under analysis needs more than a month to complete the execution of the test cases for all its functionalities, producing tens of GB of daily data. The proposed architecture tries to help the testing team in testing smarter and better. It is composed of four main blocks: two related to parsing information from specification and testing logs; the other two to combine the acquired data and perform some further analysis, also exploiting Artificial Intelligence. The idea consists in extracting data from formal specifications or usage information of the system, like in the case study, a CLI (Command Line Interface) user manual of the system. In the following steps, this data is processed and combined with information from the logs produced by the testing environment to provide a detailed analysis based on the customer’s needs. Lastly, daily reports are generated, with both historical information (related to all the logs processed by the program) and those related to the latest testing log file analyzed. The testing team will then use those reports to have useful indications on how to proceed with the testing activity.