Continuous integration for End-to-End testing of mobile applications

In recent years, the mobile industry has grown exponentially, made up of millions of applications and developers, and billions of devices and users. The rapidly evolving hardware and software platforms, in conjunction with their variety, make the testing phase a complex and difficult task. This often causes companies to neglect it, due to the big amount of time and resources that it requires from developers. This thesis proposes some solutions to integrate automatic GUI end-to-end testing in a continuous integration pipeline (CI), in order to automate, facilitate and speed up some of the tests that mobile software development requires. The thesis work started with an analysis of the state of the art of automated testing frameworks, in particular for mobile applications, that could be used in a CI pipeline. From those, Android Espresso, Appium and SikuliX have been selected. The tools have been chosen because they represent different test methodologies (respectively white, grey and black box GUI testing), based on different abstraction levels of the application being tested. To evaluate each tool the application OmniNotes has been chosen, because it is open source and the code available on the GitHub repository can easily be run in a local environment. The thesis work continued with the analysis of the tools for building and managing CI pipelines, in particular, Jenkins and GitHub Actions. The automated testing frameworks have been implemented on the GitHub Actions pipeline, each one separately. Two separate experiments were run to evaluate the effectiveness of the CI approaches developed. First, we evaluated the resilience to fragmentation of each tool, i.e., the capability of running the tests against multiple devices with different characteristics. Then, we evaluated the flakiness of test cases launched in the CI pipeline, i.e. the ratio of test cases that run until the end without execution issues. We measured good results for fragmentation for all tools, whilst Appium and SikuliX had higher flakiness than Espresso. The overall result looks promising tough: the solutions are easy to set up and to work with. Also, if the developers are already using a GitHub repository, it is possible to have everything under a single ecosystem. Future works may also implement more articulated solutions using Jenkins or may try to improve the stability of SikuliX.