How to Measure Game Testing: a Survey of Coverage Metrics and an Implementation on the iv4XR Framework

With the increase in their popularity, video games have evolved to become highly complex products that require ever larger funds and huge, diverse teams to be developed. Due to the growing complexity of the software, however, video games are highly prone to bugs, often appearing on day one and sometimes in such large quantities that they cause release delays and negatively impact the product's economic success. To mitigate this issue, it is essential to conduct a structured testing phase to assess and evaluate the software's quality. In the video game industry, automated testing using autonomous agents holds particular potential. These tools are trained with artificial intelligence to explore the game environment and test various gameplay modes and situations by dynamically changing their strategies. However, the literature includes few studies on the application of automated testing to video games, and there is a complete lack of attempts to classify and standardize specific coverage metrics for video games. This gap makes very difficult to assess the quality of testing and compare results. Hence the motivation for this work, which consists of two objectives: a first phase of literature review aimed at identifying and classifying specific coverage metrics for video games, providing a foundation for building testing models that address the core aspects of a video game; and a second implementation phase, during which several gameplay coverage metrics were implemented using iv4XR, a promising open-source framework that employs autonomous agents for testing Extended Reality systems. The results of the first phase is a taxonomy of 26 specific metrics employed in video game testing, grouped into six categories depending to the domain to which they relate. In the second phase, some of these metrics were implemented in iv4XR using their designated testing game, LabRecruits, and evaluated on both newly written test cases and demo tests provided by the developers. From the results of this second phase, it becomes clear that currently it is not yet possible to build a general coverage model that is applicable to every video game, due to limitations arising from the specificities that differentiate video games and from existing testing frameworks, which are often closely tied to the specific game or require, at the very least, the development of a specific interface between the framework and the game to be tested (as in the case of iv4XR). Future work could focus on reducing these limitations by using automated testing to efficiently test as many video games as possible, leveraging and expanding the proposed taxonomy of metrics, developing tools that natively implement said taxonomy and test environments that are increasingly independent from specific games.