Bridging Realities: Evaluating Synchronous and Asynchronous Interactions with Cross-Reality Twins

Mixed realities, based on technologies like Augmented Reality (AR) and Virtual Reality (VR), are becoming increasingly accessible, enabling individuals to immerse themselves in different realities more easily. However, these technologies are not fully integrated into everyday life yet. Even when they are, not everyone may want full immersion, and some individuals may need to remain in the physical world. Additionally, practical constraints, such as having only one headset available for multiple people, may limit immersion to one person at a time, but this does not prevent users from necessitating to communicate with each other while immersed in different realities. This need can be satisfied thanks to Cross-Reality (CR), a concept which refers to interactions and transitions between different realities, such as the virtual and the real, i.e., physical world. Infact, when people exist in different realities, they need a way to interact across them. In CR literature, research has explored various aspects, including how users in different realities can interact with each other (multi-user CR). However, while most multi-user interaction studies focused on AR-VR, those involving users in the real world have not been explored as deeply. This thesis introduces a novel approach to deal with interaction in CR, by using everyday objects to facilitate real-time communication between individuals immersed in AR experiences and those in the physical world. The proposed method ensures seamless collaboration across different points on the Reality-Virtuality continuum, mimicking real-world interactions and preventing users from feeling excluded from the other’s experiences. At the base of the afore-mentioned approach there is the creation of digital replicas of physical objects to enable asymmetric interactions between virtual and real worlds. In light of this, it was necessary to explore the concept of Digital and Virtual Twins, leading to the introduction of the Cross-Reality (CR) Twin, an object able to be interacted with across different realities, helping users communicate and interact as if they were sharing a similar experience in the real world. The potential of CR Twins is demonstrated through three prototypes, i.e., Comic Book, Music Player and Checkerboard, designed to test various aspects such as collaboration (from loosely coupled to strictly coupled), bidirectional real-time data integration, and user experience. These prototypes were tested in both synchronous and asynchronous modes to compare the user experience when isolated in their own reality versus when a communication bridge connects them to other users, across different realities. A user study was conducted to assess whether these interaction modalities can provide an experience comparable to real-world interactions and whether users could envision using them in everyday scenarios. Feedback, gathered through questionnaires and interviews, was largely positive. Most participants found it useful to interact with others in different realities while engaging with everyday objects and envisioned using CR Twins in daily life, also offering suggestions about implementing this technology in other objects and scenarios. In conclusion, the study demonstrated the potential of CR Twins to bridge the gap of augmented and physical realities. Future work could expand the range of objects used, explore applications in areas like education and remote collaboration, and improve accessibility of CR in everyday use.