Machine Learning-based Hug Pose Refinement for Avatar Interactions in VR

Virtual reality (VR) holds promises for simulating interpersonal interactions, offering avenues to build social circles for individuals who may lack the possibility to interact with others in the physical world. As technology advances, avatars can better and better model the human movement, allowing to have interactions that feel more real, even with few tracking devices as is the case of consumer VR setups which comprise of only the headset and the controllers. One area where little work is present in the state of the art concerns the reconstruction of movements when multiple avatars are interacting with each other. This thesis explores the recreation of non-verbal interpersonal interactions, in particular the simulation of hugs, using machine learning-based techniques. A state-of-the-art procedural inverse kinematics (IK) system named "FinalIK" was first selected; it generates the poses of a single avatar-based on user’s hand and head positions. This research leverages the FinalIK system and aims to refine its simulation of hugging interactions by integrating a neural network-based component. This component, which is the main innovation and main focus of this thesis, simultaneously adjusts the poses of two avatars to ensure seamless and artifact-free interactions. A specialized dataset of hug movements was collected using an “Optitrack” motion capture system, and was used to train the neural network. This dataset is the second contribution of this thesis, as most of the other motion datasets focus on the movement of one avatar and do not include cases of interaction between avatars, especially hugs. The effectiveness of the system is evaluated examining the error in joint angles and positions, which define the poses of the avatars engaged in the interactions. This approach has produced a good improvement in the fidelity of the reconstructed poses during hugs, reducing artifacts and collisions between the avatars. Envisioned applications of this research range from enriching embodiment in existing social VR platforms to facilitating meaningful interactions for individuals facing social isolation. However, further refinements and developments are crucial to integrate the neural network model into user-friendly application or platform that can be effectively used in practical scenarios. In conclusion, this thesis contributes to the advancement of human motion reconstruction by delving into the realm of virtual non-verbal communication and interpersonal touch, specifically hugs, paving the way for inclusive and immersive social experiences in virtual environments.