Real-Time Latency Mitigation for SAE Level 4 Remote Driving Using Predictive Image Transformation and Vehicle State Estimation

Remote operations have received a lot of attention in recent years. In SAE level 4, autonomous vehicles are able to operate in predefined situations without human intervention. However, they still has limitations in complex scenarios like construction zones, different weather conditions and poorly mapped areas. In such case, human driver intervention is necessary for safe operation. Teleoperation is a potential alternative for remotely operating vehicles when autonomous vehicles require a human hand. Teleoperation makes the transition to full autonomy smoother and safer (SAE Level 5). Teleoperation has inevitable latency due to the communication systems, which can degrade the driving performance, increase cognitive load on remote driver. There are delays in the system while transmitting driver commands to vehicle and in receiving feedback, like video, vehicle states. Video feedback plays an important role in providing situational awareness, allowing the driver to make control decisions accurately. Since the driver is not responding to the current state of the surroundings, latency in video transmission can cause input misjudgements. The delayed video feed can cause serious safety problems like oscillatory steering input, vehicle instability, passenger discomfort, and even lane deviations that may lead to collisions. This thesis aims to develop a predictive display system to mitigate the problems caused by delay while receiving video feedback from vehicle and address the depth noises which affects the image reprojection. A remote driver perception pipeline was developed using the OpenCV library which is deployed at the remote station to generate latency compensated video. Simulink 3D simulation toolbox is used for simulating the system, simulation 3D camera is used to generate the RGB and depth images. Predictive display system has two components: depth correction and image transformation. Depth correction algorithm’s aims to correct the depth noises caused by lens distortions and environmental factors, such as transparent/reflective surfaces of vehicles. This step improves the accuracy of image reprojection. Smith predictor is implemented at the remote station to estimate the pose based on control inputs and known communication latency. Specifically, the image transformation function uses the pinhole camera equations for back-projection to generate the point-cloud of pixels. This point cloud is then rotated and translated using the predicted pose from the Smith Predictor. Finally, remapping and interpolating the RGB pixels with new pixel indices derived from the transformed point cloud, generates a predicted view. The predictive display is validated using a LQR based Lane-centering controller. LQR controller is use to mimic the human driver, eliminating inconsistent input by ensuring the unbiased and repeatable evaluation of the system. A comprehensive Simulink model was developed to simulate the behaviour of a remote vehicle and a remote station, integrated with predictive display algorithm. LQR controller results demonstrate minimised trajectory deviations with predictive display, reducing overall RMS lateral error from 0.5 to 0.07m and psi value from 4.1 to 1 degree. Similarly, experiments involving human driver shows significant improvement in system responsiveness, reduced trajectory deviations and oscillatory driver inputs. The use of predictive display minimises lateral oscillations and lane departure at higher speed, thereby enhancing both passenger safety and ride comfort.