A Novel YOLOP-Based Framework for Panoptic Driving Perception in ADAS Applications

This thesis addresses two main limitations of the default YOLOP model for autonomous driving: - The primary one being the fact that there is no direct detection of lane center, and the secondary being- no depth or distance estimation in the object detection head. While the baseline YOLOP architecture can perform lane line detection, drivable area segmentation, and object detection at 40 FPS in an efficient manner, post- detection output steps—such as finding regions of interest, OpenCV based transformations, and curve fitting for determining the lane center from the detected lane lines—introduce significant latency (0.04 seconds), reducing the effective frame rate to 15 FPS. Additionally, the bounding box outputs from the object detection head lack depth estimates, limiting their use in any control related algorithm design. To go beyond these constraints, we introduce an improved YOLOP model with two additional outputs: (1) direct lane center detection, eliminating the need for external post-processing, and (2) object distance estimation, enabling the usage of the object detection head as a virtual depth sensor. With these 2 extra outputs, a controller which can do both longitudinal and lateral control at the same time can be developed from a single model. We created two custom datasets to train the modified model: a 13,000-image dataset of straight roads with varying lane types and environments, and a 50,000-image dataset of straight and curved roads. The lane center head was trained with custom loss functions and evaluated on accuracy and Intersection-over-Union (IoU) metrics. The modified object detection head includes a regression output for distance, with an accompanying loss term. Experimental results show lane center detection accuracy of 95% with Intersection over Union scores up to 60%, and for the distance prediction part, the distance estimation error was found to be 4%. Importantly, original lane line detection performance was maintained (75% accuracy, 60% IoU), verifying that new outputs did not come at the cost of existing capabilities. Compared to the YOLOP + OpenCV pipeline, our model exhibited a significantly lower lane center offset and an inference speed of almost 40 FPS on an NVIDIA RTX 3080 system.