End-to-end autonomous driving model based on multi-modal intermediate representations

doi:10.16731/j.cnki.1671-3133.2024.03.010

Abstract

Abstract: An accurate understanding of the driving environment is one of the prerequisites for autonomous driving.In order to improve the scene understanding ability of autonomous driving vehicles,an end-to-end autonomous driving model based on semantic segmentation,horizontal disparity,and angle coding multi-modal intermediate representations was proposed.The end-to-end autonomous driving model used deep learning technology to build perception-planning network.The perception network generated multi-modal intermediate representations with RGB and depth images as inputs to realize the spatial distribution description of road environment and surrounding obstacles.The planning network used multi-modal intermediate representations to extract road environment features and predict waypoints. Model training and performance testing were conducte based on the CARLA simulation platform.The results showed that the end-to-end autonomous driving model can realize the scene understanding of urban road environment and effectively reduce collisions.Compared with the baseline model based on the single modal intermediate representation,its driving performance index is 31.47 % better than the baseline model.

Key words: autonomous driving, scene understanding, imitation learning, trajectory planning

CLC Number:

TP181

KONG Huifang, LIU Runwu, HU Jie. End-to-end autonomous driving model based on multi-modal intermediate representations[J]. Modern Manufacturing Engineering, 2024, 522(3): 70-78.

References

[1] POMERLEAU D A.ALVINN:an autonomous land vehicle in a neural network[C]//Proceedings of the 1st International Conference on Neural Information Processing Systems.Cambridge,MA,USA:[s.n.],1988:305-313.
[2] BOJARSKI M,DELTESTA D,DWORAKOWSKI D,et al.End to End Learning for Self-Driving Cars[EB/OL].(2016-04-25) [2022-12-20].http://arxiv.org/abs/1604.07316.
[3] DOSOVITSKIY A,ROS G,CODEVILLA F,et al.CARLA:An open urban driving simulator[C]//Conference on robot learning.New York,USA:PMLR,2017:1-16.
[4] CODEVILLA F,MÜLLER M,LÓPEZ A,et al.End-to-end driving via conditional imitation learning[C]//2018 IEEE international conference on robotics and automation (ICRA).Brisbane,QLD,Australia:IEEE,2018:4693-4700.
[5] LI Z,MOTOYOSHI T,SASAKI K,et al.Rethinking self-driving:Multi-task knowledge for better generalization and accident explanation ability[EB/OL].(2018-09-28) [2022-12-20].https://arxiv.org/abs/1809.11100.
[6] HU J,KONG H,ZHANG Q,et al.Enhancing scene understanding based on deep learning for end-to-end autonomous driving[J].Engineering Applications of Artificial Intellig-ence,2022,116:105474.
[7] CODEVILLA F,SANTANA E,LOPEZ A,et al.Exploring the Limitations of Behavior Cloning for Autonomous Driving[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV).Seoul,Korea (South):IEEE,2019:9328-9337.
[8] HUANG Z,LV C,XING Y,et al.Multi-modal sensor fusion-based deep neural network for end-to-end autonomous driving with scene understanding[J].IEEE Sensors Journal,2020,21(10):11781-11790.
[9] WEI B,REN M,ZENG W,et al.Perceive,attend,and drive:Learning spatial attention for safe self-driving[C]//2021 IEEE International Conference on Robotics and Automation (ICRA).Xi′an,China:IEEE,2021:4875-4881.
[10] ZHAO A,HE T,LIANG Y,et al.Sam:Squeeze-and-mimic networks for conditional visual driving policy learning[C]//Conference on Robot Learning.New York,USA:PMLR,2021:156-175.
[11] HUANG J,XIE S,SUN J,et al.Learning a decision module by imitating driver’s control behaviors[C]//Conference on Robot Learning.[S.l.]:PMLR,2021:1-10.
[12] BEHL A,CHITTA K,PRAKASH A,et al.Label efficient visual abstractions for autonomous driving[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).Las Vegas,USA:IEEE,2020:2338-2345.
[13] MUELLER M,DOSOVITSKIY A,GHANEM B,et al.Driving Policy Transfer via Modularity and Abstraction[C]//Conference on Robot Learning.New York,USA:PMLR,2018:1-15.
[14] SAX A,ZHANG J O,EMI B,et al.Learning to Navigate Using Mid-Level Visual Priors[C]//Conference on Robot Learning.New York,USA:PMLR,2020:791-812.
[15] SHELHAMER E,LONG J,DARRELL T.Fully Convolutional Networks for Semantic Segmentation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2017,39(4):640-651.
[16] KONG H,HU J,FAN L,et al.Encoder-decoder with double spatial pyramid for semantic segmentation[J].Journal of Electronic Imaging,2019,28(6):063007(1)-063007(10).
[17] HU J,KONG H,FAN L,et al.Enhancing feature fusion with spatial aggregation and channel fusion for semantic segmentation[J].IET Computer Vision,2021,15(6):418-427.
[18] DENG J,DONG W,SOCHER R,et al.ImageNet:A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami,USA:IEEE Computer Society,2009:248-255.
[19] HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.Las Vegas,USA:[s.n.],2016:770-778.
[20] GUPTA S,GIRSHICK R,ARBELÁEZ P,et al.Learning rich features from RGB-D images for object detection and segmentation[C]//Computer Vision-ECCV 2014:13th European Conference,Zurich,Switzerland,Proceedings,Part VII 13.[S.l.]: Springer International Publishing,2014:345-360.
[21] FAN R,WANG H,CAI P,et al.Sne-roadseg:Incorporating surface normal information into semantic segmentation for accurate freespace detection[C]//European Conference on Computer Vision.Cham:Springer International Publishing,2020:340-356.
[22] 李鑫.城市结构化道路下智能汽车的轨迹规划与控制研究[D].长春:吉林大学,2022.
[23] CHEN Y,WANG J,LI J,et al.LiDAR-Video Driving Dataset:Learning Driving Policies Effectively[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Salt Lake City,USA:IEEE Computer Society,2018:5870-5878.
[24] Carla autonomous driving leaderboard (accessed november 2021) [EB/OL]. (2021-12-08)(2022-12-20). https://leaderboardcarla.org/leaderboard/,2021.
[25] PRAKASH A,CHITTA K,GEIGER A.Multi-Modal Fusion Transformer for End-to-End Autonomous Driving[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville,USA:IEEE Computer Society,2021:7073-7083.
[26] CHEN D,ZHOU B,KOLTUN V,et al.Learning by cheating[C]//Conference on Robot Learning.New York,USA:PMLR,2020:66-75.
[27] HINTON G,VINYALS O,DEAN J. Distilling the Knowledge in a Neural Network[EB/OL].(2015-03-09) [2022-12-20].https://arxiv.org/abs/1503.02531.