Ahmet KAYABASI, Mehmet DURAMAZ, Berat YILDIZ, Akif DURDU

CNN based sensor fusion method for real-time autonomous robotics systems

Autonomous robotic systems (ARS) serve in many areas of daily life. The sensors have critical importance for these systems. The sensor data obtained from the environment should be as accurate and reliable as possible and correctly interpreted by the autonomous robot. Since sensors have advantages and disadvantages over each other they should be used together to reduce errors. In this study, Convolutional Neural Network (CNN) based sensor fusion was applied to ARS to contribute the autonomous driving. In a real-time application, a camera and LIDAR sensor were tested with these networks. The novelty of this work is that the uniquely collected data set was trained in a new CNN network and sensor fusion was performed between CNN layers. The results showed that CNN based sensor fusion process was more effective than the individual usage of the sensors on the ARS.

PDF

___

[1] Asama H, Fukuda T, Arai T, Endo I. Distributed autonomous robotic systems 2. Springer Science and Business Media, 2013
[2] Fraedrich E, Beiker S, Lenz B. Transition pathways to fully automated driving and its implications for the sociotechnical system of automobility. European Journal of Futures Research 2015; 3 (1): 11. doi:10.1007/s40309- 015-0067-8
[3] Virdi N, Grzybowska H, Waller ST, Dixit V. A safety assessment of mixed fleets with Connected and Autonomous Vehicles using the Surrogate Safety Assessment Module. Accident Analysis and Prevention 2019; 131: 95-111. doi:10.1016/j.aap.2019.06.001
[4] Loeb B, Kockelman KM. Fleet performance and cost evaluation of a shared autonomous electric vehicle (SAEV) fleet: A case study for Austin, Texas. Transportation Research Part A: Policy and Practice 2019; 121: 374-385. doi:10.1016/j.tra.2019.01.025
[5] Waseem M, Sherwani AF, Suhaib M. Integration of solar energy in electrical, hybrid, autonomous vehicles: a technological review. SN Applied Sciences 2019; 1 (11): 1459. doi:10.1007/s42452-019-1458-4
[6] Gao Z, Wang J, Zhang X, Dong J, Chen L et al. Traffic oscillations mitigation in vehicle platoon using a car-following control model for connected and autonomous vehicle. Journal of Advanced Transportation 2019. doi:10.1155/2019/3067291
[7] Xu Q, Yang Y, Zhang C, Zhang L. Deep convolutional neural network-based autonomous marine vehicle maneuver. International Journal of Fuzzy Systems 2018; 20 (2): 687-699. doi:10.1007/s40815-017-0393-z
[8] Tahboub K, Güera D, Reibman AR, Delp EJ. Quality-adaptive deep learning for pedestrian detection. In 2017 IEEE International Conference on Image Processing (ICIP), 2017. pp. 4187-4191.
[9] Schlosser J, Chow CK, Kira Z. Fusing lidar and images for pedestrian detection using convolutional neural networks. In 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden; 2016. pp. 2198- 2205. doi:10.1109/ICRA.2016.7487370
[10] Premebida C, Carreira J, Batista J, Nunes U. Pedestrian detection combining RGB and dense LIDAR data. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, USA; 2014. pp. 4112-4117. doi:10.1109/IROS.2014.6943141
[11] Adarsh S, Ramachandran KI. Design of sensor data fusion algorithm for mobile robot navigation using ANFIS and its analysis across the membership functions. Automatic Control and Computer Sciences 2018; 52 (5): 382-391. doi:10.3103/S0146411618050036
[12] Bostanci E, Bostanci B, Kanwal N, Clark AF. Sensor fusion of camera, GPS and IMU using fuzzy adaptive multiple motion models. Soft Computing 2018; 22 (8): 2619-2632. doi:10.1007/s00500-017-2516-8
[13] Yi C, Zhang K, Peng N. A multi-sensor fusion and object tracking algorithm for self-driving vehicles. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of automobile engineering 2019; 233 (9): 2293-2300. doi:10.1177/0954407019867492
[14] De Silva V, Roche J, Kondoz A. Robust fusion of LiDAR and wide-angle camera data for autonomous mobile robots. Sensors 2018; 18 (8): 2730. doi:10.3390/s18082730
[15] Xiao L, Wang R, Dai B, Fang Y, Liu D et al. Hybrid conditional random field based camera-LIDAR fusion for road detection. Information Sciences 2018; 432: 543-558. doi:10.1016/j.ins.2017.04.048
[16] Xie S, Yang D, Jiang K, Zhong Y. Pixels and 3-D points alignment method for the fusion of camera and LiDAR data. IEEE Transactions on Instrumentation and Measurement 2018; 68 (10): 3661-3676. doi:10.1109/TIM.2018.2879705
[17] Graves A, Mohamed A.R, Hinton G. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, Vancouver, Canada; 2013. pp. 6645-6649. doi:10.1109/ICASSP.2013.6638947
[18] Lenz I, Lee H, Saxena A. Deep learning for detecting robotic grasps. The International Journal of Robotics Research 2015; 34 (4-5): 705-724. doi:10.1177/0278364914549607
[19] Yang P.C, Sasaki K, Suzuki K, Kase K, Sugano S et al. Repeatable folding task by humanoid robot worker using deep learning. IEEE Robotics and Automation Letters 2016; 2 (2): 397-403. doi:10.1109/LRA.2016.2633383
[20] Ivakhnenko AG. Polynomial theory of complex systems. IEEE transactions on Systems, Man, and Cybernetics 1971; (4): 364-378. doi:10.1109/TSMC.1971.4308320
[21] Fukushima K. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks 1988; 1 (2): 119-130. doi:10.1016/0893-6080(88)90014-7
[22] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 1998; 86 (11): 2278-2324. doi:10.1109/5.726791
[23] NVIDIACorporation. what is CUDA. 2019.
[24] Deng J, Dong W, Socher R, Li LJ, Li K et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, Miami, FL; 2009. pp. 248-255. doi:10.1109/CVPR.2009.5206848
[25] Dhillon A, Verma GK. Convolutional neural network: a review of models, methodologies and applications to object detection. Progress in Artificial Intelligence 2020; 9 (2): 85-112. doi:10.1007/s13748-019-00203-0
[26] Person M, Jensen M, Smith AO, Gutierrez H. Multimodal Fusion Object Detection System for Autonomous Vehicles. Journal of Dynamic Systems, Measurement, and Control 2019; 141 (7): 071017. doi.org/10.1115/1.4043222
[27] Shahian Jahromi B, Tulabandhula T, Cetin S. Real-Time Hybrid Multi-Sensor Fusion Framework for Perception in Autonomous Vehicles. Sensors 2019; 19 (20): 4357. doi:10.3390/s19204357
[28] NVIDIACorporation. Convolutional Neural Network (CNN). 2019.
[29] Hinton GE. Learning multiple layers of representation. Trends in cognitive sciences 2007; 11 (10): 428-434. doi:10.1016/j.tics.2007.09.004
[30] Göranzon B, Florin M. Artifical Intelligence, Culture and Language: On Education and Work. Springer Science and Business Media, 2012.
[31] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015; 521 (7553): 436-444. doi:10.1038/nature14539
[32] Willmott CJ, Matsuura K. Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate research 2005; 30 (1): 79-82. doi:10.3354/cr03007915
[33] Durdu A, Bol N, Öztürk E, Duramaz M, Korkmaz M et al. Convolutional Neural Networks Based Active SLAM and Exploration. Avrupa Bilim ve Teknoloji Dergisi 2021; (22): 342-346. doi:10.31590/ejosat.862953
[34] Durdu A, Korkmaz M. A novel map-merging technique for occupancy grid-based maps using multiple robots: a semantic approach. Turkish Journal of Electrical Engineering and Computer Sciences 2019; 27 (5): 3980-3993. doi:10.3906/elk-1807-335