Ali Osman SELVİ, Abdullah FERİKOĞLU, Derya GÜZEL ERDOĞAN

Classification of P300 based brain computer interface systems using long short-term memory (LSTM) neural networks with feature fusion

Enabling to obtain brain activation signs, electroencephalography is currently used in many applications as a medical diagnostic method. Brain-computer interface (BCI) applications are developed to facilitate the lives of individuals who have not lost their brain functions yet have lost their motor and communication abilities. In this study, a BCI system is proposed to make classification using Bi-directional long short term memory (Bi-LSTM) neural networks. In the designed system, spectral entropy method including instantaneous frequency change of signal is used as feature fusion. In the study, electroencephalography (EEG) data of 10 participants are collected with Emotiv EPOC+ device using 2x2 visual stimulus matrix prepared on Unity. Each symbol of the 2x2 matrix includes stimulus such as doctor, police, fireman and family. These stimuli are demonstrated to participants with a fixed order. As data collection protocol, 200 ms stimulus time and 300 ms interstimulus interval are used. As the performance success of classification, the average accuracy rates are obtained to be 98.6% for training set and 96.9% for the test set. In addition, in classification of P300 EEG signals, the results obtained via Bi-LSTM are compared with the results obtained using 1 dimensional convolutional neural networks (1DCNN) and support vector machines (SVM) classification methods. Moreover, in the study, information transfer rate (ITR) is provided as 40.39 at an acceptable level.

PDF

___

[1] Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM. Brain-computer interfaces for communication and control. Clinical Neurophysiology 2002; 113 (6): 767-791. doi: 10.1016/s1388-2457(02)00057-3
[2] Abdulkader SN, Atia A, Mostafa M-SM. Brain computer interfacing: Applications and challenges. Egyptian Informatics Journal 2015; 16 (2): 213-230. doi: 10.1016/j.eij.2015.06.002
[3] Sellers EW, Ryan DB, Hauser CK. Noninvasive brain-computer interface enables communication after brainstem stroke. Science Translational Medicine 2014; 6 (257): 7. doi: 10.1126/scitranslmed.3007801
[4] Beveridge R, Wilson S, Callaghan M, Coyle D. Neurogaming with motion-onset visual evoked potentials (mVEPs): adults versus teenagers. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2019; 27 (4): 572-581. doi: 10.1109/tnsre.2019.2904260
[5] Wolpaw J, Wolpaw EW. Brain-Computer Interfaces: Something New Under The Sun. In: Wolpaw J, Wolpaw EW (editor). Brain-computer interfaces: principles and practice. Manhattan, NY, USA: Oxford University Press, 2012.
[6] Saevarsson S, Kristjánsson Á, Bach M, Heinrich SP. P300 in neglect. Clinical Neurophysiology 2012; 123 (3): 496-506. doi: 10.1016/j.clinph.2011.07.028
[7] Sambeth A, Maes JHR, Brankačk J. With long intervals, inter-stimulus interval is the critical determinant of the human P300 amplitude. Neuroscience Letters 2004; 359 (3): 143-146. doi: 10.1016/j.neulet.2004.01.064
[8] Farwell LA, Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology 1988; 70 (6): 510-523. doi: 10.1016/0013- 4694(88)90149-6
[9] Gu Z, Chen Z, Zhang J, Zhang X, Yu ZL. An online ınteractive paradigm for P300 brain–computer ınterface speller. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2019; 27 (2): 152-161. doi: 10.1109/tnsre.2019.2892967
[10] Perez AF, Oliver MA, Salas G. Development of a brain-computer ınterface based on visual stimuli for the movement of a robot joints. IEEE Latin America Transactions 2016; 14 (2): 477-484. doi: 10.1109/tla.2016.7437182
[11] Iturrate I, Antelis JM, Kubler A, Minguez J. A noninvasive brain-actuated wheelchair based on a P300 neurophysiological protocol and automated navigation. IEEE Transactions on Robotics 2009; 25 (3): 614-627. doi: 10.1109/tro.2009.2020347
[12] He S, Zhang R, Wang Q, Chen Y, Yang T et al. A P300-based threshold-free brain switch and its application in wheelchair control. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2017; 25 (6): 715-725. doi: 10.1109/tnsre.2016.2591012
[13] Wang M, Daly I, Allison BZ, Jin J, Zhang Y et al. A new hybrid BCI paradigm based on P300 and SSVEP. Journal of Neuroscience Methods 2015; 244: 16-25. doi: 10.1016/j.jneumeth.2014.06.003
[14] Kundu S, Ari S. A deep learning architecture for p300 detection with brain-computer ınterface application. IRBM 2020; 41 (1): 31-38. doi: 10.1016/j.irbm.2019.08.001
[15] Ditthapron A, Banluesombatkul N, Ketrat S, Chuangsuwanich E, Wilaiprasitporn T. universal joint feature extraction for P300 EEG classification using multi-task autoencoder. IEEE Access 2019; 7: 68415-68428. doi: 10.1109/access.2019.2919143
[16] Martínez-Cagigal V, Santamaría-Vázquez E, Gomez-Pilar J, Hornero R. Towards an accessible use of smartphonebased social networks through brain-computer interfaces. Expert Systems with Applications 2019; 120: 155-166. doi: 10.1016/j.eswa.2018.11.026
[17] Chen J, Wang H, Wang Q, Hua C. Exploring the fatigue affecting electroencephalography based functional brain networks during real driving in young males. Neuropsychologia 2019; 129: 200-211. doi: 10.1016/j.neuropsychologia.2019.04.004
[18] Oropesa E, Cycon HL, Jobert M. Sleep stage classification using wavelet transform and neural network. International computer science institute 1999.
[19] Van Hese P, Philips W, De Koninck J, Van de Walle R, Lemahieu I et al. Automatic detection of sleep stages using the EEG. In: 23rd Annual International Conference of the IEEE-Engineering-in-Medicine-and-Biology-Society; İstanbul, Turkey 2001. pp. 1944-1947.
[20] Kaur B, Singh D, Roy PP. Age and gender classification using brain-computer interface. Neural Computing & Applications 2019; 31 (10): 5887-5900. doi: 10.1007/s00521-018-3397-1
[21] Katyal EA, Singla R. EEG-based hybrid QWERTY mental speller with high information transfer rate. Medical & Biological Engineering & Computing 2021; 59 (3): 633-661. doi: 10.1007/s11517-020-02310-w
[22] Islam MK, Rastegarnia A, Nguyen AT, Yang Z. Artifact characterization and removal for in vivo neural recording. Journal of Neuroscience Methods. 2014; 226: 110-123. doi: 10.1016/j.jneumeth.2014.01.027
[23] Luke R, Wouters J. Kalman Filter Based Estimation of Auditory Steady State Response Parameters. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2017; 25 (3): 196-204. doi: 10.1109/tnsre.2016.2551302
[24] Majumdar KK, Vardhan P. Automatic Seizure Detection in ECoG by Differential Operator and Windowed Variance. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2011; 19 (4): 356-365. doi: 10.1109/tnsre.2011.2157525
[25] Martinez P, Bakardjian H, Cichocki A. Fully Online Multicommand Brain-Computer Interface with Visual Neurofeedback Using SSVEP Paradigm. Computational Intelligence and Neuroscience 2007; 2007: 094561. doi: 10.1155/2007/94561
[26] Dornhege G, Blankertz B, Curio G, Muller K-R. Boosting Bit Rates in Noninvasive EEG Single-Trial Classifications by Feature Combination and Multiclass Paradigms. IEEE Transactions on Biomedical Engineering 2004; 51 (6): 993-1002. doi: 10.1109/tbme.2004.827088
[27] Basiri M MA, Sk NM. Configurable Folded IIR Filter Design. IEEE Transactions on Circuits and Systems II: Express Briefs 2015; 62 (12): 1144-1148. doi: 10.1109/tcsii.2015.2468917
[28] Yang Z, Ren H. Feature Extraction and Simulation of EEG Signals During Exercise-Induced Fatigue. IEEE Access 2019; 7: 46389-46398. doi: 10.1109/access.2019.2909035
[29] Karimzadeh F, Boostani R, Seraj E, Sameni R. A Distributed Classification Procedure for Automatic Sleep Stage Scoring Based on Instantaneous Electroencephalogram Phase and Envelope Features. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; 26 (2): 362-370. doi: 10.1109/tnsre.2017.2775058
[30] Zhao X, Zhang R, Mei Z, Chen C, Chen W. Identification of Epileptic Seizures by Characterizing Instantaneous Energy Behavior of EEG. IEEE Access 2019; 7: 70059-70076. doi: 10.1109/access.2019.2919158
[31] Wang H, Wu C, Li T, He Y, Chen P et al. Driving Fatigue Classification Based on Fusion Entropy Analysis Combining EOG and EEG. IEEE Access 2019; 7: 61975-61986. doi: 10.1109/access.2019.2915533
[32] Mesbah M, O’ Toole JM, Colditz PB, Boashash B. Instantaneous frequency based newborn EEG seizure characterisation. EURASIP Journal on Advances in Signal Processing 2012; 2012 (1): 143. doi: 10.1186/1687-6180-2012-143
[33] Yildirim O, Baloglu UB, Tan R-S, Ciaccio EJ, Acharya UR. A new approach for arrhythmia classification using deep coded features and LSTM networks. Computer Methods and Programs in Biomedicine 2019; 176: 121-133. doi: 10.1016/j.cmpb.2019.05.004
[34] Zhao J, Mao X, Chen L. Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control 2019; 47: 312-323. doi: 10.1016/j.bspc.2018.08.035
[35] Hu X, Yuan S, Xu F, Leng Y, Yuan K et al. Scalp EEG classification using deep Bi-LSTM network for seizure detection. Computers in Biology and Medicine 2020; 124: 103919. doi: 10.1016/j.compbiomed.2020.103919
[36] Alkan A, Kiymik MK. Comparison of AR and Welch Methods in Epileptic Seizure Detection. Journal of Medical Systems 2006; 30 (6): 413-419. doi: 10.1007/s10916-005-9001-0
[37] Kaper M, Meinicke P, Grossekathoefer U, Lingner T, Ritter H. BCI Competition 2003—Data Set IIb: Support Vector Machines for the P300 Speller Paradigm. IEEE Transactions on Biomedical Engineering 2004; 51 (6): 1073- 1076. doi: 10.1109/tbme.2004.826698
[38] Xu G, Ren T, Chen Y, Che W. A One-Dimensional CNN-LSTM Model for Epileptic Seizure Recognition Using EEG Signal Analysis. Frontiers in Neuroscience 2020; 14. doi: 10.3389/fnins.2020.578126
[39] Sindi H, Nour M, Rawa M, Öztürk Ş, Polat K. A novel hybrid deep learning approach including combination of 1D power signals and 2D signal images for power quality disturbance classification. Expert Systems with Applications 2021; 174: 114785. doi: 10.1016/j.eswa.2021.114785
[40] Kundu S, Ari S. P300 Detection with Brain–Computer Interface Application Using PCA and Ensemble of Weighted SVMs. IETE Journal of Research 2018; 64 (3): 406-414. doi: 10.1080/03772063.2017.1355271
[41] Aydin EA, Bay OF, Guler I. P300-Based Asynchronous Brain Computer Interface for Environmental Control System. IEEE Journal of Biomedical and Health Informatics 2018; 22 (3): 653-663. doi: 10.1109/jbhi.2017.2690801
[42] Vo K, Pham T, Nguyen DN, Kha HH, Dutkiewicz E. Subject-Independent ERP-Based Brain–Computer Interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2018; 26 (4): 719-728. doi: 10.1109/tnsre.2018.2810332
[43] Kshirsagar GB, Londhe ND. Weighted Ensemble of Deep Convolution Neural Networks for Single-Trial Character Detection in Devanagari-Script-Based P300 Speller. IEEE Transactions on Cognitive and Developmental Systems 2020; 12 (3): 551-560. doi: 10.1109/tcds.2019.2942437
[44] Jin J, Li S, Daly I, Miao Y, Liu C et al. The Study of Generic Model Set for Reducing Calibration Time in P300- Based Brain–Computer Interface. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2020; 28 (1): 3-12. doi: 10.1109/tnsre.2019.2956488
[45] Arican M, Polat K. Pairwise and variance based signal compression algorithm (PVBSC) in the P300 based speller systems using EEG signals. Computer Methods and Programs in Biomedicine 2019; 176: 149-157. doi: 10.1016/j.cmpb.2019.05.011
[46] Ratcliffe L, Puthusserypady S. Importance of Graphical User Interface in the design of P300 based Brain–Computer Interface systems. Computers in Biology and Medicine 2020; 117: 103599. doi: 10.1016/j.compbiomed.2019.103599
[47] Li M, Yang G, Xu G. The Effect of the Graphic Structures of Humanoid Robot on N200 and P300 Potentials. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2020; 28 (9): 1944-1954. doi: 10.1109/tnsre.2020.3010250
[48] Santamaria-Vazquez E, Martinez-Cagigal V, Vaquerizo-Villar F, Hornero R. EEG-Inception: A Novel Deep Convolutional Neural Network for Assistive ERP-Based Brain-Computer Interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2020; 28 (12): 2773-2782. doi: 10.1109/tnsre.2020.3048106
[49] Oralhan Z. 3D Input Convolutional Neural Networks for P300 Signal Detection. IEEE Access 2020; 8: 19521-19529. doi: 10.1109/access.2020.2968360
[50] Shukla PK, Chaurasiya RK, Verma S. Performance improvement of P300-based home appliances control classification using convolution neural network. Biomedical Signal Processing and Control 2021; 63: 16. doi: 10.1016/j.bspc.2020.102220
[51] Liu X, Xie Q, Lv J, Huang H, Wang W. P300 event-related potential detection using one-dimensional convolutional capsule networks. Expert Systems with Applications 2021; 174: 114701. doi: 10.1016/j.eswa.2021.114701
[52] Alvarado-González M, Fuentes-Pineda G, Cervantes-Ojeda J. A few filters are enough: Convolutional Neural Network for P300 Detection. Neurocomputing 2021; 425: 37-52. doi: 10.1016/j.neucom.2020.10.104
[53] Li M, Yang G, Liu Z, Gong M, Xu G et al. The Effect of SOA on An Asynchronous ERP and VEP-Based BCI. IEEE Access 2021; 9: 9972-9981. doi: 10.1109/access.2021.3050545