Videolardan Kalp Atış Hızı Kestirimi Üzerine Bir İnceleme

Kalp atış hızı; kişinin sağlığı, aktivite seviyesi, stres durumu, zindeliği ve benzeri fizyolojik durumları hakkında önemli ipuçları vermektedir. Kalp atış hızı, elektrokardiyogram (EKG) ve nabız oksimetreleriyle ölçülebilir olmakla birlikte, bu cihazlar sürekli temas gerektirdiğinden zamanla rahatsız edici olabilmektedir. Bilgisayarlı görü (computer vision) alanındaki son gelişmeler, bir kişiye elektrot veya nabız oksimetreleri takmanın mümkün veya uygun olmadığı durumlarda, videolardan kişinin kalp atış hızını tespit etmeye olanak sağlamıştır. Uzaktan fotopletismografi (rPPG), bir video kamera aracılığıyla derideki hassas renk değişikliklerini yakalayarak, yaşamsal belirtilerin tespit edilmesine imkân sağlayan bir teknolojidir. Son yıllarda yapılan çalışmalar, uzaktan kalp atış hızı tespiti için en uygun bölgenin yüz olduğunu göstermiştir. Bu çalışmada; videolar aracılığıyla kişilerin yüz bölgesinden kalp atışı hızı kestiriminin nasıl yapılabildiği, kalp atışı hızı kestirimi sürecindeki aşamaların nasıl iyileştirilebileceği ve nasıl daha yüksek doğrulukta kalp atışı hızı tespiti yapılabileceği hakkında literatürdeki mevcut yöntemler incelenerek kapsamlı bir analiz yapılmıştır.

Anahtar Kelimeler:

Kalp atış hızı, uzaktan fotopletismografi (rPPG), yüz videoları, derin öğrenme.

Heart Rate Estimation from Videos: A Review

Heartbeat rate provides important signs about a person's health, activity level, stress level, vitality, and other related physiological conditions. Although electrocardiograms (ECG) and pulse oximeters can be exploited to measure the heart rate, these instruments might become uncomfortable with time due to the requirement for constant contact. Recent advances in computer vision have made it achievable to detect a person's heart rate from videos when it is impossible or impractical to attach electrodes or pulse oximeters to a person. Remote photoplethysmography (rPPG) is a technology that detects vital physiological information by capturing considerably precise color changes in the skin via a video camera. Recent studies have shown that the face is the most appropriate area to use for remote heart rate detection. In this study, a comprehensive analysis has been conducted by investigating the existing methods in the literature on how to estimate the heartbeat rate from a person's face through videos, how to improve the stages in the heart rate estimation process, and hence to improve the accuracy of the estimations.

Keywords:

Heartbeat rate, remote photoplethysmography (rPPG), face videos, deep learning,

PDF

___

Aydemir, B. (2019). Egzersiz esnasında toplanan fotopletismografi sinyallerinden kalp atım hızı ve ventilasyon hızı değişkenlerinin ayrıştırılması [MARMARA ÜNİVERSİTESİ]. https://acikbilim.yok.gov.tr/handle/20.500.12812/282457
Bian, M. (2019). Pattern Recognition and Computer Vision. In Fundamentals of Uncertainty Calculi with Applications to Fuzzy Inference. https://doi.org/10.1007/978-94-015-8449-4_9
Bousefsaf, F., Maaoui, C., & Pruski, A. (2013). Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate. Biomedical Signal Processing and Control, 8(6), 568–574. https://doi.org/10.1016/j.bspc.2013.05.010
Bush, I. (2016). Measuring Heart Rate from Video. https://web.stanford.edu/class/cs231a/prev_projects_2016/finalReport.pdf
Chen, W., & McDuff, D. (2018). DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11206 LNCS, 356–373. https://doi.org/10.1007/978-3-030-01216-8_22
Cheng, C. H., Wong, K. L., Chin, J. W., Chan, T. T., & So, R. H. Y. (2021). Deep learning methods for remote heart rate measurement: A review and future research agenda. In Sensors (Vol. 21, Issue 18). MDPI. https://doi.org/10.3390/s21186296
De Haan, G., & Jeanne, V. (2013). Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering, 60(10), 2878–2886. https://doi.org/10.1109/TBME.2013.2266196
Deng, Y., & Kumar, A. (2020). Standoff heart rate estimation from video – a review. 6. https://doi.org/10.1117/12.2560683
Djeldjli, D., Bousefsaf, F., Maaoui, C., & Bereksi-Reguig, F. (2019). Imaging Photoplethysmography: Signal Waveform Analysis. Proceedings of the 2019 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS 2019, 2, 830–834. https://doi.org/10.1109/IDAACS.2019.8924239
Elgendi, M. (2012). On the Analysis of Fingertip Photoplethysmogram Signals. Current Cardiology Reviews, 8(1), 14–25. https://doi.org/10.2174/157340312801215782
ElMaghraby, A., Abdalla, M., Enany, O., & Y. El Nahas, M. (2014). Detect and Analyze Face Parts Information using Viola- Jones and Geometric Approaches. International Journal of Computer Applications, 101(3), 23–28. https://doi.org/10.5120/17667-8494
Hassan, M. A., Malik, A. S., Fofi, D., Saad, N., Karasfi, B., Ali, Y. S., & Meriaudeau, F. (2017). Heart rate estimation using facial video: A review. In Biomedical Signal Processing and Control (Vol. 38, pp. 346–360). Elsevier Ltd. https://doi.org/10.1016/j.bspc.2017.07.004
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 770–778. https://doi.org/10.1109/CVPR.2016.90
Huang, B., Lin, C.-L., Chen, W., Juang, C.-F., & Wu, X. (2021). A novel one-stage framework for visual pulse rate estimation using deep neural networks. Biomedical Signal Processing and Control, 66(June 2020), 102387. https://doi.org/10.1016/j.bspc.2020.102387
Irani, R., Nasrollahi, K., & Moeslund, T. B. (2014). Improved pulse detection from head motions using DCT. VISAPP 2014 - Proceedings of the 9th International Conference on Computer Vision Theory and Applications, 3, 118–124. https://doi.org/10.5220/0004669001180124
Kazemi, V., & Sullivan, J. (2014). One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1867–1874. https://doi.org/10.1109/CVPR.2014.241
Kwon, S., Kim, J., Lee, D., & Park, K. (2015). ROI analysis for remote photoplethysmography on facial video. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2015-Novem, 4938–4941. https://doi.org/10.1109/EMBC.2015.7319499
Li, X., Chen, J., Zhao, G., & Pietikäinen, M. (2014). Remote heart rate measurement from face videos under realistic situations. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 4264–4271. https://doi.org/10.1109/CVPR.2014.543
McDuff, D., Gontarek, S., & Picard, R. W. (2014a). Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Transactions on Biomedical Engineering, 61(10), 2593–2601. https://doi.org/10.1109/TBME.2014.2323695
McDuff, D., Gontarek, S., & Picard, R. W. (2014b). Remote detection of photoplethysmographic systolic and diastolic peaks using a digital camera. IEEE Transactions on Biomedical Engineering, 61(12), 2948–2954. https://doi.org/10.1109/TBME.2014.2340991
Meinzer, H. P., Deserno, T. M., Handels, H., & Tolxdorff, T. (2013). ROI Selection for Remote Photoplethysmography. Informatik Aktuell, 99–103. https://doi.org/10.1007/978-3-642-36480-8
Niu, X., Han, H., Shan, S., & Chen, X. (2018). SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific. Proceedings - International Conference on Pattern Recognition, 2018-Augus(i), 3580–3585. https://doi.org/10.1109/ICPR.2018.8546321
Niu, X., Shan, S., Han, H., & Chen, X. (2020). RhythmNet: End-to-End Heart Rate Estimation from Face via Spatial-Temporal Representation. IEEE Transactions on Image Processing, 29, 2409–2423. https://doi.org/10.1109/TIP.2019.2947204
Pagano, T. P., Santos, V. R., Bonfim, Y. da S., Paranhos, J. V. D., Ortega, L. L., Sá, P. H. M., Nascimento, L. F. S., Winkler, I., & Nascimento, E. G. S. (2022). Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature. Electronics (Switzerland), 11(9). https://doi.org/10.3390/electronics11091473
Poh, M., Mcduff, D. J., & Picard, R. W. (2010). Noncontact automated cardiac pulse measurements using video imaging and blind.pdf. Medical Optics and Biotechnology, 18(10), 795–805.
Poh, M. Z., McDuff, D. J., & Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering, 58(1), 7–11. https://doi.org/10.1109/TBME.2010.2086456
Premkumar, S., & Hemanth, D. J. (2022). Intelligent Remote Photoplethysmography-Based Methods for Heart Rate Estimation from Face Videos: A Survey. In Informatics (Vol. 9, Issue 3). MDPI. https://doi.org/10.3390/informatics9030057
Rautaray, S. S., & Agrawal, A. (2012). R Eal T Ime H and G Esture R Ecognition. 3(1), 21–31.
Rouast, P. V., Adam, M. T. P., Chiong, R., Cornforth, D., & Lux, E. (2018). Remote heart rate measurement using low-cost RGB face video: a technical literature review. In Frontiers of Computer Science (Vol. 12, Issue 5, pp. 858–872). Higher Education Press. https://doi.org/10.1007/s11704-016-6243-6
Sabokrou, M., Pourreza, M., Li, X., Fathy, M., & Zhao, G. (2021). Deep-HR: Fast heart rate estimation from face video under realistic conditions. Expert Systems with Applications, 186. https://doi.org/10.1016/j.eswa.2021.115596
Shao, D., Liu, C., & Tsow, F. (2021). Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions. ACS Sensors, 6(2), 321–334. https://doi.org/10.1021/acssensors.0c02042
Sinhal, R., Singh, K., & Raghuwanshi, M. M. (2020). An Overview of Remote Photoplethysmography Methods for Vital Sign Monitoring. Advances in Intelligent Systems and Computing, 992, 21–31. https://doi.org/10.1007/978-981-13-8798-2_3
Song, R., Chen, H., Cheng, J., Li, C., Liu, Y., & Chen, X. (2021). PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethysmography. IEEE Journal of Biomedical and Health Informatics, 25(5), 1373–1384. https://doi.org/10.1109/JBHI.2021.3051176
Spetlik, R., Franc, V., Cech, J., & Matas, J. (2018). Visual heart rate estimation with convolutional neural network. British Machine Vision Conference 2018, BMVC 2018, 1–12.
Sun, Y., & Thakor, N. (2016). Photoplethysmography Revisited: From Contact to Noncontact, from Point to Imaging. IEEE Transactions on Biomedical Engineering, 63(3), 463–477. https://doi.org/10.1109/TBME.2015.2476337
Swinehart, D. F. (1962). The Beer-Lambert law. Journal of Chemical Education, 39(7), 333–335. https://doi.org/10.1021/ed039p333
Tamura, T., Maeda, Y., Sekine, M., & Yoshida, M. (2014). Wearable photoplethysmographic sensors—past and present. Electronics , 3(2), 282–302. https://doi.org/10.3390/electronics3020282
Tang, C., Lu, J., & Liu, J. (2018). Non-contact heart rate monitoring by combining convolutional neural network skin detection and remote photoplethysmography via a low-cost camera. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2018-June, 1390–1396. https://doi.org/10.1109/CVPRW.2018.00178
Tsou, Y. Y., Lee, Y. A., Hsu, C. T., & Chang, S. H. (2020). Siamese-rPPG network: Remote photoplethysmography signal estimation from face videos. Proceedings of the ACM Symposium on Applied Computing, 2066–2073. https://doi.org/10.1145/3341105.3373905
Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434. https://doi.org/10.1364/oe.16.021434
Wang, W. (2017). Robust And Automatic Remote Photoplethysmography (Vol. 1, Issue 2017). https://pure.tue.nl/ws/files/78340965/20171023_Wang.pdf%0Ahttps://research.tue.nl/en/publications/robust-and-automatic-remote- photoplethysmography%0Ahttps://pure.tue.nl/ws/portalfiles/portal/78340965/20171023_Wang.pdf
Wedekind, D., Trumpp, A., Gaetjen, F., Rasche, S., Matschke, K., Malberg, H., & Zaunseder, S. (2017). Assessment of blind source separation techniques for video-based cardiac pulse extraction. Journal of Biomedical Optics, 22(3), 035002. https://doi.org/10.1117/1.jbo.22.3.035002
YAMAN, A. U. (2018). Yüz tanıma sistemlerinin yanıltılmasına karşı bir yöntem: yüz videolarında nabız tespiti ile canlılık doğrulaması [Ankara Üniversitesi]. https://tez.yok.gov.tr/UlusalTezMerkezi/tezDetay.jsp?id=UwWffTuiMVwjS85blanc6Q&no=LlzkDJAuYHVHoM9tysjGIA
Yang, W., Li, X., & Zhang, B. (2018). Heart Rate Estimation from Facial Videos Based on Convolutional Neural Network. Proceedings of 2018 6th IEEE International Conference on Network Infrastructure and Digital Content, IC-NIDC 2018, 45–49. https://doi.org/10.1109/ICNIDC.2018.8525602
Yu, Z., Li, X., & Zhao, G. (2020). Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. 30th British Machine Vision Conference 2019, BMVC 2019.
Yu, Z., Peng, W., Li, X., Hong, X., & Zhao, G. (2019). Remote heart rate measurement from highly compressed facial videos: An end-to-end deep learning solution with video enhancement. Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob, 151–160. https://doi.org/10.1109/ICCV.2019.00024
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016). Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters, 23(10), 1499–1503. https://doi.org/10.1109/LSP.2016.2603342