Tek Görüntü Süper Çözünürlük Uygulamaları İçin Görsel Kaliteyi İyileştirmeye Yönelik Yeni Bir Yaklaşım

Tek görüntü süper çözünürlük problemi, literatürde çeşitli derin öğrenme tabanlı teknikler kullanılarak kapsamlı çalışmalar yapılmıştır. Derin evrişimli ağlar tabanlı süper çözünürlük, çok sayıda pratik uygulama ile beraber hızla büyüyen bir ilgi alanı haline gelmiştir. Bununla birlikte derin öğrenme tabanlı ilk çalışmalar evrişimli sinir ağları tabanlı olup, tepe sinyal gürültü oranı odaklı çalışmalardır. Son yıllardaki çekişmeli üretici ağlar tabanlı geliştirilen modeller sayesinde görsel kaliteyi artırmak esas amaç olarak belirlenmiştir; fakat bu durum görüntü kalite metrikleri incelendiğinde görülmemektedir. Bu çalışmada ise ağın eğitimi sırasında kullanılan ağ kaybı için hem ortalama kare hata hem de algısal kayıp değerlerinden faydalanılmıştır. Ayrıca, üç farklı eğitim veri setinin birleşimi yeni bir eğitim veri seti olarak kullanılmıştır. Bu etmenlerin sonucunda hem görsel kalite artırılmış hem de görüntü kalite metrik değerlerinde ciddi bir artış yakalanmıştır. Ek olarak, yığın normalleştirme katmanları ağ mimarisine dahil edilmemiş ve bağlantı atlama tekniği kullanılarak derin ağ mimarisinin eğitim hızı artırılmıştır. Önerilen modelin başarı performansı literatürde yer alan önemli modeller ile karşılaştırılmıştır. Burada, tepe sinyal gürültü oranı ve yapısal benzerlik indeksi değerleri literatürde yaygın kullanılan üç farklı test veri seti için ayrı ayrı hesaplanmış ve değerlendirilmiştir. Elde edilen sonuçlar değerlendirildiğinde önerilen modelin diğer modellere göre daha başarılı olduğu ve daha kaliteli görüntüler oluşturduğu görülmektedir. Tüm bulgular değerlendirildiğinde önerilen modelin diğer modellere kıyasla hem başarı hem de eğitim hızı bakımından daha verimli bir model olduğu görülmektedir.

Anahtar Kelimeler:

Süper çözünürlük, Derin öğrenme, Görüntü kalite metrikleri, Yeni eğitim veri seti

A New Approach to Improve Visual Quality for Single Image Super Resolution Applications

The single image super resolution problem has been extensively studied in the literature using various deep learning-based techniques. Super resolution based on deep convolutional networks has become a rapidly growing area of interest with many practical applications. However, the first studies based on deep learning were based on convolutional neural networks and peak signal-to-noise ratio (PSNR) oriented. Thanks to the models developed based on generative adversarial networks (GAN) in recent years, it has been determined as the main objective to increase the visual quality; however, this is not seen when the image quality metrics are examined. In this study, both mean square error and perceptual loss values were used for the network loss used during the training of the network. Also, the combination of three different training datasets was used as a new training dataset. As a result of these factors, both the visual quality has been increased and a significant increase has been achieved in the image quality metric values. In addition, batch normalization layers are not included in the network architecture and the training speed of the deep network architecture is increased by using the skip connection technique. The success performance of the proposed model was compared with the state-of-the-art models in the literature. Here, the peak signal to noise ratio (PSNR) and structural similarity index measure (SSIM) values were calculated and evaluated separately for three different test data sets commonly used in the literature. When the obtained results are evaluated, it is seen that the proposed model is more successful than other models and generates higher quality images. When all the findings are evaluated, it is seen that the proposed model is a more efficient model in terms of both success and training speed compared to state-of-the-art models.

Keywords:

Super resolution, Deep learning, Image quality metrics, New training dataset,

PDF

___

Anwar, S., Khan, S., Barnes, N., 2020. A deep journey into super-resolution: A Survey. ACM Computing Surveys 53:1-34. http://dx.doi.org/10.1145/3390462
Chudasama, V., Patel, H., Prajapati, K., Upla, K., Ramachandra, R., Raja, K., Busch, C., 2020. TherISuRNet- A computationally efficient thermal image super-resolution network. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp 388-397. http://dx.doi.org/10.1109/CVPRW50498.2020.00051
Dong, C., Loy, C.C., He, K., Tan, X., 2015. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38:295-307. http://dx.doi.org/10.1109/TPAMI.2015.2439281
Dong, C., Loy, C.C., Tang, X., 2016. Accelerating the Super-Resolution Convolutional Neural Network. Computer Vision ECCV 2016 Lecture Notes in Computer Science Springer 9906. https://doi.org/10.1007/978-3-319-46475-6_25
Dosovitskiy, A., Brox, T., 2016. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems (NIPS). pp 658–666. https://doi.org/10.48550/arXiv.1602.02644
Glasner, D., Bagon, S., Irani, M., 2009. Super-resolution from a single image. IEEE 12th International Conference on Computer Vision. pp 349-356. http://dx.doi.org/10.1109/ICCV.2009.5459271
Goodfellow, I., vd., 2014. Generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS). pp 2672–2680. http://dx.doi.org/10.1145/3422622
Gu, Y., vd., 2020. MedSRGAN: medical images super-resolution using generative adversarial networks. Multimedia Tools and Applications 79:21815–21840. http://dx.doi.org/10.1007/s11042-020-08980-w
Javaid, H., Babar, T.K., Rasool, A., Saghir, R.U., 2013. Video colour variation detection and motion magnification to observe subtle changes. M.Sc. Thesis, Blekinge Institute of Technology, Faisalabad, Pakistan.
Johnson, J., Alahi, A., Li, F., 2016. Perceptual losses for real-time style transfer and super resolution. In European Conference on Computer Vision (ECCV) Springer. pp 694–711. http://dx.doi.org/10.1007/978-3-319-46475-6_43
Kim, J., Lee, J.K., Lee, K.M., 2016. Accurate image super-resolution using very deep convolutional networks. IEEE CVPR. pp 1646–1654. http://dx.doi.org/10.1109/CVPR.2016.182
Ledig, C., vd., 2017. Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu. pp 105-114. http://dx.doi.org/10.1109/CVPR.2017.19
Radford, A., Metz, L., Chintala, S., 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
Senalp, F.M., Ceylan, M., 2020. Enhancement of low resolution thermal face image resolution using deep learning. European Journal of Science & Tech. pp 131-135. https://doi.org/10.31590/ejosat.802174
Senalp, F.M., Ceylan, M., 2021. Deep learning based super resolution and classification applications for neonatal thermal images. Traitement du Signal 38(5):1361-1368. https://doi.org/10.18280/ts.380511
Toyran, M., 2008. Reconstructing super resolution images from low resolution images. M.Sc. Thesis, Institute of science, Istanbul.
Wang, M., Chen, Z., Wu, Q.M.J., Jian, M., 2020. Improved face super-resolution generative adversarial networks. Machine Vision and Applications 31:22. https://doi.org/10.1007/s00138-020-01073-6
Yan, R., Yang, K., Wang, K., 2021. NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation across RGB-Depth, Polarization, and Thermal Images. 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp 1129-1135. http://dx.doi.org/10.1109/ROBIO54168.2021.9739390
Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L., 2018. Image super-resolution:the techniques, applications, and future. Signal Processing 128:389-408. http://dx.doi.org/10.1016/j.sigpro.2016.05.002
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y., 2018. Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision. pp 286–301. http://dx.doi.org/10.1007/978-3-030-01234-2_18