Uluğ BAYAZIT, Turgay ALTILAR, Nilgün GÜLER BAYAZIT

Classification and phenological staging of crops from in situ image sequences by deep learning

Accurate knowledge of crop type information is not only valuable for verifying the declaration of farmers to obtain subsidy or insurance for the grown crop, but also for generating crop type maps that serve a variety of purposes in land monitoring and policy. On the other hand, accurate knowledge of crop phenological stage can help farm personnel apply fertilization and irrigation regimes on a timely basis. Although deep learning based networks have been applied in the past to classify the type and predict the phenological stage of crops from in situ images of fields, more advanced deep learning based networks, that learn and make such inferences from temporal windows of sequences of field images taken by cameras at stationary coordinates and looking directions, have not been reported to date. This work proposes a conceivable architecture for learning and making inferences from such data. Specifically, the feature vectors of the images in a temporal window of the image sequence for a crop cycle are extracted by a first stage deep convolutional neural network and their temporal dependencies are exploited by a second stage recurrent neural network. Experiments on a dataset of image sequences from 63 fields of 5 different types of crops reveal that the proposed system can achieve over 80% accuracy in crop type classification and under 0.5 mean absolute error in phenological stage number estimation. The learning performances improve with the size of the temporal window and the fine-tuning of the deep convolutional neural network used for feature extraction. The performances achieved with the proposed system are superior to those obtained by applying classical machine learning methods to handcrafted texture and color features.

PDF

___

[1] Baccouche M, Mamalet F, Wolf C, Garcia C, Baskurt A. Sequential deep learning for human action recognition. In: Lepri B, Salah A A (editors). 2nd International Workshop on Human Behavior Understanding (HBU); 2011. pp. 29–39.
[2] Benco M, Hudec R, Kamencay P, Radilova M, Matuska S. An advanced approach to extraction of colour texture features based on GLCM. International Journal of Advanced Robotic Systems 2014; 11 (7): 1-8. doi: 10.5772/58692
[3] Bodhwani V, Acharjya D, Bodhwani U. Deep residual networks for plant identification. Procedia Computer Science 2019; 152 (1): 186-194. doi: 10.1016/j.procs.2019.05.042
[4] Castro J, Feitoza R, Cue La Rosa L, Achanccaray Diaz P, Sanches I. A comparative analysis of deep learning techniques for sub-tropical crop types recognition from multitemporal optical/SAR image sequences. In: 2017 30th Conference on Graphics, Patterns and Images (SIBGRAPI); Niteroi, Brazil; 2017. pp. 382-389.
[5] Cheng Z. Detecting phenological transition dates of vegetation based on multiple deep learning models. MS, TU Delft University of Technology, 2600 AA Delft, The Netherlands, 2018.
[6] Desai SV, Balasubramanian VN, Fukatsu T, Ninomiya S, Guo W. Automatic estimation of heading date of paddy rice using deep learning. Plant Methods 2019; 15 : 76. doi: 10.1186/s13007-019-0457-1
[7] Donahue J, Hendricks L, Guadarrama S, Rohrbach M, Venugopalan S et al. Long-term recurrent convolutional networks for visual recognition and description. In: IEEE 2015 Conference on Computer Vision and Pattern Recognition (CVPR); Boston, MA, USA; 2015. pp. 2625-2634.
[8] Forcén M, Pavón Pulido N, Pérez Noguera D, Berríos Reyes P, Pérez Pastor A et al. Machine Learning-based inference system to detect the phenological stage of a citrus crop for helping deficit irrigation techniques to be automatically applied. In: EGU General Assembly Conference Abstracts; Vienna, Austria; 2020. p. 18284.
[9] Grinblat G, Uzal L, Larese M, Granitto P. Deep learning for plant identification using vein morphological patterns. Computers and Electronics in Agriculture 2016; 127 (7): 418-424. doi: 10.1016/j.compag.2016.07.003
[10] Heredia I. Large-scale plant classification with deep neural networks. In: The Computing Frontiers Conference; Sienna, Italy; 2017. pp. 259-262.
[11] Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation 1997; 9 (8): 1735-1780. doi: 10.1162/neco.19997.9.8.1735
[12] Ji S, Zhang C, Xu A, Shi Y, Duan Y. 3D convolutional neural networks for crop classification with multi-temporal remote sensing images. Remote Sensing 2018; 10 (1): 75. doi: 10.3390/rs10010075
[13] Lancashire P, Bleiholder H, Boom T, Langeluddeke P, Stauss R et al. A uniform decimal code for growth stages of crops and weeds. The Annals of Applied Biology 1991; 119 (3): 561-601. doi: 10.1111/j.1744-7348.1991.tb04895.x
[14] Lee SH, Chan CS, Wilkin P, Remagnino P. Deep-plant: Plant identification with convolutional neural networks. In: IEEE International Conference on Image Processing; Quebec City, Canada; 2015. pp. 452-456.
[15] Mortensen AK, Dyrmann M, Karstoft H, Jørgensen RN, Gislum R. Semantic segmentation of mixed crops using deep convolutional neural network. In: International Conference on Agricultural Engineering: Automation, Environment and Food Safety; Aarhus, Denmark; 2016: 1-6.
[16] Ng J-H, Hausknecht M, Vijayanarasimhan S, Vinyals O, Monga R et al. Beyond short snippets: Deep networks for video classification. In: IEEE 2015 Conference on Computer Vision and Pattern Recognition (CVPR); Boston, MA, USA; 2015: 4694-4702.
[17] Pound M, Burgess A, Wilson M, Atkinson J, Griffiths M et al. Deep machine learning provides state-of-the-art performance in image-based plant phenotyping. GigaScience 2016; 6 (10): 1-10. doi: 10.193/gigascience/gix083
[18] Rasti S, Bleakley C, Silvestre GCM, Holden NM, Langton D et al. Crop growth stage estimation prior to canopy closure using deep learning algorithms. Neural Computing and Applications 2020; 33: 1733–1743. doi: 10.1007/s00521-020-05064-6
[19] Rebetez J, Satizábal HF, Mota M, Noll D, Büchi L et al. Augmenting a convolutional neural network with local histograms - a case study in crop classification from high-resolution UAV imagery. In: European Symposium on Artificial Neural Networks; Bruges, Belgium; 2016: 515-520.
[20] Reyes AK, Caicedo JC, Camargo JE. Fine-tuning deep convolutional networks for plant recognition. In: Proceedings of the Working Notes of CLEF; Toulouse, France; 2015.
[21] Rußwurm M, Körner M. Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images. In: IEEE 2017 Conference on Computer Vision and Pattern Recognition Workshops; Honolulu, HI, USA; 2017: 1496–1504.
[22] Sun Y, Liu Y, Wang G, Zhang H. Deep learning for plant identification in natural environment. Computational Intelligence and Neuroscience 2017; 2017: 1–6. doi: 10.1155/2017/7361042
[23] Taghavi Namin S, Esmaeilzadeh M, Najafi M, Brown TB, Borevitz JO. Deep phenotyping: deep learning for temporal phenotype/genotype classification. Plant Methods 2018; 14: 66. doi:10.1186/s13007-018-0333-4
[24] Velumani K, Madec S, de Solan B, Lopez-Lozano R, Gillet J et al. An automatic method based on daily in situ images and deep learning to date wheat heading stage. Field Crops Research 2020; 252: 107793. doi: 10.1016/j.fcr.2020.107793
[25] Yalcin H. Plant phenology recognition using deep learning: Deep-Pheno. In: 6th International Conference on Agro-Geoinformatics (Agro-Geoinformatics); Fairfax, VA, USA; 2017: 1–5.
[26] Yalcin H, Razavi S. Plant classification using convolutional neural networks. In: 5th International Conference on Agro-Geoinformatics (Agro-Geoinformatics); Tianjin, China; 2016: 1–5.