Multi-class detection of cherry tomatoes using improved Yolov4-tiny model
Keywords:
y tomatoes, deep learning, data augmentation, YOLOv4, occlusion, multi-class detectionAbstract
The rapid and accurate detection of cherry tomatoes is of great significance to realizing automatic picking by robots. However, so far, cherry tomatoes are detected as only one class for picking. Fruits occluded by branches or leaves are detected as pickable objects, which may cause damage to the plant or robot end-effector during picking. This study proposed the Feature Enhancement Network Block (FENB) based on YOLOv4-Tiny to solve the above problem. Firstly, according to the distribution characteristics and picking strategies of cherry tomatoes, cherry tomatoes were divided into four classes in the nighttime, and daytime included not occluded, occluded by branches, occluded by fruits, and occluded by leaves. Secondly, the CSPNet structure with the hybrid attention mechanism was used to design the FENB, which pays more attention to the effective features of different classes of cherry tomatoes while retaining the original features. Finally, the Feature Enhancement Network (FEN) was constructed based on the FENB to enhance the feature extraction ability and improve the detection accuracy of YOLOv4-Tiny. The experimental results show that under the confidence of 0.5, average precision (AP) of non-occluded, branch-occluded, fruit-occluded, and leaf-occluded fruit over the day test images were 95.86%, 92.59%, 89.66%, and 84.99%, respectively, which were 98.43%, 95.62%, 95.50%, and 89.33% on the night test images, respectively. The mean Average Precision (mAP) of four classes over the night test set was higher (94.72%) than that of the day (90.78%), which were both better than YOLOv4 and YOLOv4-Tiny. It cost 32.22 ms to process a 416×416 image on the GPU. The model size was 39.34 MB. Therefore, the proposed model can provide a practical and feasible method for the multi-class detection of cherry tomatoes. Keywords: cherry tomatoes, deep learning, data augmentation, YOLOv4, occlusion, multi-class detection DOI: 10.25165/j.ijabe.20231602.7744 Citation: Zhang F, Chen Z J, Ali S, Yang N, Fu S L, Zhang Y K. Multi-class detection of cherry tomatoes using improved YOLOv4-Tiny. Int J Agric & Biol Eng, 2023; 16(2): 225-231.References
[1] Rong J C, Wang P B, Yang Q, Huang F. A field-tested harvesting robot for oyster mushroom in greenhouse. Agronomy, 2021; 11(6): 1210. doi: 10.3390/agronomy11061210.
[2] Zhang F, Chen Z J, Wang Y F, Bao R F, Chen X G, Fu S L, et al. Research on flexible end-effectors with humanoid grasp function for small spherical fruit picking. Agriculture, 2023; 13(1): 123. doi: 10.3390/agriculture13010123.
[3] Rong J C, Wang P B, Wang T J, Ling H, Yuan T. Fruit pose recognition and directional orderly grasping strategies for tomato harvesting robots. Computers and Electronics in Agriculture, 2022; 202: 107430. doi: 10.1016/j.compag.2022.107430.
[4] Afroza A, Ambreen N, Baseerat A; Nigeena N, Ahmad S P, Azrah I S, Amreena S; Insha J, Majid R. Evaluation of Cherry Tomato (Solanum lycopersicum L. var. cerasiforme) Genotypes for Yield and Quality Traits. Journal of Community Mobilization and Sustainable Development, 2021; 16(1): 72-76.
[5] Yamamoto K, Guo W, Ninomiya S S. Node detection and internode length estimation of tomato seedlings based on image analysis and machine learning. Sensors, 2016; 16(7): 1044.
[6] Wang Z L. Underwood J. Walsh KB. Machine vision assessment of mango orchard flowering. Computers and Electronics in Agriculture, 2018; 151: 501–511.
[7] Wu J G, Zhang B H, Zhou J, Xiong Y J, Gu B X, Yang X L. Automatic Recognition of Ripening Tomatoes by Combining Multi-Feature Fusion with a Bi-Layer Classification Strategy for Harvesting Robots. Sensors, 2019; 19(3): 612- 612.
[8] Xiong J T, Lin R, Liu Z, He Z L, Tang L Y, Yang Z G Zou X J. The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment. Biosystems Engineering, 2018; 166: 44-57.
[9] Bechar A, Vigneault C. Agricultural robots for field operations: Concepts and components. Biosystems Engineering, 2016; 149: 94-111.
[10] Silwal A, Davidson J R, Karkee M, Mo C K, Zhang Q, Lewis K. Design, integration, and field evaluation of a robotic apple harvester. Journal of Field Robotics, 2017; 34(6: 1140-1159.
[11] Silwal A, Karkee M, Zhang Q. A hierarchical approach to apple identification for robotic harvesting. Transactions of the ASABE, 2016; 59(5): 1079-1086.
[12] Guo Y M, Liu Y, Oerlemans A, Lao S Y, Lew M S. Deep learning for visual understanding: A review. Neurocomputing, 2016; 187: 27-48.
[13] He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; 770-778.
[14] Liu F, Liu Y K, Lin S, Guo W Z, Xu F, Zhang B. Fast recognition method for tomatoes under complex environments based on improved YOLO. Transactions of the Chinese Society for Agriculture Machinery, 2020; 51(6): 229-237. (in Chinese)
[15] Vougioukas S G. Annual Review of Control, Robotics, and Autonomous Systems. Annual Reviews, 2019; 2: 365-392.
[16] Zhang F, Chen Z J, Bao R F, Zhang C C, Wang Z H. Recognition of dense cherry tomatoes based on improved YOLOv4-LITE lightweight neural network. Transactions of the Chinese Society of Agricultural Engineering, 2021; 37(16): 270-278. (in Chinese)
[17] Xu Z F, Jia R S, Liu Y B, Zhao C Y, Sun H M. Fast Method of Detecting Tomatoes in a Complex Scene for Picking Robots. IEEE Access, 2020; 8: 55289 - 55299.
[18] Zhang W J, Zhao X X, Ding R R, Zhang Z, Jiang H H, Liu P Z. A Detection and Recognition Method for Tomato on Faster R-CNN Algorithm. Journal of Shandong Agricultural University (Natural Science Edition), 2021; 52(4): 624-630. (in Chinese)
[19] Xu C, Xiong Z, Jiang X P, Deng M, Huang G C. Design and research of the cluster tomato picking robot. Modern Agricultural Equipment, 2021; 42(6): 15-23. (in Chinese)
[20] Zhang Q, Liu F P, Jiang X P, Xiong Z, Xu C. Motion planning method and experiments of tomato bunch harvesting manipulator. Transactions of the CSAE, 2021; 39(7): 149-156. (in Chinese)
[21] Gao F F, Fu L S, Zhang X, Majeed Y, Li R, Karkee M, et al. Multi-class fruit-on-plant detection for apple in SNAP system using Faster RCNN. Computers and Electronics in Agriculture, 2020; 176: 105634. doi: 10.1016/j.compag.2020.105634.
[22] Suo R, Gao F F, Zhou Z X, Fu L X, Song Z Z, Dhupia J, et al. Improved multi-classes kiwifruit detection in orchard to avoid collisions during robotic picking. Computers and Electronics in Agriculture, 2021; 182: 106052. doi: 10.1016/j.compag.2021.106052.
[23] Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137-1149.
[24] Zhao D A, Wu R D, Liu X Y, Zhao Y Y. Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background. Transactions of the CSAE, 2019; 35(3): 164-173. (in Chinese)
[25] Wang C, Bochkovskiy A, Liao H M. Scaled-YOLOv4: Scaling cross stage partial network. In: 20221 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021; pp.13024-13033. doi: 10.1109/CVPR46437.2021.01283.
[26] Li H P, Li C Y, Li G B, Chen L X. A real-time table grape detection method based on improved YOLOv4-tiny network in complex background. Biosystems Engineering, 2021; 212: 347-359.
[27] Xu B, Wang N Y, Chen T Q, Li M. Empirical evaluation of rectified activations in convolutional network. arXiv preprint, 2015; arXiv:1505.00853.
[28] Zheng Z H, Wang P, Liu W, Li J Z, Ye R G, Ren D W. Distance-IoU loss: Faster and better learning for bounding box regression. arXiv preprint, 2020; arXiv:1911.08287.
[29] Wang C Y, Liao H M, Wu Y H, Chen P Y, Hsieh J W, Yeh I H. CSPNet: A new backbone that can enhance learning capability of CNN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020; pp.1571-1580.
[30] Woo S, Park J C, Lee J, Lweon I. CBAM: Convolutional Block Attention Module. In: Computer Vision - ECCV, 2018; pp.3-19.
[2] Zhang F, Chen Z J, Wang Y F, Bao R F, Chen X G, Fu S L, et al. Research on flexible end-effectors with humanoid grasp function for small spherical fruit picking. Agriculture, 2023; 13(1): 123. doi: 10.3390/agriculture13010123.
[3] Rong J C, Wang P B, Wang T J, Ling H, Yuan T. Fruit pose recognition and directional orderly grasping strategies for tomato harvesting robots. Computers and Electronics in Agriculture, 2022; 202: 107430. doi: 10.1016/j.compag.2022.107430.
[4] Afroza A, Ambreen N, Baseerat A; Nigeena N, Ahmad S P, Azrah I S, Amreena S; Insha J, Majid R. Evaluation of Cherry Tomato (Solanum lycopersicum L. var. cerasiforme) Genotypes for Yield and Quality Traits. Journal of Community Mobilization and Sustainable Development, 2021; 16(1): 72-76.
[5] Yamamoto K, Guo W, Ninomiya S S. Node detection and internode length estimation of tomato seedlings based on image analysis and machine learning. Sensors, 2016; 16(7): 1044.
[6] Wang Z L. Underwood J. Walsh KB. Machine vision assessment of mango orchard flowering. Computers and Electronics in Agriculture, 2018; 151: 501–511.
[7] Wu J G, Zhang B H, Zhou J, Xiong Y J, Gu B X, Yang X L. Automatic Recognition of Ripening Tomatoes by Combining Multi-Feature Fusion with a Bi-Layer Classification Strategy for Harvesting Robots. Sensors, 2019; 19(3): 612- 612.
[8] Xiong J T, Lin R, Liu Z, He Z L, Tang L Y, Yang Z G Zou X J. The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment. Biosystems Engineering, 2018; 166: 44-57.
[9] Bechar A, Vigneault C. Agricultural robots for field operations: Concepts and components. Biosystems Engineering, 2016; 149: 94-111.
[10] Silwal A, Davidson J R, Karkee M, Mo C K, Zhang Q, Lewis K. Design, integration, and field evaluation of a robotic apple harvester. Journal of Field Robotics, 2017; 34(6: 1140-1159.
[11] Silwal A, Karkee M, Zhang Q. A hierarchical approach to apple identification for robotic harvesting. Transactions of the ASABE, 2016; 59(5): 1079-1086.
[12] Guo Y M, Liu Y, Oerlemans A, Lao S Y, Lew M S. Deep learning for visual understanding: A review. Neurocomputing, 2016; 187: 27-48.
[13] He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; 770-778.
[14] Liu F, Liu Y K, Lin S, Guo W Z, Xu F, Zhang B. Fast recognition method for tomatoes under complex environments based on improved YOLO. Transactions of the Chinese Society for Agriculture Machinery, 2020; 51(6): 229-237. (in Chinese)
[15] Vougioukas S G. Annual Review of Control, Robotics, and Autonomous Systems. Annual Reviews, 2019; 2: 365-392.
[16] Zhang F, Chen Z J, Bao R F, Zhang C C, Wang Z H. Recognition of dense cherry tomatoes based on improved YOLOv4-LITE lightweight neural network. Transactions of the Chinese Society of Agricultural Engineering, 2021; 37(16): 270-278. (in Chinese)
[17] Xu Z F, Jia R S, Liu Y B, Zhao C Y, Sun H M. Fast Method of Detecting Tomatoes in a Complex Scene for Picking Robots. IEEE Access, 2020; 8: 55289 - 55299.
[18] Zhang W J, Zhao X X, Ding R R, Zhang Z, Jiang H H, Liu P Z. A Detection and Recognition Method for Tomato on Faster R-CNN Algorithm. Journal of Shandong Agricultural University (Natural Science Edition), 2021; 52(4): 624-630. (in Chinese)
[19] Xu C, Xiong Z, Jiang X P, Deng M, Huang G C. Design and research of the cluster tomato picking robot. Modern Agricultural Equipment, 2021; 42(6): 15-23. (in Chinese)
[20] Zhang Q, Liu F P, Jiang X P, Xiong Z, Xu C. Motion planning method and experiments of tomato bunch harvesting manipulator. Transactions of the CSAE, 2021; 39(7): 149-156. (in Chinese)
[21] Gao F F, Fu L S, Zhang X, Majeed Y, Li R, Karkee M, et al. Multi-class fruit-on-plant detection for apple in SNAP system using Faster RCNN. Computers and Electronics in Agriculture, 2020; 176: 105634. doi: 10.1016/j.compag.2020.105634.
[22] Suo R, Gao F F, Zhou Z X, Fu L X, Song Z Z, Dhupia J, et al. Improved multi-classes kiwifruit detection in orchard to avoid collisions during robotic picking. Computers and Electronics in Agriculture, 2021; 182: 106052. doi: 10.1016/j.compag.2021.106052.
[23] Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137-1149.
[24] Zhao D A, Wu R D, Liu X Y, Zhao Y Y. Apple positioning based on YOLO deep convolutional neural network for picking robot in complex background. Transactions of the CSAE, 2019; 35(3): 164-173. (in Chinese)
[25] Wang C, Bochkovskiy A, Liao H M. Scaled-YOLOv4: Scaling cross stage partial network. In: 20221 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021; pp.13024-13033. doi: 10.1109/CVPR46437.2021.01283.
[26] Li H P, Li C Y, Li G B, Chen L X. A real-time table grape detection method based on improved YOLOv4-tiny network in complex background. Biosystems Engineering, 2021; 212: 347-359.
[27] Xu B, Wang N Y, Chen T Q, Li M. Empirical evaluation of rectified activations in convolutional network. arXiv preprint, 2015; arXiv:1505.00853.
[28] Zheng Z H, Wang P, Liu W, Li J Z, Ye R G, Ren D W. Distance-IoU loss: Faster and better learning for bounding box regression. arXiv preprint, 2020; arXiv:1911.08287.
[29] Wang C Y, Liao H M, Wu Y H, Chen P Y, Hsieh J W, Yeh I H. CSPNet: A new backbone that can enhance learning capability of CNN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020; pp.1571-1580.
[30] Woo S, Park J C, Lee J, Lweon I. CBAM: Convolutional Block Attention Module. In: Computer Vision - ECCV, 2018; pp.3-19.
Downloads
Published
2023-05-12
How to Cite
Zhang, F., Chen, Z., Ali, S., Yang, N., Fu, S., & Zhang, Y. (2023). Multi-class detection of cherry tomatoes using improved Yolov4-tiny model. International Journal of Agricultural and Biological Engineering, 16(2), 225–231. Retrieved from https://ijabe.migration.pkpps03.publicknowledgeproject.org/index.php/ijabe/article/view/7744
Issue
Section
Information Technology, Sensors and Control Systems
License
IJABE is an international peer reviewed open access journal, adopting Creative Commons Copyright Notices as follows.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
