Enhanced progressive fusion method for the efficient detection of multi-scale lightweight citrus fruits
Keywords:
citrus fruit detection, enhanced progressive fusion model, multi-scale lightweight, attention mechanismAbstract
Human labor efficiency has become unable to keep the pace with gradually annual citrus increasing production. Highly efficient and intelligent citrus picking and accurate yield estimation is the key to solve the problem. Success heavily depends on detection accuracy, prediction speed, and easy model deployment. Traditional target detection methods often fail to achieve balanced results in all those aspects. An improved YOLOv8 network model with four significant features is proposed. First, a lightweight FasterNet network structure was introduced to the backbone network, which reduced the number of parameters and computations while maintaining high-precision detection. Second, a progressive feature pyramid network AFPN structure was added to the neck network. Third, a parallel multi-branch attention mechanism PMBA was added before the detection head to improve the sensing ability after the feature fusion network. Fourth, a Wise-IoU was introduced to replace the original CIoU loss function to make the whole training process converge faster. Based on this, this study proposes an improved version of the YOLOv8 model: the FAP-YOLOv8. This improved model achieved an average accuracy (mAP@0.5) of 97.2% on the citrus datasets, with an accuracy that was 4.7% higher than the original YOLOv8, which was 19.2%, 7.4%, 5.1%, 4.9%, and 5.2% higher than the other models: Faster R-CNN, CenterNet, YOLOv5s, YOLOx-s, and YOLOv7, respectively. The number of parameters was reduced by 55.45%, the computation was reduced by 20% compared to the YOLOv8 benchmark, and the frame rate reached 46.51 fps to meet the detection requirements of lightweight networks. The experiments showed that the FAP-YOLOv8 models all outperformed the comparison models. Consequently, the proposed FAP-YOLOv8 model can help solve the citrus detection problem in orchards, which can be better applied to edge devices and provides strong support for intelligent orchard management. Keywords: citrus fruit detection, enhanced progressive fusion model, multi-scale lightweight, attention mechanism DOI: 10.25165/j.ijabe.20241706.8802 Citation: Zeng Y L, Lin Y, He Y T, Li T, Li J, Wang B J, et al. Enhanced progressive fusion method for the efficient detection of multi-scale lightweight citrus fruits. Int J Agric & Biol Eng, 2024; 17(6): 218–229.References
[1] Liu S Y. Artificial intelligence (AI) in agriculture. IT Professional, 2020; 22(3): 14–15.
[2] Tang Y C, Chen M Y, Wang C L, Luo L F, Li J H, Lian G P, et al. Recognition and localization methods for vision-based fruit picking robots: A review. Frontiers in Plant Science, 2020; 11: 510.
[3] Díaz I, Mazza S M, Combarro E F, Giménez L I, Gaiad J E. Machine learning applied to the prediction of citrus production. Spanish Journal of Agricultural Research, 2017; 15(2). doi: 10.5424/sjar/2017152-9090.
[4] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015; 521(7553): 436–444.
[5] Sa I, Ge Z Y, Dayoub F, Upcroft B, Perez T, McCool C. DeepFruits: A fruit detection system using deep neural networks. Sensors, 2016; 16(8): 1222.
[6] Bargoti S, Underwood J. Deep fruit detection in orchards. In: 2017 IEEE international conference on robotics and automation (ICRA), Singapore: IEEE, 2017; pp.3626–3633.
[7] Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.
[8] Koirala A, Walsh K B, Wang Z, McCarthy C. Deep learning - Method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 2019; 162: 219–234.
[9] Gao F F, Fu L S, Zhang X, Majeed Y, Li R, Karkee M, et al. Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN. Computers and Electronics in Agriculture, 2020; 176: 105634.
[10] Kukreja V, Dhiman P. A Deep Neural Network based disease detection scheme for Citrus fruits. In: 2020 International conference on smart electronics and communication (ICOSEC), Trichy, India: IEEE, 2020; pp.97–101.
[11] Horng G J, Liu M X, Chen C C. The smart image recognition mechanism for crop harvesting system in intelligent agriculture. IEEE Sensors Journal, 2019; 20(5): 2766–2781.
[12] Liu Z S, Xiang X Y, Qin J H, Tan Y, Zhang Q, Xiong N N. Image recognition of citrus diseases based on deep learning. CMC-Computers Materials & Continua, 2021; 66(1): 457–466.
[13] Qiu W J, Ye J, Hu L Q, Yang J, Li Q L, Mo J Y, et al. Distilled-MobileNet Model of convolutional neural network simplified structure for plant disease recognition. Smart Agriculture, 2021; 3(1): 109–117.
[14] Liu Y P, Yang C H, Ling H, Mabu S, Kuremoto T. A visual system of citrus picking robot using convolutional neural networks. In: 2018 5th international conference on systems and informatics (ICSAI), Nanjing, China: IEEE, 2018; pp.344–349.
[15] Lu J, Hu X W. Detecting green citrus fruit on trees in low light and complex background based on MSER and HCA. Transactions of the CSAE, 2017; 33(19): 196–201. (in Chinese)
[16] Bi S, Gao F, Chen J W, Zhang L. Detection method of citrus based on deep convolution neural network. Transactions of the CSAM, 2019; 50(5): 181–186. (in Chinese)
[17] Zhang W L, Wang J Q, Liu Y X, Chen K Z, Li H B, Duan Y L, et al. Deep-learning-based in-field citrus fruit detection and tracking. Horticulture Research, 2022; 9: uhac003.
[18] Zhuang J J, Luo S M, Hou C J, Tang Y, He Y, Xue X Y. Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications. Computers and Electronics in Agriculture, 2018; 152: 64–73.
[19] Lin G C, Tang Y C, Zou X J, Li J H, Xiong J T. In-field citrus detection and localisation based on RGB-D image analysis. Biosystems Engineering, 2019; 186: 34–44.
[20] Chen J Y, Liu H, Zhang Y T, Zhang D K, Ouyang H K, Chen X Y. A multiscale lightweight and efficient model based on YOLOv7: Applied to citrus orchard. Plants, 2022; 11(23): 3260.
[21] Lyu S L, Li R Y, Zhao Y W, Li Z, Fan R J, Liu S Y. Green citrus detection and counting in orchards based on YOLOv5-CS and AI edge system. Sensors, 2022; 22(2): 576.
[22] Yang H W, Liu Y Z, Wang S W, Qu H X, Li N, Wu J, et al. Improved apple fruit target recognition method based on YOLOv7 model. Agriculture, 2023; 13(7): 1278.
[23] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, 2016; pp.779–788.
[24] Redmon J, Farhadi A. Yolov3: An incremental improvement. arXiv preprint arXiv: 1804.02767, 2018. doi: 10.48550/arxiv.1804.02767.
[25] Bochkovskiy A, Wang C Y, Liao H Y M. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv: 2004.10934, 2020. doi: 10.48550/arXiv.2004.10934.
[26] Wang C Y, Bochkovskiy A, Liao H Y M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE, 2023; pp.7464–7475.
[27] Chen J R, Kao S-H, He H, Zhuo W P, Wen S, Lee C-H, et al. Run, don’t walk: chasing higher FLOPS for faster neural networks. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE, 2023; pp.12021–12031.
[28] Yang G Y, Lei J, Zhu Z K, Cheng S Y, Feng Z L, Liang R H. AFPN: Asymptotic feature pyramid network for object detection. In: 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Honolulu, Oahu, HI, USA: IEEE, 2023; 2184–2189.
[29] Liu Y C, Shao Z R, Teng Y Y, Hoffmann N. NAM: Normalization-based attention module. arXiv preprint arXiv: 2111.12419, 2021; doi: 10.48550/arXiv.2111.12419.
[30] Woo S, Park J, Lee J-Y, Kweon I S. Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), Munich, Germany: Springer, 2018; doi: 10.1007/978-3-030-01234-2_1.
[31] Wang Q L, Wu B G, Zhu P F, Li P H, Zuo W M, Hu Q H. ECA-Net: Efficient channel attention for deep convolutional neural networks. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA: IEEE, 2020; pp.11534–11542.
[32] Zheng Z H, Wang P, Liu W, Li J Z, Ye R G, Ren D W. Distance-IoU loss: Faster and better learning for bounding box regression. In: Proceedings of the AAAI conference on artificial intelligence, USA: AAAI Press, 2020; pp.12993–13000. doi: 10.1609/aaai.v34i07.6999.
[33] Tong Z J, Chen Y H, Xu Z W, Yu R. Wise-IoU: Bounding box regression loss with dynamic focusing mechanism. arXiv preprint arxiv: 2301.10051, doi: 10.48550/arXiv.2301.10051.
[34] Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137–1149.
[35] Duan K W, Bai S, Xie L X, Qi H G, Huang Q M, Tian Q. Centernet: Keypoint triplets for object detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South): IEEE, 2019; 6568–6577.
[36] Ultralytics/yolov5. 2021; Available: https://github.com/ ultralytics/yolov5. Accessed on [2023-04-25].
[37] Ge Z, Liu S T, Wang F, Li Z M, Sun J. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv: 2107.08430, 2021; doi: 10.48550/arXiv.2107.08430.
[38] ultralytics, 2023. Available: https://github.com/ultralytics/ultralytics. Accessed on [2023-04-19].
[2] Tang Y C, Chen M Y, Wang C L, Luo L F, Li J H, Lian G P, et al. Recognition and localization methods for vision-based fruit picking robots: A review. Frontiers in Plant Science, 2020; 11: 510.
[3] Díaz I, Mazza S M, Combarro E F, Giménez L I, Gaiad J E. Machine learning applied to the prediction of citrus production. Spanish Journal of Agricultural Research, 2017; 15(2). doi: 10.5424/sjar/2017152-9090.
[4] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015; 521(7553): 436–444.
[5] Sa I, Ge Z Y, Dayoub F, Upcroft B, Perez T, McCool C. DeepFruits: A fruit detection system using deep neural networks. Sensors, 2016; 16(8): 1222.
[6] Bargoti S, Underwood J. Deep fruit detection in orchards. In: 2017 IEEE international conference on robotics and automation (ICRA), Singapore: IEEE, 2017; pp.3626–3633.
[7] Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.
[8] Koirala A, Walsh K B, Wang Z, McCarthy C. Deep learning - Method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 2019; 162: 219–234.
[9] Gao F F, Fu L S, Zhang X, Majeed Y, Li R, Karkee M, et al. Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN. Computers and Electronics in Agriculture, 2020; 176: 105634.
[10] Kukreja V, Dhiman P. A Deep Neural Network based disease detection scheme for Citrus fruits. In: 2020 International conference on smart electronics and communication (ICOSEC), Trichy, India: IEEE, 2020; pp.97–101.
[11] Horng G J, Liu M X, Chen C C. The smart image recognition mechanism for crop harvesting system in intelligent agriculture. IEEE Sensors Journal, 2019; 20(5): 2766–2781.
[12] Liu Z S, Xiang X Y, Qin J H, Tan Y, Zhang Q, Xiong N N. Image recognition of citrus diseases based on deep learning. CMC-Computers Materials & Continua, 2021; 66(1): 457–466.
[13] Qiu W J, Ye J, Hu L Q, Yang J, Li Q L, Mo J Y, et al. Distilled-MobileNet Model of convolutional neural network simplified structure for plant disease recognition. Smart Agriculture, 2021; 3(1): 109–117.
[14] Liu Y P, Yang C H, Ling H, Mabu S, Kuremoto T. A visual system of citrus picking robot using convolutional neural networks. In: 2018 5th international conference on systems and informatics (ICSAI), Nanjing, China: IEEE, 2018; pp.344–349.
[15] Lu J, Hu X W. Detecting green citrus fruit on trees in low light and complex background based on MSER and HCA. Transactions of the CSAE, 2017; 33(19): 196–201. (in Chinese)
[16] Bi S, Gao F, Chen J W, Zhang L. Detection method of citrus based on deep convolution neural network. Transactions of the CSAM, 2019; 50(5): 181–186. (in Chinese)
[17] Zhang W L, Wang J Q, Liu Y X, Chen K Z, Li H B, Duan Y L, et al. Deep-learning-based in-field citrus fruit detection and tracking. Horticulture Research, 2022; 9: uhac003.
[18] Zhuang J J, Luo S M, Hou C J, Tang Y, He Y, Xue X Y. Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications. Computers and Electronics in Agriculture, 2018; 152: 64–73.
[19] Lin G C, Tang Y C, Zou X J, Li J H, Xiong J T. In-field citrus detection and localisation based on RGB-D image analysis. Biosystems Engineering, 2019; 186: 34–44.
[20] Chen J Y, Liu H, Zhang Y T, Zhang D K, Ouyang H K, Chen X Y. A multiscale lightweight and efficient model based on YOLOv7: Applied to citrus orchard. Plants, 2022; 11(23): 3260.
[21] Lyu S L, Li R Y, Zhao Y W, Li Z, Fan R J, Liu S Y. Green citrus detection and counting in orchards based on YOLOv5-CS and AI edge system. Sensors, 2022; 22(2): 576.
[22] Yang H W, Liu Y Z, Wang S W, Qu H X, Li N, Wu J, et al. Improved apple fruit target recognition method based on YOLOv7 model. Agriculture, 2023; 13(7): 1278.
[23] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, 2016; pp.779–788.
[24] Redmon J, Farhadi A. Yolov3: An incremental improvement. arXiv preprint arXiv: 1804.02767, 2018. doi: 10.48550/arxiv.1804.02767.
[25] Bochkovskiy A, Wang C Y, Liao H Y M. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv: 2004.10934, 2020. doi: 10.48550/arXiv.2004.10934.
[26] Wang C Y, Bochkovskiy A, Liao H Y M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE, 2023; pp.7464–7475.
[27] Chen J R, Kao S-H, He H, Zhuo W P, Wen S, Lee C-H, et al. Run, don’t walk: chasing higher FLOPS for faster neural networks. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada: IEEE, 2023; pp.12021–12031.
[28] Yang G Y, Lei J, Zhu Z K, Cheng S Y, Feng Z L, Liang R H. AFPN: Asymptotic feature pyramid network for object detection. In: 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Honolulu, Oahu, HI, USA: IEEE, 2023; 2184–2189.
[29] Liu Y C, Shao Z R, Teng Y Y, Hoffmann N. NAM: Normalization-based attention module. arXiv preprint arXiv: 2111.12419, 2021; doi: 10.48550/arXiv.2111.12419.
[30] Woo S, Park J, Lee J-Y, Kweon I S. Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), Munich, Germany: Springer, 2018; doi: 10.1007/978-3-030-01234-2_1.
[31] Wang Q L, Wu B G, Zhu P F, Li P H, Zuo W M, Hu Q H. ECA-Net: Efficient channel attention for deep convolutional neural networks. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA: IEEE, 2020; pp.11534–11542.
[32] Zheng Z H, Wang P, Liu W, Li J Z, Ye R G, Ren D W. Distance-IoU loss: Faster and better learning for bounding box regression. In: Proceedings of the AAAI conference on artificial intelligence, USA: AAAI Press, 2020; pp.12993–13000. doi: 10.1609/aaai.v34i07.6999.
[33] Tong Z J, Chen Y H, Xu Z W, Yu R. Wise-IoU: Bounding box regression loss with dynamic focusing mechanism. arXiv preprint arxiv: 2301.10051, doi: 10.48550/arXiv.2301.10051.
[34] Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137–1149.
[35] Duan K W, Bai S, Xie L X, Qi H G, Huang Q M, Tian Q. Centernet: Keypoint triplets for object detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South): IEEE, 2019; 6568–6577.
[36] Ultralytics/yolov5. 2021; Available: https://github.com/ ultralytics/yolov5. Accessed on [2023-04-25].
[37] Ge Z, Liu S T, Wang F, Li Z M, Sun J. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv: 2107.08430, 2021; doi: 10.48550/arXiv.2107.08430.
[38] ultralytics, 2023. Available: https://github.com/ultralytics/ultralytics. Accessed on [2023-04-19].
Downloads
Published
2024-12-24
How to Cite
Zeng, Y., Lin, Y., He, Y., Li, T., Li, J., Wang, B., & Yang, Y. (2024). Enhanced progressive fusion method for the efficient detection of multi-scale lightweight citrus fruits. International Journal of Agricultural and Biological Engineering, 17(6), 218–229. Retrieved from https://ijabe.migration.pkpps03.publicknowledgeproject.org/index.php/ijabe/article/view/8802
Issue
Section
Information Technology, Sensors and Control Systems
License
IJABE is an international peer reviewed open access journal, adopting Creative Commons Copyright Notices as follows.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
