Novel green-fruit detection algorithm based on D2D framework
Keywords:
green-fruit detection, D2D framework, automatic harvesting, MobileNetV2 FPN, binary mask prediction, anchor-freeAbstract
In the complex orchard environment, the efficient and accurate detection of object fruit is the basic requirement to realize the orchard yield measurement and automatic harvesting. Sometimes it is hard to differentiate between the object fruits and the background because of the similar color, and it is challenging due to the ambient light and camera angle by which the photos have been taken. These problems make it hard to detect green fruits in orchard environments. In this study, a two-stage dense to detection framework (D2D) was proposed to detect green fruits in orchard environments. The proposed model was based on multi-scale feature extraction of target fruit by using feature pyramid networks MobileNetV2 +FPN structure and generated region proposal of target fruit by using Region Proposal Network (RPN) structure. In the regression branch, the offset of each local feature was calculated, and the positive and negative samples of the region proposals were predicted by a binary mask prediction to reduce the interference of the background to the prediction box. In the classification branch, features were extracted from each sub-region of the region proposal, and features with distinguishing information were obtained through adaptive weighted pooling to achieve accurate classification. The new proposed model adopted an anchor-free frame design, which improves the generalization ability, makes the model more robust, and reduces the storage requirements. The experimental results of persimmon and green apple datasets show that the new model has the best detection performance, which can provide theoretical reference for other green object detection. Keywords: green-fruit detection, D2D framework, automatic harvesting, MobileNetV2+FPN, binary mask prediction, anchor-free DOI: 10.25165/j.ijabe.20221501.6943 Citation: Wei J M, Ding Y H, Liu J, Ullah M Z, Yin X, Jia W K. Novel green-fruit detection algorithm based on D2D framework. Int J Agric & Biol Eng, 2022; 15(1): 251–259.References
[1] Jia W, Zhang Y, Lian J, Zheng Y J, Zhao D A, Li C J. Apple harvesting robot under information technology: A review. International Journal of Advanced Robotic Systems, 2020; 17(3): 1729881420925310. doi: 10.1177/179881420925310.
[2] Xiong Y, Ge Y, Grimstad L, From P J. An autonomous
strawberry-harvesting robot: Design, development, integration, and field evaluation. Journal of Field Robotics, 2020; 37(2): 202–224.
[3] Tang Y C, Chen M Y, Wang C L, Luo L F, Li J H, Lian G P, et al. Recognition and localization methods for vision–based fruit picking robots: A review. Frontiers in Plant Science, 2020; 11: 510. doi: 10.3389/ fpls.2020.00510.
[4] Fu L S, Gao F F, Wu J Z, Li R, Karkee M, Zhang Q. Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review. Computers and Electronics in Agriculture, 2020; 177: 105687. doi: 10.1016/j.compag.2020.105687.
[5] Ilea D E, Whelan P F. Image segmentation based on the integration of colour–texture descriptors-A review. Pattern Recognition, 2011; 44(10-11): 2479–2501.
[6] Sharma P, Suji J. A review on image segmentation with its clustering techniques. International Journal of Signal Processing, Image Processing and Pattern Recognition, 2016; 9(5): 209–218.
[7] Jia W K, Zheng Y J, Zhao D A, Yin Xiang, Liu X Y, Du R C. Preprocessing method of night vision image application in apple harvesting robot. Int J Agric & Biol Eng, 2018; 11(2): 158–163.
[8] Arefi A, Motlagh A M, Mollazade K, Teimourlou R F. Recognition and localization of ripen tomato based on machine vision. Australian Journal of Crop Science, 2011; 5(10): 1144–1149.
[9] Linker R, Cohen O, Naor A. Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture, 2012; 81: 45–57.
[10] Liao W, Zheng L H, Li M Z, Sun H, Yang W. Green apple recognition in natural illuminations based on random forest algorithm. Transactions of the Chinese Society for Agricultural Machinery, 2017; 48(S1): 86–91. (in Chinese)
[11] Tian Y Y, Duan H C, Luo R, Zhang Y, Jia W K, Lian J, et al. Fast recognition and location of target fruit based on depth information. IEEE Access, 2019; 7: 170553–170563.
[12] Li B R, Long Y, Song H B. Detection of green apples in natural scenes based on saliency theory and Gaussian curve fitting. Int J Agric & Biol Eng, 2018; 11(1): 192–198.
[13] Koirala A, Walsh K B, Wang Z L, McCarthy C. Deep learning-method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 2019; 162: 219–234.
[14] Boogaard F P, Rongen K H, Kootstra G W. Robust node detection and tracking in fruit-vegetable crops using deep learning and multi-view imaging. Biosystems Engineering, 2020; 192: 117–132.
[15] Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.
[16] Kang H, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture, 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.
[17] Jia W K, Tian Y Y, Luo R, Zhang Z H, Lian J, Zheng Y J. Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot. Computers and Electronics in Agriculture, 2020; 172: 105380. doi: 10.1016/j.compag.2020.105380.
[18] Wang D, He D. Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network. Transactions of the CSAE, 2019; 35(3): 156–163. (in Chinese)
[19] Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L C. MobilenetV2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Lake City, USA: IEEE, 2018; pp.4510–4520. doi: 10.1109/CVPR.2018.00474.
[20] Lin T Y, Dollár P, Girshick R, He B. Feature pyramid networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA: IEEE, 2017; pp.935–944. doi: 10.1109/CVPR.207.106.
[21] Girshick R. Fast R-CNN. In: 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile: IEEE, 2015; pp.1440–1448.
[22] Cao J, Cholakkal H, Anwer R M, Khan F S, Pang Y, Shao L. D2Det: Towards high quality object detection and instance segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020; pp.11485–11494. doi: 10.1109/CVPR42600.2020.01150.
[23] Howard A, Sandler M, Chu G, Chen L C, Chen B, Tan M X, et al. Searching for MobilenetV3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE, 2019; pp.1314–1324. arXiv: 1905.02244v1.
[24] Zhang Y Q, Chu J, Leng L, Miao J. Mask-refined R-CNN: A network for
refining object details in instance segmentation. Sensors, 2020; 20(4): 1010. doi: 10.3390/s20041010.
[25] Zhang K, Sun M, Han T X, Yuan X F, Guo L R, Liu T. Residual networks of residual networks: Multilevel residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 2017; 28(6): 1303–1314.
[26] Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016; 39(6): 1137–1149.
[27] Tian Z, Shen C H, Chen H, He T. FCOS: Fully Convolutional One-stage Object Detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019; pp. 9627–9636. doi: 10.1109/
ICCV.2019.00972.
[28] Dai J F, Qi H Z, Xiong Y W, Li Y, Zhang G D, Hu H, et al. Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), 2017; pp.764–773. doi: 10.1109/ICCV.2017.89.
[29] He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. 2017; pp.2961–2969.
[30] Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y. SSD: Single shot multibox detector. In: Computer Vision-ECCV 2016. Lecture Notes in Computer Science, Springer, Cham, 2016; 9905: 21–37. doi: 10.1007/978-3-319-46448-0_2.
[2] Xiong Y, Ge Y, Grimstad L, From P J. An autonomous
strawberry-harvesting robot: Design, development, integration, and field evaluation. Journal of Field Robotics, 2020; 37(2): 202–224.
[3] Tang Y C, Chen M Y, Wang C L, Luo L F, Li J H, Lian G P, et al. Recognition and localization methods for vision–based fruit picking robots: A review. Frontiers in Plant Science, 2020; 11: 510. doi: 10.3389/ fpls.2020.00510.
[4] Fu L S, Gao F F, Wu J Z, Li R, Karkee M, Zhang Q. Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review. Computers and Electronics in Agriculture, 2020; 177: 105687. doi: 10.1016/j.compag.2020.105687.
[5] Ilea D E, Whelan P F. Image segmentation based on the integration of colour–texture descriptors-A review. Pattern Recognition, 2011; 44(10-11): 2479–2501.
[6] Sharma P, Suji J. A review on image segmentation with its clustering techniques. International Journal of Signal Processing, Image Processing and Pattern Recognition, 2016; 9(5): 209–218.
[7] Jia W K, Zheng Y J, Zhao D A, Yin Xiang, Liu X Y, Du R C. Preprocessing method of night vision image application in apple harvesting robot. Int J Agric & Biol Eng, 2018; 11(2): 158–163.
[8] Arefi A, Motlagh A M, Mollazade K, Teimourlou R F. Recognition and localization of ripen tomato based on machine vision. Australian Journal of Crop Science, 2011; 5(10): 1144–1149.
[9] Linker R, Cohen O, Naor A. Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture, 2012; 81: 45–57.
[10] Liao W, Zheng L H, Li M Z, Sun H, Yang W. Green apple recognition in natural illuminations based on random forest algorithm. Transactions of the Chinese Society for Agricultural Machinery, 2017; 48(S1): 86–91. (in Chinese)
[11] Tian Y Y, Duan H C, Luo R, Zhang Y, Jia W K, Lian J, et al. Fast recognition and location of target fruit based on depth information. IEEE Access, 2019; 7: 170553–170563.
[12] Li B R, Long Y, Song H B. Detection of green apples in natural scenes based on saliency theory and Gaussian curve fitting. Int J Agric & Biol Eng, 2018; 11(1): 192–198.
[13] Koirala A, Walsh K B, Wang Z L, McCarthy C. Deep learning-method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 2019; 162: 219–234.
[14] Boogaard F P, Rongen K H, Kootstra G W. Robust node detection and tracking in fruit-vegetable crops using deep learning and multi-view imaging. Biosystems Engineering, 2020; 192: 117–132.
[15] Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.
[16] Kang H, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture, 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.
[17] Jia W K, Tian Y Y, Luo R, Zhang Z H, Lian J, Zheng Y J. Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot. Computers and Electronics in Agriculture, 2020; 172: 105380. doi: 10.1016/j.compag.2020.105380.
[18] Wang D, He D. Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network. Transactions of the CSAE, 2019; 35(3): 156–163. (in Chinese)
[19] Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L C. MobilenetV2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Lake City, USA: IEEE, 2018; pp.4510–4520. doi: 10.1109/CVPR.2018.00474.
[20] Lin T Y, Dollár P, Girshick R, He B. Feature pyramid networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA: IEEE, 2017; pp.935–944. doi: 10.1109/CVPR.207.106.
[21] Girshick R. Fast R-CNN. In: 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile: IEEE, 2015; pp.1440–1448.
[22] Cao J, Cholakkal H, Anwer R M, Khan F S, Pang Y, Shao L. D2Det: Towards high quality object detection and instance segmentation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020; pp.11485–11494. doi: 10.1109/CVPR42600.2020.01150.
[23] Howard A, Sandler M, Chu G, Chen L C, Chen B, Tan M X, et al. Searching for MobilenetV3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE, 2019; pp.1314–1324. arXiv: 1905.02244v1.
[24] Zhang Y Q, Chu J, Leng L, Miao J. Mask-refined R-CNN: A network for
refining object details in instance segmentation. Sensors, 2020; 20(4): 1010. doi: 10.3390/s20041010.
[25] Zhang K, Sun M, Han T X, Yuan X F, Guo L R, Liu T. Residual networks of residual networks: Multilevel residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 2017; 28(6): 1303–1314.
[26] Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016; 39(6): 1137–1149.
[27] Tian Z, Shen C H, Chen H, He T. FCOS: Fully Convolutional One-stage Object Detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019; pp. 9627–9636. doi: 10.1109/
ICCV.2019.00972.
[28] Dai J F, Qi H Z, Xiong Y W, Li Y, Zhang G D, Hu H, et al. Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), 2017; pp.764–773. doi: 10.1109/ICCV.2017.89.
[29] He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. 2017; pp.2961–2969.
[30] Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C Y. SSD: Single shot multibox detector. In: Computer Vision-ECCV 2016. Lecture Notes in Computer Science, Springer, Cham, 2016; 9905: 21–37. doi: 10.1007/978-3-319-46448-0_2.
Downloads
Published
2022-02-26
How to Cite
Wei, J., Ding, Y., Liu, J., Ullah, M. Z., Yin, X., & Jia, W. (2022). Novel green-fruit detection algorithm based on D2D framework. International Journal of Agricultural and Biological Engineering, 15(1), 251–259. Retrieved from https://ijabe.migration.pkpps03.publicknowledgeproject.org/index.php/ijabe/article/view/6943
Issue
Section
Information Technology, Sensors and Control Systems
License
IJABE is an international peer reviewed open access journal, adopting Creative Commons Copyright Notices as follows.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
