Automatic detection of pecan fruits based on Faster RCNN with FPN in orchard
Keywords:
pecan fruit, fruit detection, Faster RCNN, FPN, uneven illumination correctionAbstract
Although the development of the robot picking vision system is widely applied, it is very challenging for fruit detection in orchards with complex light and environment, especially for fruit colors similar to the background. In recent, there are few studies on pecan fruit detection and location based on machine vision. In this study, an accurate and efficient pecan fruit detection method was proposed based on machine vision under natural pecan orchards. In order to solve the illumination problem, a light compensation algorithm was first utilized to process the collected samples, and then an improved Faster Region Convolutional Neural Network (Faster RCNN) with the Feature Pyramid Networks (FPN) was established to train the samples. Finally, the pecan number counting method was introduced to count the number of pecan. A total of 241 pecan images were tested, and comparison experiments were carried out. The mean average precision (mAP) of the proposed detection method was 95.932%, compared with the result without uneven illumination correction (UIC), which was increased by 0.849%, while the mAP of the Single Shot Detector (SSD)+FPN was 92.991%. In addition, the number of clusters was counted using the proposed method with an accuracy rate of 93.539% compared with the actual clusters. The results demonstrate that the proposed network has good robustness for pecan fruit detection in different illumination and various unstructured environments, and the experimental achievement has great potential for robot-picking visual systems. Keywords: pecan fruit, fruit detection, Faster RCNN, FPN, uneven illumination correction DOI: 10.25165/j.ijabe.20221506.7241 Citation: Hu C H, Shi Z F, Wei H L, Hu X D, Xie Y N, Li P P. Automatic detection of pecan fruits based on Faster RCNN with FPN in orchard. Int J Agric & Biol Eng, 2022; 15(6): 189–196.References
[1] Kurtulmus F, Lee W S, Vardar A. Immature peach detection in colour images acquired in natural illumination conditions using statistical classifiers and neural network. Precision Agriculture, 2014; 15: 57–79.
[2] Yang C, Lee W S, Williamson J G. Classification of blueberry fruit and leaves based on spectral signatures. Biosystems Engineering, 2012; 113(4): 351–362.
[3] Stajnko D, Lakota, M, Hočevar M. Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Computers and Electronics in Agriculture, 2004; 42: 31–42.
[4] Méndez Perez R, Cheein F A, Rosell-Polo J R. Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry. Computers and Electronics in Agriculture, 2017; 139: 231–242.
[5] Tsoulias N, Paraforos D S, Xanthopoulo G, Zude-Sasse M. Apple shape detection based on geometric and radiometric features using a LiDAR laser scanner. Remote Sensing, 2020; 12(15): 2481. doi: 10.3390/rs12152481.
[6] Zhou R, Damerow L, Sun Y, Blanke M M. Using colour features of cv. ‘Gala’ apple fruits in an orchard in image processing to predict yield. Precision Agriculture, 2012; 13: 568–580.
[7] Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.
[8] Liakos K G, Busato P, Moshou D, Pearson S, Bochtis D. Machine learning in agriculture: A review. Sensors, 2018; 18(8): 2674. doi: 10.3390/s18082674.
[9] Wang C L, Tang Y C, Zou X J, SiTu W M. A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot. Optik, 2017; 131: 626–631.
[10] Zhao C Y, Lee W S, He D J. Immature green citrus detection based on colour feature and sum of absolute transformed difference (SATD) using colour images in the citrus grove. Computers and Electronics in Agriculture, 2016; 124: 243–253. doi: 10.1016/j.ijleo.2016.11.177.
[11] Ye Q, Huang P, Zhang Z, Zheng Y, Fu L, Yang W. Multiview learning with robust double-sided twin SVM. IEEE Transactions on Cybernetics, 2021; 52(12): 12745-12758.
[12] Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus: IEEE, 2014; pp.580–587. doi: 10.1109/CVPR.2014.81.
[13] Ren S, He K, Girshick, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transaction Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137–1149.
[14] He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN, 2017. doi: 10.1109/TPAMI.2018.2844175.
[15] Wang J M, Chen X X, Cao L, An F, Chen B Q, Xue L F, et al. Individual rubber tree segmentation based on ground-based LiDAR data and faster R-CNN of Deep Learning. Forests 2019; 10: 793. doi: 10.3390/f10090793.
[16] Sun J, He X, Ge X, Wu X, Shen J, Song Y. Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture 2018; 8: 196. doi: 10.3390/agriculture8120196.
[17] Kang H, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.
[18] Redmon J, Divvala S, Girshick R, Farhadi A. You Only Look Once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; pp.779-788. doi: 10.1109/CVPR.2016.91.
[19] Fu C Y, Liu W, Ranga A, Tyagi A, Berg A C. DSSD: Deconvolutional Single Shot Detector, arXiv, 2017; arXiv: 1701.06659.
[20] Kamilaris A, Prenafeta-Boldú F X. Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 2018; 147: 70–90.
[21] Yu Y, Zhang K L, Zhang D X, Yang L, Cui T. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Computers and Electronics in Agriculture, 2019; 163: 104846. doi: 10.1016/j.compag.2019.06.001.
[22] Vasconez J P, Delpiano J, Vougioukas S, Auat Cheein F. Comparison of convolutional neural networks in fruit detection and counting: A comprehensive evaluation. Computers and Electronics in Agriculture, 2020; 173: 105348. doi: 10.1016/j.compag.2020.105348.
[23] Tian Y N, Yang G D, Wang Z, Wang H, Li E, Liang Z Z. Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Computers and Electronics in Agriculture, 2019; 157: 417–426.
[24] Koirala A, Walsh K B, Wang Z, McCarthy C. Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO’. Precision Agriculture, 2019; 20: 1107–1135.
[25] He K M, Sun J, Tang X O. Single image haze removal using dark
channel prior. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami: IEEE, 2009; pp.1956–1963. doi: 10.1109/TPAMI.2010.168.
[26] Guo C L, Li C Y, Guo J C, Loy C C, Hou J H, Sam K W. Zero-reference deep curve estimation for low-light image enhancement. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020; pp.1777-1786. doi: 10.1109/CVPR42600.2020.00185.
[27] Everingham M, Zisserman A, Williams C K I, Van Gool L, Allan M, Bishop C M, Chapelle O, et al. The 2005 PASCAL visual object classes challenge. In: Machine Learning Recognising Tectual Entailment, MLCW 2005, Springer, 2005; 3944: pp.117-176. doi: 10.1007/11736790_8.
[28] He K M, Zhang X Y, Ren S Q, Sun J. Identity mappings in deep residual networks. In: Computer Vision – ECCV 2016, Springer, 2016; 9908: 630-645. doi: 10.1007/978-3-319-46493-0_38.
[29] Fu L Y, Zhang D< Ye Q L. Recurrent thrifty attention network for remote sensing scene recognition. IEEE Transactions on Geoscience and Remote Sensing, 2021; 59(10): 8257–8268. .
[30] Lin T Y, Dollar P, Girshick R, He K, Hariharan B, Belongie S. Feature Pyramid Networks for Object Detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu: IEEE, 2017; pp.936–944. doi: 10.1109/CVPR.2017.106.
[31] Yang Q M, Xiao D Q, Lin S C. Feeding behavior recognition for group-housed pigs with the Faster R-CNN. Computers and Electronics in Agriculture, 2018; 155: 453–460. doi: 10.1016/j.compag.2018.11.002.
[32] Vahidi H, Klinkenberg B, Johnson B A, Moskal L M, Yan W. Mapping the individual trees in urban orchards by incorporating volunteered geographic information and very high resolution optical remotely sensed data: A template matching-based approach. Remote Sensensing, 2018; 10(7): 1134. doi: 10.3390/rs10071134.
[33] Fan P, Lang G D, Yan B, Lei X Y, Guo P J, Liu Z J, et al. A method of segmenting apples based on gray-centered RGB color space. Remote Sensing, 2021; 13(6): 1211. doi: 10.3390/rs13061211.
[34] Kang H W, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture, 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.
[35] Zhuang J J, Luo S M, Hou C J, Tang Y, He Y, Xue X Y. Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications. Computers and Electronics in Agriculture, 2018; 152: 64–73.
[36] Sun J, He X F, Ge X, Wu X H, Shen J F, Song Y Y. Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture, 2018; 8(12): 196. doi: 10.3390/agriculture8120196.
[37] Liang C X, Xiong J T, Zheng Z H, Zhong Z, Li Z H, Chen S M, et al. A visual detection method for nighttime litchi fruits and fruiting stems. Computers and Electronics in Agriculture, 2020; 169(6): 105192. doi: 10.1016/j.compag.2019.105192.
[38] Wan Nurazwin Syazwani R, Muhammad Asraf H, Megat Syahirul Amin M A, Nur Dalila K A. Automated image identification, detection and fruit counting of top-view pineapple crown using machine learning. Alexandria Engineering Journal, 2022; 61: 1265–1276.
[39] Chen T C, Zhang R H, Zhu L X, Zhang S A, Li X M. A method of fast segmentation for banana stalk exploited lightweight multi-feature fusion deep neural network. Machines, 2021; 9(3): 66. doi: 10.3390/ machines9030066.
[2] Yang C, Lee W S, Williamson J G. Classification of blueberry fruit and leaves based on spectral signatures. Biosystems Engineering, 2012; 113(4): 351–362.
[3] Stajnko D, Lakota, M, Hočevar M. Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Computers and Electronics in Agriculture, 2004; 42: 31–42.
[4] Méndez Perez R, Cheein F A, Rosell-Polo J R. Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry. Computers and Electronics in Agriculture, 2017; 139: 231–242.
[5] Tsoulias N, Paraforos D S, Xanthopoulo G, Zude-Sasse M. Apple shape detection based on geometric and radiometric features using a LiDAR laser scanner. Remote Sensing, 2020; 12(15): 2481. doi: 10.3390/rs12152481.
[6] Zhou R, Damerow L, Sun Y, Blanke M M. Using colour features of cv. ‘Gala’ apple fruits in an orchard in image processing to predict yield. Precision Agriculture, 2012; 13: 568–580.
[7] Bargoti S, Underwood J P. Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 2017; 34(6): 1039–1060.
[8] Liakos K G, Busato P, Moshou D, Pearson S, Bochtis D. Machine learning in agriculture: A review. Sensors, 2018; 18(8): 2674. doi: 10.3390/s18082674.
[9] Wang C L, Tang Y C, Zou X J, SiTu W M. A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot. Optik, 2017; 131: 626–631.
[10] Zhao C Y, Lee W S, He D J. Immature green citrus detection based on colour feature and sum of absolute transformed difference (SATD) using colour images in the citrus grove. Computers and Electronics in Agriculture, 2016; 124: 243–253. doi: 10.1016/j.ijleo.2016.11.177.
[11] Ye Q, Huang P, Zhang Z, Zheng Y, Fu L, Yang W. Multiview learning with robust double-sided twin SVM. IEEE Transactions on Cybernetics, 2021; 52(12): 12745-12758.
[12] Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus: IEEE, 2014; pp.580–587. doi: 10.1109/CVPR.2014.81.
[13] Ren S, He K, Girshick, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transaction Pattern Analysis and Machine Intelligence, 2017; 39(6): 1137–1149.
[14] He K, Gkioxari G, Dollár P, Girshick R. Mask R-CNN, 2017. doi: 10.1109/TPAMI.2018.2844175.
[15] Wang J M, Chen X X, Cao L, An F, Chen B Q, Xue L F, et al. Individual rubber tree segmentation based on ground-based LiDAR data and faster R-CNN of Deep Learning. Forests 2019; 10: 793. doi: 10.3390/f10090793.
[16] Sun J, He X, Ge X, Wu X, Shen J, Song Y. Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture 2018; 8: 196. doi: 10.3390/agriculture8120196.
[17] Kang H, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.
[18] Redmon J, Divvala S, Girshick R, Farhadi A. You Only Look Once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; pp.779-788. doi: 10.1109/CVPR.2016.91.
[19] Fu C Y, Liu W, Ranga A, Tyagi A, Berg A C. DSSD: Deconvolutional Single Shot Detector, arXiv, 2017; arXiv: 1701.06659.
[20] Kamilaris A, Prenafeta-Boldú F X. Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 2018; 147: 70–90.
[21] Yu Y, Zhang K L, Zhang D X, Yang L, Cui T. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Computers and Electronics in Agriculture, 2019; 163: 104846. doi: 10.1016/j.compag.2019.06.001.
[22] Vasconez J P, Delpiano J, Vougioukas S, Auat Cheein F. Comparison of convolutional neural networks in fruit detection and counting: A comprehensive evaluation. Computers and Electronics in Agriculture, 2020; 173: 105348. doi: 10.1016/j.compag.2020.105348.
[23] Tian Y N, Yang G D, Wang Z, Wang H, Li E, Liang Z Z. Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Computers and Electronics in Agriculture, 2019; 157: 417–426.
[24] Koirala A, Walsh K B, Wang Z, McCarthy C. Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO’. Precision Agriculture, 2019; 20: 1107–1135.
[25] He K M, Sun J, Tang X O. Single image haze removal using dark
channel prior. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami: IEEE, 2009; pp.1956–1963. doi: 10.1109/TPAMI.2010.168.
[26] Guo C L, Li C Y, Guo J C, Loy C C, Hou J H, Sam K W. Zero-reference deep curve estimation for low-light image enhancement. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020; pp.1777-1786. doi: 10.1109/CVPR42600.2020.00185.
[27] Everingham M, Zisserman A, Williams C K I, Van Gool L, Allan M, Bishop C M, Chapelle O, et al. The 2005 PASCAL visual object classes challenge. In: Machine Learning Recognising Tectual Entailment, MLCW 2005, Springer, 2005; 3944: pp.117-176. doi: 10.1007/11736790_8.
[28] He K M, Zhang X Y, Ren S Q, Sun J. Identity mappings in deep residual networks. In: Computer Vision – ECCV 2016, Springer, 2016; 9908: 630-645. doi: 10.1007/978-3-319-46493-0_38.
[29] Fu L Y, Zhang D< Ye Q L. Recurrent thrifty attention network for remote sensing scene recognition. IEEE Transactions on Geoscience and Remote Sensing, 2021; 59(10): 8257–8268. .
[30] Lin T Y, Dollar P, Girshick R, He K, Hariharan B, Belongie S. Feature Pyramid Networks for Object Detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu: IEEE, 2017; pp.936–944. doi: 10.1109/CVPR.2017.106.
[31] Yang Q M, Xiao D Q, Lin S C. Feeding behavior recognition for group-housed pigs with the Faster R-CNN. Computers and Electronics in Agriculture, 2018; 155: 453–460. doi: 10.1016/j.compag.2018.11.002.
[32] Vahidi H, Klinkenberg B, Johnson B A, Moskal L M, Yan W. Mapping the individual trees in urban orchards by incorporating volunteered geographic information and very high resolution optical remotely sensed data: A template matching-based approach. Remote Sensensing, 2018; 10(7): 1134. doi: 10.3390/rs10071134.
[33] Fan P, Lang G D, Yan B, Lei X Y, Guo P J, Liu Z J, et al. A method of segmenting apples based on gray-centered RGB color space. Remote Sensing, 2021; 13(6): 1211. doi: 10.3390/rs13061211.
[34] Kang H W, Chen C. Fruit detection, segmentation and 3D visualisation of environments in apple orchards. Computers and Electronics in Agriculture, 2020; 171: 105302. doi: 10.1016/j.compag.2020.105302.
[35] Zhuang J J, Luo S M, Hou C J, Tang Y, He Y, Xue X Y. Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications. Computers and Electronics in Agriculture, 2018; 152: 64–73.
[36] Sun J, He X F, Ge X, Wu X H, Shen J F, Song Y Y. Detection of key organs in tomato based on deep migration learning in a complex background. Agriculture, 2018; 8(12): 196. doi: 10.3390/agriculture8120196.
[37] Liang C X, Xiong J T, Zheng Z H, Zhong Z, Li Z H, Chen S M, et al. A visual detection method for nighttime litchi fruits and fruiting stems. Computers and Electronics in Agriculture, 2020; 169(6): 105192. doi: 10.1016/j.compag.2019.105192.
[38] Wan Nurazwin Syazwani R, Muhammad Asraf H, Megat Syahirul Amin M A, Nur Dalila K A. Automated image identification, detection and fruit counting of top-view pineapple crown using machine learning. Alexandria Engineering Journal, 2022; 61: 1265–1276.
[39] Chen T C, Zhang R H, Zhu L X, Zhang S A, Li X M. A method of fast segmentation for banana stalk exploited lightweight multi-feature fusion deep neural network. Machines, 2021; 9(3): 66. doi: 10.3390/ machines9030066.
Downloads
Published
2022-12-27
How to Cite
Hu, C., Shi, Z., Wei, H., Hu, X., Xie, Y., & Li, P. (2022). Automatic detection of pecan fruits based on Faster RCNN with FPN in orchard. International Journal of Agricultural and Biological Engineering, 15(6), 189–196. Retrieved from https://ijabe.migration.pkpps03.publicknowledgeproject.org/index.php/ijabe/article/view/7241
Issue
Section
Information Technology, Sensors and Control Systems
License
IJABE is an international peer reviewed open access journal, adopting Creative Commons Copyright Notices as follows.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
