Detection method for the cucumber robotic grasping pose in clutter scenarios via instance segmentation
Keywords:
Clutter scenarios, Cucumber grasp, Convolutional neural network, Instance segmentationAbstract
The application of robotic grasping for agricultural products pushes automation in agriculture-related industries. Cucumber, a common vegetable in greenhouses and supermarkets, often needs to be grasped from a cluttered scene. In order to realize efficient grasping in cluttered scenes, a fully automatic cucumber recognition, grasping, and palletizing robot system was constructed in this paper. The system adopted Yolact++ deep learning network to segment cucumber instances. An early fusion method of F-RGBD was proposed, which increases the algorithm's discriminative ability for these appearance-similar cucumbers at different depths, and at different occlusion degrees. The results of the comparative experiment of the F-RGBD dataset and the common RGB dataset on Yolact++ prove the positive effect of the F-RGBD fusion method. Its segmentation masks have higher quality, are more continuous, and are less false positive for prioritizing-grasping prediction. Based on the segmentation result, a 4D grab line prediction method was proposed for cucumber grasping. And the cucumber detection experiment in cluttered scenarios is carried out in the real world. The success rate is 93.67% and the average sorting time is 9.87 s. The effectiveness of the cucumber segmentation and grasping pose acquisition method is verified by experiments. Keywords: Clutter scenarios, Cucumber grasp, Convolutional neural network, Instance segmentation DOI: 10.25165/j.ijabe.20231606.7542 Citation: Zhang F, Hou Z Y, Gao J, Zhang J X, Deng X. Detection method for the cucumber robotic grasping pose in clutter scenarios via instance segmentation. Int J Agric & Biol Eng, 2023; 16(6): 215–225.References
[1] Cheng W-C, Hsiao H-C, Lin Y-P, Liu Y-H. Robotic arm pick-and-place system for L-shaped water pipe joint with arbitrary angle. In: 2020 International Automatic Control Conference (CACS), Hsinchu: IEEE, 2020; pp.1–5.
[2] Triantafyllou P, Mnyusiwalla H, Sotiropoulos P, Roa M A, Russell D, Deacon G. A Benchmarking Framework for systematic evaluation of robotic pick-and-place systems in an industrial grocery setting. In: 2019 International Conference on Robotics and Automation (ICRA), Montreal: IEEE, 2019; pp.6692–6698.
[3] Ye H J, Liu C Q, Niu P Y. Cucumber appearance quality detection under complex background based on image processing. Int J Agric & Biol Eng, 2018; 11(4): 193–199.
[4] Feng Q C, Zou W, Fan P F, Zhang C F, Wang X, Design and test of robotic harvesting system for cherry tomato. Int J Agric & Biol Eng, 2018; 11(1): 96–100.
[5] Cornelius H, Kragic D, Eklundh J O. Object and pose recognition using contour and shape information. In: ICAR '05 Proceedings 12th International Conference on Advanced Robotics, 2005; pp.613–620.
[6] Rahardja K, Kosaka A. Vision-based bin-picking: recognition and localization of multiple complex objects using simple visual cues. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS '96, Osaka: IEEE, 1996; pp.1448–1457.
[7] Hinterstoisser S, Cagniart C, Ilic S, Sturm P, Navab N, Fua P, et al. Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012; 34(5): 876–888.
[8] Liu W H, Pan Z Y, Liu W J, Shao Q Q, Hu J, Wang W M, et al. Deep learning for picking point detection in dense cluster. In: 2017 11th Asian Control Conference (ASCC), Gold Coast: IEEE, 2017; pp.1644-1649.
[9] Redmon J, Angelova A. Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle: IEEE, pp.1316–1322.
[10] Kehl W, Manhardt F, Tombari F, Ilic S, Navab N. SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, 2017; pp.1530–1538.
[11] Mao S H, Li Y H, Ma Y, Zhang B H, Zhou J, Wang K. Automatic cucumber recognition algorithm for harvesting robots in the natural environment using deep learning and multi-feature fusion. Computers and Electronics in Agriculture, 2020; 170: 105254.
[12] Liu X Y, Zhao D A, Jia W K, Ji W, Ruan C Z, Sun Y P. Cucumber fruits detection in greenhouses based on instance segmentation. IEEE Access, 2019; 7: 139635–139642.
[13] Zhao W R, Wang J C, Chen W D, Huang Y. Planning for grasping cluttered objects based on obstruction degree. International Journal of Advanced Robotic Systems, 2021; 18(6): 17298814211040632.
[14] Tu S Q, Pang J, Liu H F, Zhuang N, Chen Y, Zheng C, et al. Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images. Precision Agriculture, 2020; 21: 1072–1091.
[15] Sa I, Ge Z Y, Dayoub F, Upcroft B, Perez T, McCool C. DeepFruits: A fruit detection system using deep neural networks. Sensors, 2016; 16(8): 1222.
[16] Russell B C, Torralba A, Murphy K P, Freeman W T. LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 2008; 77: 157–173.
[17] Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft COCO: Common objects in context. In: Computer Vision – ECCV 2014; 8693: 740–755.
[18] Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT: Real-time instance segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul: IEEE, 2019; pp.9156–9165.
[19] Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT++: Better real-time instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022; 44(2): 1108–1121.
[20] Lin T-Y, Dollar P, Girshick R, He K M, Hariharan B, Belongie S. Feature Pyramid Networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu: IEEE, 2017; pp.936-944.
[21] He K M, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, 2017; pp.2980-2988.
[22] Wu M H, Yue H H, Wang J, Huang Y X, Liu M, Jiang Y H, et al. Object detection based on RGC mask R-CNN. IET Image Processing, 2020; 14(8): 1502–1508.
[23] Cheng F-T, Hour T-L, Sun Y-Y, Chen T-H. Study and resolution of singularities for a 6-DOF PUMA manipulator. IEEE Transactions on System, Man, Cybernetics, 1997; 27(2): 332–343.
[24] Yu S S, Wang C, Wen C L, Cheng M, Liu M H, Zhang Z H, et al. LiDAR-based localization using universal encoding and memory-aware regression. Pattern Recognition, 2022; 128: 108685.
[2] Triantafyllou P, Mnyusiwalla H, Sotiropoulos P, Roa M A, Russell D, Deacon G. A Benchmarking Framework for systematic evaluation of robotic pick-and-place systems in an industrial grocery setting. In: 2019 International Conference on Robotics and Automation (ICRA), Montreal: IEEE, 2019; pp.6692–6698.
[3] Ye H J, Liu C Q, Niu P Y. Cucumber appearance quality detection under complex background based on image processing. Int J Agric & Biol Eng, 2018; 11(4): 193–199.
[4] Feng Q C, Zou W, Fan P F, Zhang C F, Wang X, Design and test of robotic harvesting system for cherry tomato. Int J Agric & Biol Eng, 2018; 11(1): 96–100.
[5] Cornelius H, Kragic D, Eklundh J O. Object and pose recognition using contour and shape information. In: ICAR '05 Proceedings 12th International Conference on Advanced Robotics, 2005; pp.613–620.
[6] Rahardja K, Kosaka A. Vision-based bin-picking: recognition and localization of multiple complex objects using simple visual cues. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS '96, Osaka: IEEE, 1996; pp.1448–1457.
[7] Hinterstoisser S, Cagniart C, Ilic S, Sturm P, Navab N, Fua P, et al. Gradient response maps for real-time detection of textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012; 34(5): 876–888.
[8] Liu W H, Pan Z Y, Liu W J, Shao Q Q, Hu J, Wang W M, et al. Deep learning for picking point detection in dense cluster. In: 2017 11th Asian Control Conference (ASCC), Gold Coast: IEEE, 2017; pp.1644-1649.
[9] Redmon J, Angelova A. Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle: IEEE, pp.1316–1322.
[10] Kehl W, Manhardt F, Tombari F, Ilic S, Navab N. SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, 2017; pp.1530–1538.
[11] Mao S H, Li Y H, Ma Y, Zhang B H, Zhou J, Wang K. Automatic cucumber recognition algorithm for harvesting robots in the natural environment using deep learning and multi-feature fusion. Computers and Electronics in Agriculture, 2020; 170: 105254.
[12] Liu X Y, Zhao D A, Jia W K, Ji W, Ruan C Z, Sun Y P. Cucumber fruits detection in greenhouses based on instance segmentation. IEEE Access, 2019; 7: 139635–139642.
[13] Zhao W R, Wang J C, Chen W D, Huang Y. Planning for grasping cluttered objects based on obstruction degree. International Journal of Advanced Robotic Systems, 2021; 18(6): 17298814211040632.
[14] Tu S Q, Pang J, Liu H F, Zhuang N, Chen Y, Zheng C, et al. Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images. Precision Agriculture, 2020; 21: 1072–1091.
[15] Sa I, Ge Z Y, Dayoub F, Upcroft B, Perez T, McCool C. DeepFruits: A fruit detection system using deep neural networks. Sensors, 2016; 16(8): 1222.
[16] Russell B C, Torralba A, Murphy K P, Freeman W T. LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 2008; 77: 157–173.
[17] Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft COCO: Common objects in context. In: Computer Vision – ECCV 2014; 8693: 740–755.
[18] Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT: Real-time instance segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul: IEEE, 2019; pp.9156–9165.
[19] Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT++: Better real-time instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022; 44(2): 1108–1121.
[20] Lin T-Y, Dollar P, Girshick R, He K M, Hariharan B, Belongie S. Feature Pyramid Networks for object detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu: IEEE, 2017; pp.936-944.
[21] He K M, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, 2017; pp.2980-2988.
[22] Wu M H, Yue H H, Wang J, Huang Y X, Liu M, Jiang Y H, et al. Object detection based on RGC mask R-CNN. IET Image Processing, 2020; 14(8): 1502–1508.
[23] Cheng F-T, Hour T-L, Sun Y-Y, Chen T-H. Study and resolution of singularities for a 6-DOF PUMA manipulator. IEEE Transactions on System, Man, Cybernetics, 1997; 27(2): 332–343.
[24] Yu S S, Wang C, Wen C L, Cheng M, Liu M H, Zhang Z H, et al. LiDAR-based localization using universal encoding and memory-aware regression. Pattern Recognition, 2022; 128: 108685.
Downloads
Published
2024-02-06
How to Cite
Zhang, F., Hou, Z., Gao, J., Zhang, J., & Deng, X. (2024). Detection method for the cucumber robotic grasping pose in clutter scenarios via instance segmentation. International Journal of Agricultural and Biological Engineering, 16(6), 215–225. Retrieved from https://ijabe.migration.pkpps03.publicknowledgeproject.org/index.php/ijabe/article/view/7542
Issue
Section
Information Technology, Sensors and Control Systems
License
IJABE is an international peer reviewed open access journal, adopting Creative Commons Copyright Notices as follows.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
