Method for the real-time detection of tomato ripeness using a phenotype robot and RP-YolactEdge
Keywords:
instance segmentation, phenotype robot, tomato, greenhouse-based plant phenotyping, ripeness detectionAbstract
In order to address the challenge of non-destructive detection of tomato fruit ripeness in controlled environments,this study proposed a real-time instance segmentation method based on the edge device. This method combined the principlesof phenotype robots and machine vision based on deep learning. A compact and remotely controllable phenotype detectionrobot was employed to acquire precise data on tomato ripeness. The video data were then processed by using an efficientbackbone and the FeatFlowNet structure for feature extraction and analysis of key-frame to non-key-frame mapping from videodata. To enhance the diversity of training datasets and the generalization of the model, an innovative approach was chosen byusing random enhancement techniques. Besides, the PolyLoss optimization technique was applied to further improve theaccuracy of the ripeness multi-class detection tasks. Through validation, the method of this study achieved real-time processingspeeds of 90.1 fps (RTX 3070Ti) and 65.5 fps (RTX 2060 S), with an average detection accuracy of 97% compared tomanually measured results. This is more accurate and efficient than other instance segmentation models according to actualtesting in a greenhouse. Therefore, the results of this research can be deployed in edge devices and provide technical support forunmanned greenhouse monitoring devices or fruit-picking robots in facility environments. Keywords: instance segmentation, phenotype robot, tomato, greenhouse-based plant phenotyping, ripeness detection DOI: 10.25165/j.ijabe.20241702.8403. Citation: Wang Y Q, Gou W B, Wang C Y, Fan J C, Wen W L, Lu X J, et al. Method for the real-time detection of tomato ripeness using a phenotype robot and RP-YolactEdge. Int J Agric & Biol Eng, 2024; 17(2): 200–210.References
[1] Begum N, Hazarika M K. Maturity detection of tomatoes using transfer learning. Measurement: Food, 2022; 7: 100038.
[2] Chen G, Muriki H, Pradalier C, Chen Y, Dellaert F. A hybrid cable-driven robot for non-destructive leafy plant monitoring and mass estimation using structure from motion. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023; pp.11809–11816.
[3] Chen S M, Xiong J T, Jiao J M, Xie Z M, Huo Z W, Hu W X. Citrus fruits maturity detection in natural environments based on convolutional neural networks and visual saliency map. Precision Agriculture, 2022; 23: 1515–1531.
[4] Huang Y P, Si W, Chen K J, Sun Y. Assessment of tomato maturity in different layers by spatially resolved spectroscopy. Sensors, 2020; 20: 7229.
[5] Zheng T X, Jiang M Z, Li Y F, Feng M C. Research on tomato detection in natural environment based on RC-YOLOv4. Computers and Electronics in Agriculture, 2022; 198: 107029.
[6] Atefi A, Ge Y, Pitla S, Schnable J. Robotic technologies for high-throughput plant phenotyping: Contemporary reviews and future perspectives. Front Plant Sci, 2021; 12: 611940.
[7] Dong M, Yu H Y, Zhang L, Wu M Z, Sun Z P, Zeng D Q, et al. Measurement method of plant phenotypic parameters based on image deep learning. Wireless Communications and Mobile Computing, 2022; 2022: 7664045.
[8] Kolhar S, Jagtap J. Convolutional neural network based encoder-decoder architectures for semantic segmentation of plants. Ecological Informatics, 2021; 64: 101373.
[9] Rahul M S P, Rajesh M. Image processing based automatic plant disease detection and stem cutting robot. In: 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli: IEEE, 2020; pp.889–894.
[10] Numsong A, Posom J, Chuan-Udom S. Artificial neural network-based repair and maintenance cost estimation model for rice combine harvesters. Int J Agric & Biol Eng, 2023; 16(2): 38–47.
[11] Feng X B, He P J, Zhang H X, Yin W Q, Qian Y, Cao P, et al. Rice seeds identification based on back propagation neural network model. Int J Agric & Biol Eng, 2019; 12(6): 122–128.
[12] Lu W, Zeng M J, Qin H H. Intelligent navigation algorithm of plant phenotype detection robot based on dynamic credibility evaluation. Int J Agric & Biol Eng, 2021; 14(6): 195–206.
[13] Xiao D Y, Liang G, Liu C L, Huang Y X. Phenotype-based robotic screening platform for leafy plant breeding. IFAC-PapersOnLine, 2016; 49(16): 237–241.
[14] Li Y T, He L Y, Jia J M, Chen J N, Lyu J, Wu C Y. High-efficiency tea shoot detection method via a compressed deep learning model. Int J Agric & Biol Eng, 2022; 15(3): 159–166.
[15] Lu Y, Chen X Y, Wu Z X, Yu J Z, Wen L. A novel robotic visual perception framework for underwater operation. Frontiers of Information Technology & Electronic Engineering, 2022; 23(11): 1602–1619.
[16] Wang Y Q, Fan J C, Yu S, Cai S Z, Guo X Y, Zhao C J. Research advance in phenotype detection robots for agriculture and forestry. Int J Agric & Biol Eng, 2023; 16(1): 14–25.
[17] Yu S, Fan J C, Lu X J, Wen W L, Shao S, Guo X Y, et al. Hyperspectral technique combined with deep learning algorithm for prediction of phenotyping traits in lettuce. Frontiers in Plant Science, 2022; 13: 927832.
[18] Hu H M, Kaizu Y, Zhang H D, Xu Y W, Imou K, Li M, et al. Recognition and localization of strawberries from 3D binocular cameras for a strawberry picking robot using coupled YOLO/Mask R-CNN. Int J Agric & Biol Eng, 2022; 15(6): 175–179.
[19] Widiyanto S, Nugroho D P, Daryanto A, Yunus M, Tri D. Monitoring the growth of tomatoes in real time with deep learning-based image segmentation. International Journal of Advanced Computer Science and Applications, 2021; 12(12): 0121247.
[20] Zhang Y, Tian Z H, Ma W Q, Zhang M, Yang L L. Hyperspectral detection of walnut protein contents based on improved whale optimized algorithm. Int J Agric & Biol Eng, 2022; 15(6): 235–241.
[21] Liu W, Zou S S, Xu X L, Gu Q Y, He W Z, Huang J, et al. Development of UAV-based shot seeding device for rice planting. Int J Agric & Biol Eng, 2022; 15(6): 1–7.
[22] Weyler J, Milioto A, Falck T, Behley J, Stachniss C. Joint plant instance detection and leaf count estimation for in-field plant phenotyping. IEEE Robot and Automation Letters, 2021; 6(2): 3599–3606.
[23] Hosoi F, Nakabayashi K, Omasa K. 3-D modeling of tomato canopies using a high-resolution portable scanning Lidar for extracting structural information. Sensors, 2011; 11(2): 2166–2174.
[24] Yang L L, Tian W Z, Zhai W X, Wang X X, Chen Z B, Wen L, et al. Behavior recognition and fuel consumption prediction of tractor sowing operations using smartphone. Int J Agric & Biol Eng, 2022; 15(4): 154–162.
[25] Li H, Issaka Z, Jiang Y, Tang P, Chen C. Overview of emerging technologies in sprinkler irrigation to optimize crop production. Int J Agric & Biol Eng, 2019; 123): 1–9.
[26] Zu L L, Zhao Y P, Liu J Q, Su F, Zhang Y, Liu P Z. Detection and segmentation of mature green tomatoes based on mask R-CNN with automatic image acquisition approach. Sensors, 2021; 21(23): 7842.
[27] Fan J C, Zhang Y, Wen W L, Gu S H, Lu X J, Guo X Y. The future of Internet of Things in agriculture: Plant high-throughput phenotypic platform. Journal of Cleaner Production, 2021; 280: 123651.
[28] Jin Y, Liu J, Xu Z, Yuan S, Li P, Wang J, et al. Development status and trend of agricultural robot technology. Int J Agric & Biol Eng, 2021; 14(4): 1–19.
[29] Xiang L R, Nolan T M, Bao Y, Elmore M, Tuel T, Gai J Y, et al. Robotic assay for drought (RoAD): An automated phenotyping system for brassinosteroid and drought responses. Plant Journal, 2021; 107: 1837–1853.
[30] Yang C Z. Plant leaf recognition by integrating shape and texture features. Pattern Recognition, 2021; 112: 107809.
[31] He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. 2017 IEEE international Conference on Computer Vision (ICCV), Venice: IEEE, 2017; pp.2980–2988.
[32] Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT: Real-time instance segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul: IEEE, 2019; pp.9156–9165.
[33] Liu H, Rivera Soto R A, Xiao F, Jae Lee Y. YolactEdge: Real-time instance segmentation on the edge. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an: IEEE, 2021; pp.9579–9585.
[34] Young S N, Kayacan E, Peschel J M. Design and field evaluation of a ground robot for high-throughput phenotyping of energy sorghum. Precision Agriculture, 2019; 20: 697–722.
[35] Miao Z H, Yu X Y, Li N, Zhang Z, He C X, Li Z, et al. Efficient tomato harvesting robot based on image processing and deep learning. Precision Agriculture, 2023; 24: 254–287.
[36] Peng H X, Xue C, Shao Y Y, Chen K Y, Liu H N, Xiong J T, et al. Litchi detection in the field using an improved YOLOv3 model. Int J Agric & Biol Eng, 2022; 15(2): 211–220.
[37] Omasa K, Ono E, Ishigami Y, Shimizu Y, Araki Y. Plant functional remote sensing and smart farming applications. Int J Agric & Biol Eng, 2022; 15: 1–6.
[38] Li H H, Wei Y Y, Zhang H M, Chen H, Meng J F. Fine-grained classification of grape leaves via a pyramid residual convolution neural network. Int J Agric & Biol Eng, 2022; 15(2): 197–203.
[39] Yin X, Li W H, Li Z, Yi L L. Recognition of grape leaf diseases using MobileNetV3 and deep transfer learning. Int J Agric & Biol Eng, 2022; 15(3): 184–194.
[40] Yang Z K, Li W Y, Li M, Yang X T. Automatic greenhouse pest recognition based on multiple color space features. Int J Agric & Biol Eng, 2021; 14(2): 188–195.
[41] Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City: IEEE, 2018; pp.4510–4520.
[42] Leng Z Q, Tan M X, Liu C X, Cubuk E D, Shi X J, Cheng S Y, et al. Polyloss: A polynomial expansion perspective of classification loss functions. arXiv Preprint, 2022; arXiv: 2204.12511.
[2] Chen G, Muriki H, Pradalier C, Chen Y, Dellaert F. A hybrid cable-driven robot for non-destructive leafy plant monitoring and mass estimation using structure from motion. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023; pp.11809–11816.
[3] Chen S M, Xiong J T, Jiao J M, Xie Z M, Huo Z W, Hu W X. Citrus fruits maturity detection in natural environments based on convolutional neural networks and visual saliency map. Precision Agriculture, 2022; 23: 1515–1531.
[4] Huang Y P, Si W, Chen K J, Sun Y. Assessment of tomato maturity in different layers by spatially resolved spectroscopy. Sensors, 2020; 20: 7229.
[5] Zheng T X, Jiang M Z, Li Y F, Feng M C. Research on tomato detection in natural environment based on RC-YOLOv4. Computers and Electronics in Agriculture, 2022; 198: 107029.
[6] Atefi A, Ge Y, Pitla S, Schnable J. Robotic technologies for high-throughput plant phenotyping: Contemporary reviews and future perspectives. Front Plant Sci, 2021; 12: 611940.
[7] Dong M, Yu H Y, Zhang L, Wu M Z, Sun Z P, Zeng D Q, et al. Measurement method of plant phenotypic parameters based on image deep learning. Wireless Communications and Mobile Computing, 2022; 2022: 7664045.
[8] Kolhar S, Jagtap J. Convolutional neural network based encoder-decoder architectures for semantic segmentation of plants. Ecological Informatics, 2021; 64: 101373.
[9] Rahul M S P, Rajesh M. Image processing based automatic plant disease detection and stem cutting robot. In: 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli: IEEE, 2020; pp.889–894.
[10] Numsong A, Posom J, Chuan-Udom S. Artificial neural network-based repair and maintenance cost estimation model for rice combine harvesters. Int J Agric & Biol Eng, 2023; 16(2): 38–47.
[11] Feng X B, He P J, Zhang H X, Yin W Q, Qian Y, Cao P, et al. Rice seeds identification based on back propagation neural network model. Int J Agric & Biol Eng, 2019; 12(6): 122–128.
[12] Lu W, Zeng M J, Qin H H. Intelligent navigation algorithm of plant phenotype detection robot based on dynamic credibility evaluation. Int J Agric & Biol Eng, 2021; 14(6): 195–206.
[13] Xiao D Y, Liang G, Liu C L, Huang Y X. Phenotype-based robotic screening platform for leafy plant breeding. IFAC-PapersOnLine, 2016; 49(16): 237–241.
[14] Li Y T, He L Y, Jia J M, Chen J N, Lyu J, Wu C Y. High-efficiency tea shoot detection method via a compressed deep learning model. Int J Agric & Biol Eng, 2022; 15(3): 159–166.
[15] Lu Y, Chen X Y, Wu Z X, Yu J Z, Wen L. A novel robotic visual perception framework for underwater operation. Frontiers of Information Technology & Electronic Engineering, 2022; 23(11): 1602–1619.
[16] Wang Y Q, Fan J C, Yu S, Cai S Z, Guo X Y, Zhao C J. Research advance in phenotype detection robots for agriculture and forestry. Int J Agric & Biol Eng, 2023; 16(1): 14–25.
[17] Yu S, Fan J C, Lu X J, Wen W L, Shao S, Guo X Y, et al. Hyperspectral technique combined with deep learning algorithm for prediction of phenotyping traits in lettuce. Frontiers in Plant Science, 2022; 13: 927832.
[18] Hu H M, Kaizu Y, Zhang H D, Xu Y W, Imou K, Li M, et al. Recognition and localization of strawberries from 3D binocular cameras for a strawberry picking robot using coupled YOLO/Mask R-CNN. Int J Agric & Biol Eng, 2022; 15(6): 175–179.
[19] Widiyanto S, Nugroho D P, Daryanto A, Yunus M, Tri D. Monitoring the growth of tomatoes in real time with deep learning-based image segmentation. International Journal of Advanced Computer Science and Applications, 2021; 12(12): 0121247.
[20] Zhang Y, Tian Z H, Ma W Q, Zhang M, Yang L L. Hyperspectral detection of walnut protein contents based on improved whale optimized algorithm. Int J Agric & Biol Eng, 2022; 15(6): 235–241.
[21] Liu W, Zou S S, Xu X L, Gu Q Y, He W Z, Huang J, et al. Development of UAV-based shot seeding device for rice planting. Int J Agric & Biol Eng, 2022; 15(6): 1–7.
[22] Weyler J, Milioto A, Falck T, Behley J, Stachniss C. Joint plant instance detection and leaf count estimation for in-field plant phenotyping. IEEE Robot and Automation Letters, 2021; 6(2): 3599–3606.
[23] Hosoi F, Nakabayashi K, Omasa K. 3-D modeling of tomato canopies using a high-resolution portable scanning Lidar for extracting structural information. Sensors, 2011; 11(2): 2166–2174.
[24] Yang L L, Tian W Z, Zhai W X, Wang X X, Chen Z B, Wen L, et al. Behavior recognition and fuel consumption prediction of tractor sowing operations using smartphone. Int J Agric & Biol Eng, 2022; 15(4): 154–162.
[25] Li H, Issaka Z, Jiang Y, Tang P, Chen C. Overview of emerging technologies in sprinkler irrigation to optimize crop production. Int J Agric & Biol Eng, 2019; 123): 1–9.
[26] Zu L L, Zhao Y P, Liu J Q, Su F, Zhang Y, Liu P Z. Detection and segmentation of mature green tomatoes based on mask R-CNN with automatic image acquisition approach. Sensors, 2021; 21(23): 7842.
[27] Fan J C, Zhang Y, Wen W L, Gu S H, Lu X J, Guo X Y. The future of Internet of Things in agriculture: Plant high-throughput phenotypic platform. Journal of Cleaner Production, 2021; 280: 123651.
[28] Jin Y, Liu J, Xu Z, Yuan S, Li P, Wang J, et al. Development status and trend of agricultural robot technology. Int J Agric & Biol Eng, 2021; 14(4): 1–19.
[29] Xiang L R, Nolan T M, Bao Y, Elmore M, Tuel T, Gai J Y, et al. Robotic assay for drought (RoAD): An automated phenotyping system for brassinosteroid and drought responses. Plant Journal, 2021; 107: 1837–1853.
[30] Yang C Z. Plant leaf recognition by integrating shape and texture features. Pattern Recognition, 2021; 112: 107809.
[31] He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. 2017 IEEE international Conference on Computer Vision (ICCV), Venice: IEEE, 2017; pp.2980–2988.
[32] Bolya D, Zhou C, Xiao F Y, Lee Y J. YOLACT: Real-time instance segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul: IEEE, 2019; pp.9156–9165.
[33] Liu H, Rivera Soto R A, Xiao F, Jae Lee Y. YolactEdge: Real-time instance segmentation on the edge. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an: IEEE, 2021; pp.9579–9585.
[34] Young S N, Kayacan E, Peschel J M. Design and field evaluation of a ground robot for high-throughput phenotyping of energy sorghum. Precision Agriculture, 2019; 20: 697–722.
[35] Miao Z H, Yu X Y, Li N, Zhang Z, He C X, Li Z, et al. Efficient tomato harvesting robot based on image processing and deep learning. Precision Agriculture, 2023; 24: 254–287.
[36] Peng H X, Xue C, Shao Y Y, Chen K Y, Liu H N, Xiong J T, et al. Litchi detection in the field using an improved YOLOv3 model. Int J Agric & Biol Eng, 2022; 15(2): 211–220.
[37] Omasa K, Ono E, Ishigami Y, Shimizu Y, Araki Y. Plant functional remote sensing and smart farming applications. Int J Agric & Biol Eng, 2022; 15: 1–6.
[38] Li H H, Wei Y Y, Zhang H M, Chen H, Meng J F. Fine-grained classification of grape leaves via a pyramid residual convolution neural network. Int J Agric & Biol Eng, 2022; 15(2): 197–203.
[39] Yin X, Li W H, Li Z, Yi L L. Recognition of grape leaf diseases using MobileNetV3 and deep transfer learning. Int J Agric & Biol Eng, 2022; 15(3): 184–194.
[40] Yang Z K, Li W Y, Li M, Yang X T. Automatic greenhouse pest recognition based on multiple color space features. Int J Agric & Biol Eng, 2021; 14(2): 188–195.
[41] Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City: IEEE, 2018; pp.4510–4520.
[42] Leng Z Q, Tan M X, Liu C X, Cubuk E D, Shi X J, Cheng S Y, et al. Polyloss: A polynomial expansion perspective of classification loss functions. arXiv Preprint, 2022; arXiv: 2204.12511.
Downloads
Published
2024-05-21
How to Cite
Wang, Y., Gou, W., Wang, C., Fan, J., Wen, W., Lu, X., … Zhao, C. (2024). Method for the real-time detection of tomato ripeness using a phenotype robot and RP-YolactEdge. International Journal of Agricultural and Biological Engineering, 17(2), 200–210. Retrieved from https://ijabe.migration.pkpps03.publicknowledgeproject.org/index.php/ijabe/article/view/8403
Issue
Section
Information Technology, Sensors and Control Systems
License
IJABE is an international peer reviewed open access journal, adopting Creative Commons Copyright Notices as follows.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
