On-field grading of strawberries in a greenhouse environment with deep learning-based occlusion handling

Abstract

Grading plays a vital role in strawberry production, especially to meet industry or market standards and maintain uniformity in the strawberries’ ripeness, appearance, and size within a batch. However, grading has been a post-harvest operation, which incorporates the risk of mechanical damage and fungal infection. Therefore, this study proposed a computer vision-based on-field grading of strawberries before harvesting, utilizing size, shape, and ripeness as key parameters estimated from images. RGB-Depth images were collected from a commercial strawberry farm, and a deep learning-based instance segmentation model, Mask R-CNN, was trained to detect and segment strawberries as ideal, mildly occluded, and heavily occluded. The strawberry size and shape were estimated using segmented masks and depth frames, while ripeness was determined from RGB images utilizing the proposed methods. Subsequently, final grading was performed following the multi-attribute decision criteria. Finally, three field trials were undertaken to assess the efficacy of the proposed system in size, shape and ripeness estimation, and overall grading. The Mask R-CNN demonstrated precision and recall values of 0.91 and 0.92, respectively, in detecting strawberry occlusion levels. Likewise, size estimation exhibited a root mean square error of 2.94 mm compared to ground truth measurements, while ripeness classification yielded high precision and recall of 0.967 and 0.960, respectively. Ultimately, the proposed system achieved an accuracy of 84.5% in final grading, showcasing its potential to aid in harvesting strawberries and mitigate risks of mechanical damage and fungal infection.