Advancing Orchard Fruit Detection: An Innovative Agricultural Foundation Model Approach

Fruit harvesting poses a significant labor and financial burden on the fruit industry, which underscore the urgent need for advancements in robotic harvesting solutions. Despite considerable progress in leveraging deep learning and machine learning techniques for fruit detection, a common shortfall is the inability to swiftly extend the developed models across different orchards and/or various fruit species. Additionally, the limited availability of pertinent data further compounds these challenges. In this work, we introduce MetaFruit, the largest publicly available multi-class fruit dataset, comprising 4,248 images and 248,015 manually labeled instances across diverse U.S. orchards. Furthermore, this study proposes an innovative open-set fruit detection system leveraging advanced Vision Foundation Models (VFMs) for fruit detection that can adeptly identify a wide array of fruit types under varying orchard conditions. This system not only demonstrates remarkable adaptability in learning from minimal data through few-shot learning but also shows the ability to interpret human instructions for subtle detection tasks. The performance of the developed foundation model is comprehensively evaluated across several metrics, outperforming existing state-of-the-art algorithms in our MetaFruit, thereby setting a new benchmark in the field of agricultural technology and robotic harvesting. The MetaFruit dataset (https://www.kaggle.com/datasets/jiajiali/metafruit) and detection framework (https://github.com/JiajiaLi04/FMFruit) are open-sourced to foster future research in vision-based fruit harvesting, marking a significant stride toward addressing the urgent needs of the agricultural sector.