Branch detection with apple trees trained in fruiting wall architecture using stereo vision and Regions-Convolutional Neural Network(R-CNN)

Due to the rising cost and decreasing availability of labor, manual picking is becoming increasingly challenging for apple and other tree fruit growers. A targeted shake-and-catch apple harvesting machine is under development at Washington State University to address this challenge. This machine is showing a promise for harvesting some varieties of apples. However, the performance and productivity of such a harvesting system could greatly be increased if the shaking process could be automated. First step towards automated shaking is the detection of branches and localization of shaking points according to the position of branches in real world. A b r a n c h detection method was developed in this work for apple trees trained in fruiting wall architecture using a stereo vision system and a Regions-Convolutional Neural Network (R-CNN). A stereo vision camera was used to acquire RGB images, depth images as well as index images in natural orchard environment. The R-CNN composed of improved AlexNet, which was trained to detect the apple tree branches. In this study, a fusion detection method called Depth & Index (D&I) was proposed to fuse the detection results of branches from both depth images and index images. The results showed that the value of average recall and Average Accuracy (AA) from the D&I method was 70.5% and 63.3% when the R-CNN confidence of depth image was 50.0%. However, in the same conditions, the average recall and AA was only 62.7% and 59.2% using the depth images alone. Furthermore, the D&I method also had better performance in terms of the morphology fitting of apple tree branches. This study showed a great potential using both of depth and index images to detect and fit apple tree branches in real-time.