Safe Leaf Manipulation for Accurate Shape and Pose Estimation of Occluded Fruits

Shaoxiong Yao∗,1, Sicong Pan∗,2, Maren Bennewitz2, Kris Hauser1
1University of Illinois at Urbana-Champaign, IL, USA. 2Humanoid Robots Lab, University of Bonn, Germany. ∗These authors contributed equally to this work.
arXiv Video (Coming soom) Code (Coming soom)
Fruit shape and pose estimation with leaf manipulation
We propose an active fruit shape and pose estimation method that physically manipulates occluding leaves to reveal hidden fruits.

Abstract

Fruit monitoring plays an important role in crop management, and rising global fruit consumption combined with labor shortages necessitates automated monitoring with robots. However, occlusions from plant foliage often hinder accurate shape and pose estimation. Therefore, we propose an active fruit shape and pose estimation method that physically manipulates occluding leaves to reveal hidden fruits. This paper introduces a framework that plans robot actions to maximize visibility and minimize leaf damage. We developed a novel scene-consistent shape completion technique to improve fruit estimation under heavy occlusion and utilize a perception-driven deformation graph model to predict leaf deformation during planning. Experiments on artificial and real sweet pepper plants demonstrate that our method enables robots to safely move leaves aside, exposing fruits for accurate shape and pose estimation, outperforming baseline methods.

Method

Method overview
Overview of the proposed active fruit shape and pose estimation system for plant-safe occlusion removal. Our system plans robot action that maximizes fruit visibility and minimizes plant damage risk.

Shape Completion Visualization

Proposed

Without Scene-Consistency

Leaf Deformation Visualization

Low Energy Action

High Energy Action

Results

We tested 12 artificial plant scenarios, including two types of artificial leaves (small/large with thin/thick branches), three artificial peppers with varying shapes (circular and striped) and colors (red and yellow), and two different occlusion conditions. We show our results in a list of 2x2 small figures. Left top: initial RGB image; right top: initial shape completion; left bottom: final RGB image; right bottom: final shape completion.

Unsafe Actions

Here we illustrates three types of unsafe actions.

Hazard

Fruit Collision

Leaf Damage

Multi-fruit scenario

Our system can scale to multi-fruit scenario and achieves maximum visibility for all fruits while preserving integrity of all leaves. Video in 6x speed.