[Defense] Exploring the Intersection of Computer Vision and Smart Agriculture: Towards the automation of Mushroom Harvesting, Growth Monitoring, and 3D Pose Estimation
Monday, April 21, 2025
11:30 am - 1:00 pm
In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy
Abdollah Zakeri
will defend his doctoral proposal
Exploring the Intersection of Computer Vision and Smart Agriculture: Towards the automation of Mushroom Harvesting, Growth Monitoring, and 3D Pose Estimation
Abstract
This proposal report details the development of comprehensive methodologies for mushroom de- tection, segmentation, and 3D pose estimation through two newly created datasets and customized algorithms. The M18K real-world RGB-D dataset is introduced, featuring 2,000 annotated im- ages (over 100,000 mushroom instances) for detection and pose estimation, plus 3,838 time-lapse images for growth monitoring. Additionally, SMS3D, a synthetic 3D dataset of 40,000 photo- realistic scenes generated with high-definition mushroom models and Perlin-noise-based terrain, is presented. Multiple detection and segmentation architectures (Mask R-CNN, Faster R-CNN, YOLOv8, RT-DETR) were evaluated to identify optimal speed-accuracy trade-offs for mushroom- focused tasks, and F1 scores of 0.881 and 0.907 were achieved for detection and segmentation, respectively. Building on these, a two-stage pipeline couples 2D instance segmentation with a 3D rotation estimation module (Point Transformer and 6D rotation) to achieve an average orientation error of 1.67◦ on synthetic data and 3.68◦ on a newly annotated real subset of M18K. In parallel, a diffusion-driven depth inpainting approach was designed and completed to resolve large missing regions caused by reflective caps, reaching a masked L1 error of 0.011 and outperforming both classical and alternative deep-learning methods. Contributions include the creation of two novel datasets for mushroom perception, extensive benchmarking of detection and segmentation models, a robust 3D pose pipeline transferring effectively from synthetic to real data, and an advanced inpainting network that mitigates sensor noise. Experimental findings confirm the practicality of these solutions for automating mushroom farming, from real-time harvesting to long-term growth monitoring, and pave the way for further optimizations. Future work includes integrating depth data into the M18K yield-monitoring subset for accurate volumetric measurements, exploring time- series models like LSTMs or GRUs to predict growth and maturity, performing ablation studies on the inpainting framework, and assessing how improved depth quality enhances downstream tasks such as 3D pose estimation.
Monday, April 21, 2025
11:30 AM - 1:00 PM
PGH 550
Dr. Sen Lin, chair and Dr. Fatima Merchant, co-chair and research advisor
Faculty, students, and the general public are invited.