Deep learning detection and segmentation of mushrooms in visually complex environments

This Ph.D. thesis traces a systematic investigation into DL instance segmentation for oyster mushroom (Pleurotus spp.) farming automation, culminating in a generalizable crop growth monitoring pipeline that can assist farmers in data-driven decision support and provide a baseline for broader applications in smart mushroom farming. Accurate growth monitoring is hindered by complex mushroom cluster morphology, characterized by non-symmetrical, overlapping fruiting bodies with varying orientations. To address these challenges, this thesis explores the integration of deep learning (DL)-based instance segmentation into oyster mushroom farming. A comprehensive framework employing timelapse RGB imagery and advanced machine vision techniques was developed to enable continuous, non-invasive monitoring of mushroom clusters. Initially, a systematic literature review of 77 instance segmentation works in agriculture identified Mask R-CNN as the prevailing architecture in various applications e.g., crop detection and growth monitoring. Mask R-CNN performance was compared to multiple architectures combined with various state-of-the-art CNN- and Transformer-based backbone networks, using mushroom datasets captured in natural habitats. This analysis revealed key challenges such as background complexity, occlusion, and lighting variability. The original Mask R-CNN architecture offered a practical balance of computational efficiency and performance, especially when coupled with advanced CNN-based feature extraction networks. Furthermore, a novel annotated dataset of single oyster mushroom instances was captured in real farm environments. Mask R-CNN with CNN- and Transformer-based backbones was evaluated on how training set size affects performance, by conducting multiple rounds of model training-evaluation on progressively reduced training data. A new sampling methodology and two new instance segmentation metrics, namely Correctness and Instance Segmentation Quality Index (ISQI) were proposed to supplement the already established evaluation methods. ConvNeXt backbone network emerged as the most resilient option, delivering strong performance even under significant data reduction. Building upon these findings, an end-to-end DL-based pipeline was developed for automated mushroom growth monitoring using dense timelapse image data. Results demonstrated that ConvNeXt backbone achieved superior performance in detecting and segmenting mushroom clusters. A set of post-processing visual filters managed issues of occlusion, varying illumination, and complex background dynamics, discarding erroneous predictions and ensuring high quality instance segmentation outputs. Moreover, a custom lightweight tracking algorithm utilizing segmented masks effectively tracked individual mushroom instances growth trajectories. Additionally, the study proposed a novel methodology for converting pixel dimensions of segmented mushroom clusters into real-world measurements, enabling size monitoring without manual intervention. An evaluation of the growth monitoring pipeline was executed on an independent mushroom dataset featuring different visual conditions. The pipeline demonstrated its robust performance without reconfiguration, validating its transferability. In conclusion, the study confirms the use of instance segmentation not just in visual monitoring but as an enabler of downstream automation, including size approximation, maturity classification, and growth monitoring.