Enhanced Precision of Fluorescence In Situ Hybridization (FISH) Analysis Using Neural Network-Based Nuclear Segmentation for Digital Microscopy Samples

Introduction: Accurate nuclear segmentation is essential for the reliable diagnostic interpretation of fluorescence in situ hybridization (FISH) results. However, automated 2D digital algorithms often fail in samples with dense or overlapping nuclei, such as lymphomas, due to the loss of spatial depth information. Here, we tested if AI-based 3D nuclear segmentation can improve the accuracy, reproducibility, and diagnostic reliability of FISH reading in critical situations. Materials and Methods: Formalin-fixed follicular lymphoma sections were FISH-labeled for BCL2 gene rearrangements and digitally scanned in multilayer Z-stacks. The analytic performance in nuclear segmentation of the adaptive thresholding-based FISHQuant, and the freely accessible AI-based NucleAIzer, StarDist, and Cellpose algorithms, were compared to the eye control-based traditional FISH testing, primarily focusing on nuclear segmentation. Results: We revealed that the Cellpose algorithm showed limited sensitivity to low-intensity signals and the adaptive thresholding 2D segmentation, and FISHQuant struggled to resolve densely packed nuclei, occasionally underestimating their counts. In contrast, 3D segmentation across focal planes significantly improved the nuclear separation and signal localization. AI-driven 3D models, especially NucleAIzer and StarDist, showed improved precision, lower variance (VP/VS ≈ 0.96), and improved gene spot correlation (r > 0.82) across multiple focal planes. The similar average number of gene spots per cell nuclei in the AI-based analyses as the eye control counting, despite the elevated number of cell nuclei found with AI, validated the AI nuclear segmentation results. Conclusions: Inaccurate segmentation limits automated diagnostic FISH signal evaluation. Deep learning 3D approaches, particularly NucleAIzer and StarDist, may overcome thresholding and 2D constraints and improve the consistency of nuclear detection, resulting in better classification of pathogenetic gene aberrations with automated workflows in digital pathology.