Topic-guided cell segmentation for image-based spatial transcriptomics

Spatial transcriptomics enables the in situ mapping of gene expression, but the validity of downstream cellular analyses relies heavily on accurate cell segmentation. Current image-based segmentation tools rely on cellular staining rather than true molecular assignment; conversely, transcript-driven tools incur prohibitive computational costs and are prone to inferring morphologically aberrant cellular boundaries. To address these limitations, TOPIC (TOpic-guided Probabilistic Image-based Cell-segmentation), a novel transcript-driven framework utilizing Latent Dirichlet Allocation (LDA), was developed. By projecting spatial RNA coordinates onto a hexagonal grid and modeling these spatial bins as documents comprising distinct gene topics, TOPIC performs a robust region-growing expansion governed by biological consistency and localized transcript density; additionally, subsequent probabilistic purification allows the model to filter out statistically unlikely transcripts to eliminate spatial cross-contamination and background noise. TOPIC was validated by comparing it against state-of-the-art tools (Xenium Multimodal, Baysor, and Proseg) across diverse tissues and disease contexts, (e.g., human breast cancer, human lung cancer, mouse brain, and mouse lung). Performance was evaluated in terms of computational scalability (global segmentation yield, overall processing speed and memory requirements), transcriptomic fidelity (cell type recovery against orthogonal scRNA-seq references, cosine similarity to reference profiles, signal sensitivity versus contamination, and spatial characterization of extracellular RNA), and morphological accuracy (2D tissue area coverage, boundary concordance with image-based ground truths, and preservation of cell-specific morphological signatures and size dynamics). TOPIC successfully overcomes image-derived geometric biases by delineating highly irregular cytoplasmic extensions (such as sprawling Alveolar Type 1 cells) that standard image-based pipelines systematically underestimate. Furthermore, TOPIC balances morphological fidelity with high biological sensitivity and signal purity; in doing so, TOPIC avoids the conservative over-segmentation frequently observed in Baysor and the unbounded acellular expansion seen in Proseg.