Polyp segmentation in colonoscopy images presents challenges in complex clinical scenarios, particularly for small polyps, blurred boundaries, and low-contrast regions. Traditional vision-only methods often exhibit limited performance in these cases due to insufficient semantic understanding and difficulty in capturing fine-grained contextual information. To mitigate these limitations, we propose SCLiP-Polyp, a language-guided multimodal segmentation framework that incorporates semantic priors derived from clinical expertise without requiring paired image-text supervision during training. The framework utilizes sentence-level textual prompts that describe challenging clinical scenarios, such as “a flat polyp flush with the colon surface” for morphological complexity, or “pools of liquid visible in the colon” for background interference cases. These prompts are encoded using a frozen BiomedCLIP text encoder to enable flexible integration of textual cues. The method strategically categorizes prompts into foreground prompts (e.g. small polyps, flat morphology, blurred boundaries) and background prompts (e.g. illumination, fluid, folds), providing nuanced guidance for the segmentation model. SCLiP-Polyp employs a PVTv2-b2 visual encoder and introduces three key architectural components: (1) Cross-modal Relevance Estimation that computes correlations between deep visual features and prompt embeddings; (2) Attention-guided Feature Injection (AFI) blocks that inject prompt-conditioned guidance across multiple encoder scales through foreground-background separation and channel attention mechanisms; and (3) a Hierarchical Decoder that enhances multi-scale feature representations with spatial detail preservation. Comprehensive experimental evaluation on five benchmark datasets indicates competitive performance. Notably, on a curated subset of 278 challenging images, SCLiP-Polyp achieves 0.874 DSC and 0.800 IoU, suggesting a possible advantage in handling difficult clinical scenarios compared to existing vision-only approaches.
Li et al. (Sun,) studied this question.