Fine-Tuned Segment Anything Model with Low-Rank Adaptation for Chest X-Ray Images
Abstract
Background: This paper investigates the use of the Segment Anything Model (SAM) for chest X-ray (CXR) image segmentation, with a focus on improving its performance using low-rank adaptation (LoRA). Methods: We evaluate three versions of SAM: two zero-shot methods (using coordinate and bounding box prompts) and a fine-tuned SAM using LoRA. To support these approaches, we also trained two standard convolutional neural networks (CNNs), U-Net and DeepLabv3+, to generate draft lung segmentations that serve as input prompts for the SAM methods. Our fine-tuning approach uses LoRA to add lightweight trainable adapters within the Transformer blocks of the SAM, allowing only a small subset of parameters to be updated. The rest of the SAM remains frozen, helping preserve its pre-trained knowledge while reducing memory and computational needs. We tested all models on a dataset of CXR images labeled for COVID-19, viral pneumonia, and normal cases. Results: Results show that fine-tuned SAM with LoRA outperforms zero-shot SAM methods and CNN baselines in terms of segmentation accuracy and efficiency. Conclusions: This demonstrates the potential of combining LoRA with SAM for practical and effective medical image segmentation.
Key Points
Objective
The research aims to enhance the Segment Anything Model's performance for chest X-ray image segmentation using low-rank adaptation.
Methods
- Evaluated three SAM versions: two zero-shot methods and one fine-tuned with low-rank adaptation.
- Trained U-Net and DeepLabv3+ CNNs for generating draft lung segmentations as SAM input prompts.
- Implemented lightweight adapters within SAM's Transformer blocks for efficient fine-tuning.
Results
- Fine-tuned SAM with low-rank adaptation significantly outperformed zero-shot SAM techniques.
- Achieved higher segmentation accuracy and efficiency compared to CNN baselines.