The main task for this finetuning is to finetune the SAM model to segment your custom data based on promt bounding boxes. Further, this tool can autogenerate segmentation masks on unlabeled data via the following pipeline.

git clone https://github.com/everguard-inc/segment_anything_tuning
cd segment_anything_tuning
conda create -n "fineSAM" python=3.9 -y
caonda activate fineSAM
pip install -r requirements.txt

1. Prepare your custom dataset. The recommended structure should be the following

   ├── train
   │   ├── img
   │   │   ├── train_img1.jpeg
   │   │   ...
   │   └── masks
   │   │   ├── train_mask1.png
   │   │   ...
   ├── val
   │   ├── img
   │   │   ├── val_img1.jpeg
   │   │   ...
   │   └── masks
   │   │   ├── val_mask1.png
   │   │   ...
   

   Or any other, you anyway need to set a path to every of four dirs in finetune_sam/config.py.

2. Edit finetune_sam/config.py with your dataset paths. Also, check other params:cuda-devices-ID, out-path, etc.

3. Run finetune_sam/train.py

4. To inference eg_dataset (in folder segment_anything_tuning; fineSAM - activated):

pwd # returns ..../segment_anything_tuning
export PYTHONPATH="${PYTHONPATH}:$(pwd)/finetune_sam"
python inference.py

• Supports custom datasets
• Performed caching of embeddings:
  • If embedding is not cached: 1-2 sec/img
  • If embedding cached: 0.2-0.4 sec/img
• Image preprocess encapsulated into the model. As input into the model, you must pass raw RGB image and promt bounding boxes for each image, both in patched form.
• Setuped Neptune for tracking training progress.

• train_dice_loss
• train_focal_loss
• train_iou_loss
• val_f1
• val_iou

Trained on dataset of tracks (~3500 imgs - train set, ~700 imgs - test set). After 20-th epoch metrics are mainly stable. Intresting is tha it dows not overfit after huge amoun of pochs (100).

• For generating bounding boxes from text and prompt them to SAM, you may check: lang-segment-anything
• Add noise to promt boxes for better adapting for low-quality promts.
• Add training on different classes simultaneously.

• Segment Anything
• Lightning AI
• Lightning Segment-Anything Model

This project is licensed the same as the SAM model.

• Uses the original implementation of SAM.
• Loss calculated as stated on the paper (20 * focal loss + dice loss + mse loss).
• Only supports bounding box input prompts.