byungkwanlee / causal-unsupervised-segmentation Goto Github PK

The calculation of loss_linear needs to get the label. But the unsupervised training process doesn't have label. How can I train the model with no label data?

I runed the test_tr.py file and the program stuct when it finetunes the segmentation with CRF.

How can I facilitate this process?

Hi, thanks for the great work. I cannot reproduce the DINOv2+CAUSE-TR on cocostuff27. I get 35.1 mIoU instead of 45.3. Could you help? Thanks a lot!

Here is the log

test_tr.py --dataset cocostuff27 --ckpt checkpoint/dinov2_vit_base_14.pth --gpu 4,5,6,7
------------------Configurations------------------
NAME_TAG: CAUSE-TR
data_dir: data
dataset: cocostuff27
port: 12355
ckpt: checkpoint/dinov2_vit_base_14.pth
distributed: False
load_segment: True
load_cluster: True
train_resolution: 322
test_resolution: 322
batch_size: 16
num_workers: 22
gpu: 4,5,6,7
num_codebook: 2048
reduced_dim: 90
projection_dim: 2048
dim: 768
num_queries: 529
-------------------------------------------------
# of Parameters: 9.01(M)
[TEST] Acc (Linear): 80.0% | [mIoU]: 33.0, [mAP]: 55.1, [Acc]: 66.1, : 100%|████████████████| 136/136 [00:06<00:00, 22.23it/s]
[mIoU]: 35.1, [mAP]: 60.3, [Acc]: 69.4, : 100%|███████████████████████████| 136/136 [08:20<00:00,  3.68s/it]

and here is the script to reproduce the bug

#!/bin/bash
######################################
# [OPTION] DATASET

# cocostuff27
dataset="cocostuff27"

######################################
# [OPTION] STRUCTURE
structure="TR"
######################################

# DINOv2
ckpt="checkpoint/dinov2_vit_base_14.pth"

######################################

######################################
# GPU and PORT
train_gpu="4,5,6,7"

# Non-Changeable Variable
test_gpu="${train_gpu:0}"
port=$(($RANDOM%800+1200))
######################################

######################################
# [STEP1] MEDIATOR
python train_mediator.py --dataset $dataset --ckpt $ckpt --gpu $train_gpu --port $port
######################################

######################################
# [STEP2] CAUSE
python train_front_door_tr.py --dataset $dataset --ckpt $ckpt --gpu $train_gpu --port $port
python fine_tuning_tr.py --dataset $dataset --ckpt $ckpt --gpu $train_gpu --port $port
######################################

######################################
# TEST
python test_tr.py --dataset $dataset --ckpt $ckpt --gpu $test_gpu
######################################

Computation required for training and training time

Thank you for your outstanding work. I would like to ask whether the DINO pretrained model used is the "backbone only" of the page you gave or "full ckpt". If it is the "backbone only", Runtime message _IncompatibleKeys(missing_keys=['head.weight', 'head.bias'], unexpected_keys=[])

How can I use Causal for training on custom dataset? Is it possible to use it for video semantic segmentation??

When I try to run, I get this error. DINOMAEVisionTransformer model fails to init. Any suggestions? Thanks.

./run
------------------Configurations------------------
epoch: 1
distributed: False
load_segment: False
load_cluster: False
train_resolution: 320
test_resolution: 320
batch_size: 16
num_workers: 4
data_dir: /home/root/moredata/segment/fast/
dataset: cocostuff27
ckpt: checkpoint/dino_vit_base_8.pth
gpu: 0
port: 1243
grid: True
num_codebook: 2048
reduced_dim: 90
projection_dim: 2048
dim: 768

Traceback (most recent call last):
File "/home/root/code/Causal-Unsupervised-Segmentation/train_mediator.py", line 173, in
main(rank=gpu_list[0], args=args, ngpus_per_node=1)
File "/home/root/code/Causal-Unsupervised-Segmentation/train_mediator.py", line 67, in main
net = network_loader(args, rank)
File "/home/root/code/Causal-Unsupervised-Segmentation/loader/netloader.py", line 9, in network_loader
net = load_model(args.ckpt, rank).cuda()
File "/home/root/code/Causal-Unsupervised-Segmentation/loader/netloader.py", line 98, in load_model
net = getattr(model, arch)()
File "/home/root/code/Causal-Unsupervised-Segmentation/models/dinomaevit.py", line 103, in dino_vit_base_8
model = DINOMAEVisionTransformer(
File "/home/root/code/Causal-Unsupervised-Segmentation/models/dinomaevit.py", line 41, in init
super(DINOMAEVisionTransformer, self).init(embed_layer=PatchEmbed_DimensionFree, **kwargs)
File "/home/root/anaconda3/envs/neurips/lib/python3.9/site-packages/timm/models/vision_transformer.py", line 471, in init
self.patch_embed = embed_layer(
TypeError: init() got an unexpected keyword argument 'dynamic_img_pad'

According to the paper, the step 2 should perform self-supervised learning with contrastive learning objective. However, in the script train_front_door_tr.py and fine_tuning_tr.py , the dataset ground-truth labels are used to train the linear probe. And that linear probe is also used to generate predictions during the inference .

I understand that these scripts are used to generate Table 2 result, which uses a linear probe to compare against other baselines.

Does there any script perform fully unsupervised learning training & evaluation in this repo ? That would be really helpful if such scripts exist for us to evaluate this method's performance in a fully unsupervised learning manner without linear probing. :)

Is the modularity loss used in the paper equivalent to the modularity loss proposed in ACSeg? I didn't work fully in the math but it seems that these two losses are really similar.