haochen-wang409 / u2pl Goto Github PK

Hi, thanks for the code sharing and I plan to run some basic experiments based on the tutorial in README file. I use the link [http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar] to download the VOC dataset, but I just found that it does not contain the SegmentationClassAug folder which is required in your code [https://github.com/Haochen-Wang409/U2PL/blob/97284dbbddce4a65d00621bfe77863f39db9ddcd/u2pl/dataset/base.py#L27]. Could you please have a double check? Thanks

Hi, are the results achieved after tta?

Hi, thanks for the previous prompt responses.

Today, I ran an experiment based on this repo (u2pl/experiments/pascal/1464/ours/train.sh) with 4 * V100 (32G/gpu) and I got an CUDA OOM error: RuntimeError: CUDA out of memory. Tried to allocate 16.63 GiB (GPU 0; 31.75 GiB total capacity; 5.13 GiB already allocated; 16.63 GiB free; 13.83 GiB reserved in total by PyTorch). This error is thrown out when the program comes into second epoch, i.e., the contrastive loss term is activated.

Since the tutorial in README only mentions the number of GPU but not the capacity, I wonder how many resouces are required for U2PL? Thanks

Hi, really appreciate your work!

I notice that in comparison with CPS and AEL on the Cityscapes (Table 3), you reproduce these two approaches and report the corresponding results. But the reproduced results are far below their original ones. For example, on the 1/8 protocol, reproduced CPS is 74.31, but original CPS is 77.62, and reproduced AEL is 75.55, but original AEL is 77.90.

So why the performance gap is so large? Is it due to different labeled data splits or other reasons? Thanks a lot.

作者您好，感谢你们精彩工作，有个问题想请教你们一下，为何在您的paper中对比学习损失权重只有0.1？这样会不会导致对比学习loss对模型优化的贡献太少？

Thank you for your expressive and remarkable work and delicate code!! But when I reproducting the results on VOC2012 with 92 examples,I only get 59 with examples,while the reported result is 67.98.Can you offer some suggest? Thank you sincerely!

This is my log file!
seg_20220414_150853.txt

This is my tb about contrastive_loss and unsupervised loss! they're kind of weired and on an upward trend.

i run VOC dataset with 1464 unlabel，the unsupervised_loss is not a gradual reduction，Is this normal ? Or what do you think went wrong？

Hi Haochen, this is a great job. I have a problem with this, i.e., in the semi-supervised task, the "apply_aug" used in the config.yaml is "cutmix", but the generate_cutout_mask() is used in the generate_unsup_data(). Is there a special consideration here or a typo?

学长这个1个gpu能跑起来吗，这个sh train.sh <num_gpu> 中的port是填什么呀...这个port..

Hi, thanks for the code sharing and I just found the problem as mentioned in the title, could you please have a double check and share the complete config file? thanks

Hi, great work! But I just found out that all dataloaders(sup or unsup) are all set to shuffle=False during training. Can you explain why?
For example, in pascal_voc.py code 116 ~ 157

        data_list_unsup = cfg["data_list"].replace("labeled.txt", "unlabeled.txt")
        dset_unsup = voc_dset(
            cfg["data_root"], data_list_unsup, trs_form_unsup, seed, n_sup, split
        )

        sample_sup = DistributedSampler(dset)
        loader_sup = DataLoader(
            dset,
            batch_size=batch_size,
            num_workers=workers,
            sampler=sample_sup,
            shuffle=False,
            pin_memory=True,
            drop_last=True,
        )

        sample_unsup = DistributedSampler(dset_unsup)
        loader_unsup = DataLoader(
            dset_unsup,
            batch_size=batch_size,
            num_workers=workers,
            sampler=sample_unsup,
            shuffle=False,
            pin_memory=True,
            drop_last=True,
        )
        return loader_sup, loader_unsup

Thanks in advance.

我的服务器配置是这样的，然后我按照环境是按照readme文档上来的，为什么输入运行命令会没有反应呢

nice works
When i set num_classes=1 in config file , can i continue to use compute_contra_memobank_loss？

感谢分享代码！请问数据集使用Cityscapes，训练半监督模型需要多少显存？我单卡训练（Tesla P40），显存22.38 G，batchsize2，lr=0.00125，按照其他issues里说的，因为单卡训练所以屏蔽了train_semi.py里的相关代码：
# if cfg.get("sync_bn", True):
# model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)

刚开始训练就爆显存了，训练时卡上没有跑其他模型，想问一下是我设置的问题还是有其他问题？

报错：
RuntimeError: CUDA out of memory. Tried to allocate 146.00 MiB (GPU 0; 22.38 GiB total capacity; 21.32 GiB already allocated; 79.56 MiB free; 21.53 GiB reserved in total by PyTorch)

想请问一下，为什么需要一个表示头g对特征做进一步映射呢？跟直接在编码器的输出特征图上算对比损失有什么不一样呢？
而且表示头是随机初始化的，训练过程中只有对比损失会在这个分支更新参数是吗？怎样保证表示头朝着正确的方向在更新参数呢？

Could you provide the test code on PASCAL VOC? Or how do I get miou after PASCAL training.

Hi,
It is totally unfair to including all your high quality labels in your AuG experiment settings, according to your 5291 sup list in here https://github.com/Haochen-Wang409/U2PL/blob/main/data/splits/pascal/5291/labeled.txt and also your 2646 sup list https://raw.githubusercontent.com/Haochen-Wang409/U2PL/main/data/splits/pascal/2646/labeled.txt. From CPS 1/2 ratio experiment https://github.com/charlesCXK/TorchSemiSeg/blob/main/DATA/pascal_voc/subset_train_aug/pseudoseg_unlabeled_1-2.txt, it contains 733 high quality data with 0.5006830601092896 percentage for all high quality data in total (which is expected). And what you are doing now is boosting your performance based on the supervised label's quality.

IT IS VERY TRICKY and DISHONESTY !!!

Could you report your result based on CPS partial protocol of data splits ?????

I manage cityscapes data dir structure as Prepare data in readme. But, raise an error of not find the file.
I find that file path has been changed in BaseDataset, it add "gtFine_labelTrainIds.png" into file path and this cause error to occur.
What should I do to fix the bug? Thank you!

I repeated experiments on U2PL according to your settings for many times, and the accuracy in the paper can not be achieved under the condition of eight V100. Can you provide the training log to help check details?

作者您好，感谢您精彩的工作，有两个问题想请教 您一下：
（1）通过置信度和熵值过滤噪声伪标签，这两种方式哪个更适合半监督语义分割？
（2）关于您下图中过滤伪标签阈值的确定，按照您paper中所示，通过np.percentile函数选出一定比例的伪标签，但是在训练的初期，会不会存在生成的伪标签大部分noisy的，这样反而会影响训练。是不是得先设置一个熵的阈值，当阈值低于这个阈值时候再用您paper中方式进行过滤伪标签会更好些，这仅是我看完您paper后一些想法，和您交流一下。

nice work!
Follow your implement of ResNet, could you provide the per_trained model of ResNet-50 and others ? I just see ResNet-101 in your README.

Hello, great work, and thanks for sharing it!

I'm trying to train U2PL with a lighter version of resnet as encoder. I see that others have wanted to do the same (#37, #21). I understand that your version of resnet is slightly modified from the official version, so the weights for the official can't be used.

I'm wondering then where you got the weights for resnet101, did you train it from scratch yourself? Additionally, did you write the entire resnet.py code just for this project, or is it borrowed from somewhere? It's strange that resnet.py seems to have support for resnet50, 34, and 18 despite those sizes not being used for U2PL.

Thank you!

Hi, thanks for the awesome work.

I have a question about the ablation study part. In table 6, Dynamic Partition Adjustment (DPA) and high entropy filtering (Unreliable) are ablated. For me, the Unreliable component means the pseudo-label generation process in Eq.(6) and the DPA component means the threshold dynamic adjusment process in Eq.(7). So it means the use of DPA should be based on the use of Unreliable component. However, I found the penultimate row of table 6 shows the result for a method w/ DPA and w/o Unreliable. It is weird to me and I am not sure if my understanding is correct or not.

I tried to use U2PL to train under PASCAL VOC on different split, I modified n_sup:92 in config.yaml, but the final result is similar to full labeled data, have you encountered such problem?

首先，感谢您精彩的工作，我有一个问题想要请教一下您。
我使用您的代码运行了ISPRS的遥感数据集，在运行过程中碰到了一个问题：

[Info] Load ImageNet pretrain from '/home/qyc/MyProject/U2PL-main/path/to/resnet101.pth' 
missing_keys:  [] 
unexpected_keys:  ['fc.weight', 'fc.bias']
[2022-05-24 22:03:30,950][    INFO] # samples: 562
[2022-05-24 22:03:30,950][    INFO] # samples: 112
[2022-05-24 22:03:30,950][    INFO] Get loader Done...
=> loading checkpoint '/home/qyc/MyProject/U2PL-main/path/to/resnet101.pth'
Traceback (most recent call last):
  File "../../../../train_sup.py", line 316, in <module>
    main()
  File "../../../../train_sup.py", line 130, in main
    load_state(cfg["saver"]["pretrain"], model, key="model_state")
  File "/home/qyc/MyProject/U2PL-main/u2pl/utils/utils.py", line 597, in load_state
    state_dict = checkpoint[key]
KeyError: 'model_state'
Traceback (most recent call last):
  File "/home/qyc/anaconda3/envs/pytorch/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/qyc/anaconda3/envs/pytorch/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/qyc/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/home/qyc/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/qyc/anaconda3/envs/pytorch/bin/python', '-u', '../../../../train_sup.py', '--local_rank=0', '--config=config.yaml', '--seed', '2', '--port', '23500']' returned non-zero exit status 1.

          unsup_loss = (
                   compute_unsupervised_loss(
                       pred_u_large,
                       label_u_aug.clone(),
                       drop_percent,
                       pred_u_large_teacher.detach(),
                   )
                   * cfg["trainer"]["unsupervised"].get("loss_weight", 1)
           )

In line 387 of train_semi.py, I did not see Eq. 8 in the paper. Is it a mistake when submitting the code?

Hello, Wang. Long time no see. I have to bother you again!
When calculate contrast loss, we need to get the entropy map and filt it with alpha_t, with this code below:

                    prob = torch.softmax(pred_u_large_teacher, dim=1)
                    entropy = -torch.sum(prob * torch.log(prob + 1e-10), dim=1)

                    low_thresh = np.percentile(entropy[label_u_aug != 255].cpu().numpy().flatten(), alpha_t)
                    low_entropy_mask = (entropy.le(low_thresh).float() * (label_u_aug != 255).bool())

                    high_thresh = np.percentile(entropy[label_u_aug != 255].cpu().numpy().flatten(),100 - alpha_t)
                    high_entropy_mask = (entropy.ge(high_thresh).float() * (label_u_aug != 255).bool())

                    low_mask_all = torch.cat(((label_l.unsqueeze(1) != 255).float(), low_entropy_mask.unsqueeze(1),))
                    high_mask_all = torch.cat(((label_l.unsqueeze(1) != 255).float(), high_entropy_mask.unsqueeze(1),))

But I save the output of low_mask_all and high_mask_all, I found something strange.

For example, as we know that the ideal entropy map should be tend to be 0 at every class, but at least we want that low-entropy representations the target classes we want.

And the result low_mask_all shows nothing (all zero map), but high_mask_allshows the frame of the segmentation target! why? Is it the converse of the real result,

like that low_mask show the frame of the objects,
and high_mask maybe looks like pepper-salt noise or sth.

Best,

你好，我在模型训练的时候碰到这个问题，根据报错的位置检索，发现是教师模块的模型出现了问题，但学生模块的模型和教师模块在结构上和输入上是一样的，为什么只有教师模块的模型报错呢

您好，在训练过程中我使用 sh train.sh 1 0 报错，subprocess.CalledProcessError: Command '['/home/glc/anaconda3/envs/torch1.8/bin/python', '-u', '../../../../train_sup.py', '--local_rank=0', '--config=config.yaml', '--seed', '2', '--port', '0']' returned non-zero exit status 1.
我不知道port 指的是什么？ 需要根据什么信息设置？ 另外单卡时 可以不采用train.sh 可以直接用train_sup.py训练吗？

May I ask why the loss here needs to be divided by the number of threads.

Am I right in thinking that the core of the paper lies in using unreliable pixels( 1[H(pij) > γt]) as a basis for the selection of negative samples for contrastive learning or negative learning? Because I note that Table 4 indicates the implication that adding this filtering rule improves the metric.

Thanks for your works!
when i run semi-supervised model on PASCAL VOC 2012 , but i got all unsup_loss and contrativate loss are 0 in the log.
Why is this?

Hello, thank you for this nice work!
I am trying to train it with ResNet-50 backbone. However, I found that your code cannot load the ImageNet pretrained backbone from torchvision (https://download.pytorch.org/models/resnet50-0676ba61.pth). The pretrained ResNet-101 model from torchvision cannot work as well.

I am wondering what is the difference between your pretrained model and that provided by torchvision.

非常感谢作者的工作！
我在训练时发现 调用cityscapes/gtFine/train/bochum/bochum_000000_022210_gtFine_labelTrainIds.png，而下载的原版cityspace数据集当中只有bochum_000000_000313_gtFine_color.png，bochum_000000_000313_gtFine_instanceIds.png，bochum_000000_000313_gtFine_labelIds.png， bochum_000000_000313_gtFine_polygons.json这四种后缀格式的文件，并没有labelTrainIds.png这种，请问有解决方法吗？另外单卡也可以使用sh train.sh训练吗

Thank you for your work .
Now I use your method to train on my dataset, but my dataset has only one class, just need to distinguish foreground and background.
When I set my num_classes=2 to training , but i got all contrastive loss is 0，why？
In your code ，two-class contrastive loss （compute_binary_memobank_loss）is miss，could you provide it？

Hi, @Haochen-Wang409 . Your work inspired me a lot, especially the use of reliable and unreliale labels. But when I look into the code (line 376 in the train_semi.py), it seems that the unsup_loss uses a fixed percent of reliable labels. In the contrastive loss, the dynamic partition adjustment was applied. I want to know that is it necessary to use the dynamic partition adjustment in the unsupervised loss? Thank you very much.

hi，thank for your works.
if i want to train with only one GPU without DDP, I should set what the learning rate is？

The model structure of deeplabv3+ in U2PL is different from the standard. Have you ever experimented with the same structure as deeplabv3+ ?

感谢各位的文章，这很有启发性！
我想请问下，关于调用loss_helper.compute_contra_memobank_loss中输入的参数low_mask和high_mask，
用法如下：
low_valid_pixel = torch.cat((label_l, label_u), dim=0) * low_mask
high_valid_pixel = torch.cat((label_l, label_u), dim=0) * high_mask

但是，这个mask是怎么定义呢？我留意到在train_semi line419中定义了
low_mask_all = torch.cat( ( (label_l.unsqueeze(1) != 255).float(), low_entropy_mask.unsqueeze(1), ) )
high_mask_all = torch.cat( ( (label_l.unsqueeze(1) != 255).float(), high_entropy_mask.unsqueeze(1), ) )
这边是为了达到什么效果呢？

感谢百忙之中抽空看我的问题！

Hi, I try to reproduce your results reported in your paper but can't reach your results as your paper report.
Because of computation resource limited, I use batch size 8 and learning rate 0.0005 which are half in your paper.

When try to reproduce "full" method using the superparameters mentioned before，I just reach 77.12 (running 3 times and average).
Could you give me some advice to reproduce your method, thanks.
The config file I used in reproduce experiment is as following. Besides, the annotations files are get as you mentioned in your repo.

dataset: # Required.
  type: pascal_semi
  train:
    data_root: xxx/VOCdevkit/VOC2012
    data_list: xxx/U2PL/data/splits/pascal/1464/labeled.txt
    flip: True
    GaussianBlur: False
    rand_resize: [0.5, 2.0]
    #rand_rotation: [-10.0, 10.0]
    crop:
      type: rand
      size: [513, 513] # crop image with HxW size
  val:
    data_root: xxx/VOCdevkit/VOC2012
    data_list: xxx/U2PL/data/splits/pascal/val.txt
    crop:
      type: center
      size: [513, 513] # crop image with HxW size
  batch_size: 2
  n_sup: 1464
  noise_std: 0.1
  workers: 2
  mean: [123.675, 116.28, 103.53]
  std: [58.395, 57.12, 57.375]
  ignore_label: 255

trainer: # Required.
  epochs: 80
  eval_on: True
  optimizer:
    type: SGD
    kwargs:
      lr: 0.0005  # 4GPUs
      momentum: 0.9
      weight_decay: 0.0001
  lr_scheduler:
    mode: poly
    kwargs:
      power: 0.9
  unsupervised:
    TTA: False
    drop_percent: 80
    apply_aug: cutmix
  contrastive:
    negative_high_entropy: True
    low_rank: 3
    high_rank: 20
    current_class_threshold: 0.3
    current_class_negative_threshold: 1
    unsupervised_entropy_ignore: 80
    low_entropy_threshold: 20
    num_negatives: 50
    num_queries: 256
    temperature: 0.5

saver:
  snapshot_dir: checkpoints
  pretrain: ''

criterion:
  type: CELoss
  kwargs:
    use_weight: False

net: # Required.
  num_classes: 21
  sync_bn: True
  ema_decay: 0.99
  encoder:
    type: u2pl.models.resnet.resnet101
    kwargs:
      multi_grid: True
      zero_init_residual: True
      fpn: True
      replace_stride_with_dilation: [False, True, True]  #layer0...1 is fixed, layer2...4
  decoder:
    type: u2pl.models.decoder.dec_deeplabv3_plus
    kwargs:
      inner_planes: 256
      dilations: [12, 24, 36]

hi, i find a diff between paper and your config.yml. num_queries(N) in paper is 50, num_negatives(M) in paper(section 3.1) is 256. but in your config file, they are contrary, as follow:

    num_negatives: 50
    num_queries: 256
    temperature: 0.5

did I understand something wrong?

在这里是将有标签数据量扩充到无标签的数据量吗？如果是，这样扩充不会对网络训练产生影响吗？

你好，我想问一下这个预训练的ResNet网络权重是用什么数据进行预训练呢，之前下载过torchvision中的resnet101的权重包，发现网络有所改动，无法导入，如果可以的话，能不能分享一下预训练的权重包呢，谢谢

想请教一下，代码里没有找到对解码器和ASPP模块的参数初始化的代码

Thank you for your expressive work and delicate code. When reproducing the results on VOC, I can reproduce the results with 183, 366, 732, 1464 examples, but only get about 52 with 92 examples, while the reported result is 67.98. Can you offer some help? Sincere gratitude to you.

dequeue_and_enqueue为什么返回batch_size？
期待你的回答。

According to your article, the innovation point of the method you proposed is to use unreliable pixels as negative samples for training. For the segmentation problem with only two categories, how can unreliable pixels be trained as negative samples? It seems that unreliable pixels cannot provide negative information at this time.

sh train.sh 1 23500 encounter a mistake ，I don't know where I am going wrong, can you help me

Hi, thanks for opening source this brilliant work.

For this teacher-student architecture, mentioned, follow the typical self-training framework ... in the 3.1. Sec.

1. But I still wonder is it ok to only use the student? like, select the positive and negative features from the student output.
2. In the code, the student and teacher are initialized in the same way, using the ImageNet pre-trained params. but the teacher is updated very slow, so it might not learn the feature well, will the positive teacher features be very different from the student anchor?

Looking forward to your reply.