I'm going to run this code. Your paper say: To be noted,coarse data are not used in this work. But your DATASETs.md say: Both Coarse data and Fine data are used. So I'm confused about the coarse data is used in this paper?

I tried to export sfnet_r18_78.pth to onnx, but encountered an error of linspace is not supported by onnx.
Have you an experience to convert your model to onnx?

Thanks,

RT

Hello.
Thanks for publishing a great code. I want to ask a question: The uploaded pre-training model resnet18-deep-inplane128.pth is pre-trained in which dataset? ImageNet or Mapillary？Thank you!

Hi,
Can you tell me what license applies to this repo?

i met problem with OhemCrossEntropy2dTensor ,when i use it in my own project, anything should i notice when use this loss?

Can you provide the TensorRT implementation code, or some useful help to implement the TensorRT deployment? Thanks

First thank you for making this promising work public !
I came across some problems while running the demo.

UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.

It seems like the version you recommend is pytorch>=1.2, which is not enough for me to deduce what value should be set here.

norm = torch.tensor([[[[out_w, out_h]]]]).type_as(input).to(input.device)
h = torch.linspace(-1.0, 1.0, out_h).view(-1, 1).repeat(1, out_w)
w = torch.linspace(-1.0, 1.0, out_w).repeat(out_h, 1)
grid = torch.cat((w.unsqueeze(2), h.unsqueeze(2)), 2)
grid = grid.repeat(n, 1, 1, 1).type_as(input).to(input.device)
grid = grid + flow.permute(0, 2, 3, 1) / norm

在torchcv中，能不能说明一下 啊
norm = torch.tensor([[[[out_w, out_h]]]]).type_as(input).to(input.device)
w = torch.linspace(-1.0, 1.0, out_h).view(-1, 1).repeat(1, out_w)
h = torch.linspace(-1.0, 1.0, out_w).repeat(out_h, 1)
grid = torch.cat((h.unsqueeze(2), w.unsqueeze(2)), 2)
grid = grid.repeat(n, 1, 1, 1).type_as(input).to(input.device)
grid = grid + flow.permute(0, 2, 3, 1) / norm

Hi, when I trained trining train_cityscapes_sfnet_dfv1.sh, an error as fllows which is from b,c,h,w = pred.size() in loss.py:
'tuple object has no attribute size‘
I fond that pred contains 2 predicted tensors whose shape is 2x19x64x64 . Then, I use pred[0] as pred but other error come out as fllows:

File "/media/ssd2/yr/CV/segmentation/SFSegNets/loss.py", line 272, in forward
prob = prob.masked_fill_(~valid_mask, 1)
RuntimeError: The expanded size of the tensor (8192) must match the existing size (2097152) at non-singleton dimension 1. Ta rget sizes: [19, 8192]. Tensor sizes: [2097152]

What shold I do to train train_cityscapes_sfnet_dfv1.sh correctly?

看论文里有说we re-implement grid sampling operator by CUDA to be used together with TensorRT
想看下这部分是怎么实现的

it’s very important！3ks

您好！
下载了您的训练模型，在使用eval，默认参数在1080下测试时，速度比较慢，DFnetv1只有9秒/帧，是我哪里设置不对吗？
该如何处理可以达到您论文中描述的速度。谢谢！

Hello.
Thanks for publishing a great code.
When I made multi-scale predictions(eval.py) using a model I had trained myself, MIoU was very low (2.5%), for unknown reasons. Using my own code(single scale prediction) for calculating MIoU, my model was able to achieve 75%MIoU on the validation set.
If you know the reason, please reply, thank you.

def draw_flow_arrow(img, flow, id, step=32):

请问这里的id是代表什么呢？

good job!

what is the purpose of cityscapes_train_cv0_tile1024.json and how to create this file for a new dateset.
thanks!

Hi, first of all, thanks for sharing your work for open access.
When reviewing the code, I found that the decoding part in the FPN. The paper said that it uses multiple FAM to process feature maps of outputs from each decoding layer, and concatenate results together into 4x stride resolution, then predict their final result.
While in the code, the FAM module only used in the step-by-step upsample in the FPN architecture, without using it to get the final concatenated logits.
I wanna know that does it matter for performance?
For now, I think the result is good, it's a good trade-off between speed and accuracy. Besides, adding more FAM may lower the speed, so if anyone tried, please share your experience.

I can get the eval result, but when I use the trained model to get the test result , I can't the result of papers.So can you provide the test code?

How to use the 'pooling' option for --inference_mode from eval.py?

请问Cityscapes_label.py中的license plate是否是多余的呢？加上这一类num_classes是19吗？

Please have a close look at your code:

def forward(self, x, gts=None):
        x_size = x.size()  # 800
        x0 = self.layer0(x)  # 400
        x1 = self.layer1(x0)  # 400
        x2 = self.layer2(x1)  # 100
        x3 = self.layer3(x2)  # 100
        x4 = self.layer4(x3)  # 100
        x = self.head([x1, x2, x3, x4])
        # main_out = Upsample(x[0], x_size[2:])
        main_out = F.upsample(x[0], size=x_size[2:], mode='bilinear', align_corners=True)
        # main_out = Upsample(x[0], x_size[2:])
        print('main out: {}'.format(main_out))
        print(main_out.shape)
        if self.training:
            if not self.fpn_dsn:
                return self.criterion(main_out, gts)
            return self.criterion(x, gts)
        return main_out

in sfnet_resnet line 178, if not self.fpn_dsn which I set dsn to be True, then:

You will using return self.criterion(x, gts)
But, x is a tuple.....
How do u able to train in this logic???

CrossEntropy2D can not performance .size or .dim on tuple, do u know?

I am totally don't know how do u able to train it.

I got Nan loss:

04-08 10:51:43.489 NaN or Inf found in input tensor.
04-08 10:51:43.559 [epoch 28], [iter 8136 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:43.560 NaN or Inf found in input tensor.
04-08 10:51:43.632 [epoch 28], [iter 8137 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:43.632 NaN or Inf found in input tensor.
04-08 10:51:43.965 [epoch 28], [iter 8138 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:43.965 NaN or Inf found in input tensor.
04-08 10:51:44.036 [epoch 28], [iter 8139 / 13576], [train main loss nan], [lr 0.001678]
04-08 10:51:44.037 NaN or Inf found in input tensor.

Please provide pretrained weights with this model: network.sfnet_resnet.DeepR18_SF_deeply_dsn

I tried both network.sfnet_resnet.DeepR18_SF_deeply_dsn and network.sfnet_resnet.DeepR18_SF_deeply, none of them work.

您好!我把从下面网址下载的预训练权重文件load进去之后，显示层不匹配。请问我该怎么处理呢？您可以提供相应的预训练权重文件吗？

'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',
'resnet101': 'https://download.pytorch.org/models/resnet101-5d3b4d8f.pth',

@lxtGH
Hi, thank you for your work,
but when i run the demo_folder.py with sfnet_r18_78.pth to test the cityscapes image(1024x2048), i got the message that: ' inference takes 75.56 seconds, which takes 0.42 seconds per image, including saving results', i have one GTX 1080Ti. the 0.42 seconds per image is much longer than the 26 fps of the SFNet(ResNet-18) in your paper.
Could you please tell me what is the problem?
thank you.

The miou of the dfv2 trained model you released is 75.8% on validation set while the miou is 77.8% on the test set reported in your paper,Would you mind to explain why the results are so different。

The fps testing file in utils/misc.py uses tensor size of 896 to do speed test. However, in paper, (1024, 2048) is used to inference. Could you please explain for this mistake?

Hi, thanks for the great work!

I am just wondering if you are considering putting your model on the Papers with Code leaderboard (https://paperswithcode.com/sota/real-time-semantic-segmentation-on-cityscapes).

It helps other people find your work more easily.

您好，我看了一下torchcv的sfnet实现，感觉和论文上的网络结构图有些出入。

文章中最后融合decoder各个层的feature maps的时候是应用了FAM，我看sfnet的代码实现中，fusion_list的实现是decoder各层feature maps先通过conv3x3_bn_relu，然后用二次线性插值加上torch.cat实现融合的，我感觉和论文的描述有点不同，也可能是我代码没理解透，我不知道您怎么看这个问题。

Hi, thank you for your nice work.
But I have some questions about the flow_make and flow_warp functions.
In your code, you use a Conv2d self.flow_make = nn.Conv2d(outplane*2, 2, kernel_size=kernel_size, padding=1, bias=False) to make flow. But the Conv2d operator cannot guarantee that the range of output Flow is reasonable. So even you use a norm operation, you can not get a reasonable grid in flow_wap.
Is this a problem?
Looking forward for your reply. Thanks.

Hello.
Thanks for publishing a great code.
I trained SFNet with default setting of scripts/train_cityscapes_sfnet_res18.sh.But i got only 73 mIoU.
Diff:

• Res18 pretrain model.(Model channels(64) does not match the model，so I don't load the pretrain model)
• CityScapes Dir name.(Such as img_dir_name = 'leftImg8bit_trainvaltest')
• Pytorch version. (Some default settings)

使用resnet-18

net = DeepR18_SF_deeply(19, None)
print(net)

可以看到在网络结构里面layer1中使用了downsample
但是我下载sf_r18_799_map_trained.pth输出参数名称发现，layer1 的参数中并不存在 downsample这一步运算，为什么这个结构在训练好的参数中并不存在。

I found the issue which talked about this in repo donnyyou/torchcv

@lxtGH
hi,
i want to train the res18+map ,however, i only find the DeepR18_SF_deeply_dsn network in sfnet_resnet.py,
there is not DeepR18_SF_deeply_map.

I retrained the sfnet with ResNet101, but I can't get the result mIoU81.2, I evaluate the trained model with whole input size, just can get 80.5mIou in single scale

Hi, I find that the SFNet w/ ResNet-18 has adopt dilated convolutions in C4 and C5:

你好同学，首先感谢你开源了这个仓库。
阅读你的代码发现，这里是通过FAM模块计算出两个特征图的流场，再以流场为grid，通过双线性插值将高层语义特征图上采样，请问我的理解是正确的吗？
如果是的话，那么在反向传播的过程中，会计算对于流场的导数吗？或者进一步说，在反向传播的过程中会更新FAM模块的参数吗？按照我浅陋的理解，感觉反向传播的时候对FAM模块的参数不会更新呀。
如果我理解有误，还望指点一二，谢谢同学！

I use train_cityscapes_sfnet_res18.sh to get 78.5. Is this corresponse "resnet18(no-balanced-sample): 78.4 mIoU"?

and how can I get "resnet18: 79.0 mIoU"?

Hi, your code trained with multi GPUs, but I only have a single GPU with 8 memory.

So what should I set to run this code with my single GPU？

I saw torch.distributed.init_process_group(), torch.distributed.all_reduce(). I think this may interrupting my training with single GPU.

我在使用DFv1作为backbone进行训练时，报错显示如下:
https://github.com/ahhLS/picture/blob/main/1.png
断点调试到错误出现的地方，显示如下：
https://github.com/ahhLS/picture/blob/main/2.png
这里的x的是一个tuple，我理解的是x[0]是输入的一个tensor，但是x[0]的shape为[2,9,64,64]，与输入的gts尺寸不匹配。这里的main_out是对特征图进行了上采样，才使得x[0]与gts尺寸一致，从而顺利进行forward。请问这块代码能具体解释一下吗？我如何修改才能进行正确的forward呢？

@lxtGH
thank you for your work.
when i train sfnet_res101 on my own data,i got the error: CUDA outof memory, my own training image is 2710x3384, i have two GTX1080Ti which have about 11G,
Could you please tell me how to solve it.
thank you.

Loss divergency in epoch 2... even I narrow down lr: