in the segmentation folder -> segmentation_val/segmentor/tester.py line183

def __relabel(self, label_map):
    height, width = label_map.shape
    label_dst = np.zeros((height, width), dtype=np.uint8)
    for i in range(self.configer.get('data', 'num_classes')):
        label_dst[label_map == i] = self.configer.get('data', 'label_list')[i]
  
    label_dst = np.array(label_dst, dtype=np.uint8)
  
    return label_dst

if self.configer.exists('data', 'reduce_zero_label') and self.configer.get('data', 'reduce_zero_label'):
    label_img = label_img + 1
    label_img = label_img.astype(np.uint8)
if self.configer.exists('data', 'label_list'):
    label_img_ = self.__relabel(label_img)
else:
    label_img_ = label_img

for cocostuff dataset (171 num classes), the origin predicted classes range from 0-170, after +1, it range from 1-171, then feed the label_img into __relabel() func. However, the loop in __relabel() range from 0-170, and the class 171 is not be operated.

I want to use a HRFormer model to do some classification tasks, can you offer me an onnx model?

I appreciate you offering the repository but I could't understand.
why comment out like this?
# + relative_position_bias.unsqueeze(0)
It seems like you don't apply positional embedding in the code.

Do you verify the download link? I fail to download it.

If ran out of memory with exact same params as in training (which worked).
Loading the model first to cpu fixes the problem:

resume_dict = torch.load(self.configer.get('network', 'resume'),map_location='cpu')

maybe it helps somebody

021-08-25 14:51:29,793 INFO [data_helper.py, 126] Input keys: ['img']
2021-08-25 14:51:29,793 INFO [data_helper.py, 127] Target keys: ['labelmap']
Traceback (most recent call last):
File "/home/rsa-key-20190908/HRFormer/seg/main.py", line 541, in
model.train()
File "/home/rsa-key-20190908/HRFormer/seg/segmentor/trainer.py", line 438, in train
self.__train()
File "/home/rsa-key-20190908/HRFormer/seg/segmentor/trainer.py", line 187, in __train
outputs = self.seg_net(*inputs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 705, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in call_impl
result = self.forward(*input, **kwargs)
File "/home/rsa-key-20190908/HRFormer/seg/lib/models/nets/hrt.py", line 117, in forward
x = self.backbone(x)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/rsa-key-20190908/HRFormer/seg/lib/models/backbones/hrt/hrt_backbone.py", line 579, in forward
y_list = self.stage3(x_list)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 119, in forward
input = module(input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/rsa-key-20190908/HRFormer/seg/lib/models/backbones/hrt/hrt_backbone.py", line 282, in forward
x[i] = self.branchesi
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 119, in forward
input = module(input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/rsa-key-20190908/HRFormer/seg/lib/models/backbones/hrt/modules/transformer_block.py", line 103, in forward
x = x + self.drop_path(self.attn(self.norm1(x), H, W))
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/rsa-key-20190908/HRFormer/seg/lib/models/backbones/hrt/modules/multihead_isa_pool_attention.py", line 41, in forward
out, _, _ = self.attn(x_permute, x_permute, x_permute, rpe=self.with_rpe, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/rsa-key-20190908/HRFormer/seg/lib/models/backbones/hrt/modules/multihead_isa_attention.py", line 116, in forward
rpe=rpe,
File "/home/rsa-key-20190908/HRFormer/seg/lib/models/backbones/hrt/modules/multihead_isa_attention.py", line 311, in multi_head_attention_forward
) + relative_position_bias.unsqueeze(0)
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 6.64 GiB already allocated; 27.25 MiB free; 6.66 GiB reserved in total by PyTorch)
Killing subprocess 6170

Thanks for your great job. I have some trouble for reproducing the segmentation results of cityscapes. Then I check the log and find out it might be the problem of pretrained models. For now I use the ImageNet model released as pretrained. Can you release the pretrained model for cityscapes? Thanks a lot!

Hi, thanks for you work.
i found there is not config for GRT tiny in pose estimation. could you provide some details about tiny's config? for my opion, i think i only needs to modify the number of chennal and module according to the cls model, is it right? but i also found the FFN_TYPES is differnt as well. should i ignore this? thank you very much!

KeyError: "TopDown: 'HRT is not in the models registry'"请问怎么解决？谢谢！

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/amFNsyUBvm or add me on WeChat (van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

Attention: please create a new virtual environment for OpenMMLab 2.0.

Hi,

I am trying test pretrained models (ade20k segmentation model) on custom data but the default loader seems to not read the images files in label path even when the data in available in the file path.

Kindly advise how test on custom data. Also please mention the file structure needed for the data testing.

Appreciate your help.

Thankyou!

I ran the run_flops.dist and got the warnings below:

WARNING:root:Skipped operation aten::layer_norm 76 time(s)
WARNING:root:Skipped operation aten::mul_ 152 time(s)
WARNING:root:Skipped operation aten::gelu 114 time(s)
WARNING:root:Skipped operation aten::upsample_nearest2d 22 time(s)
WARNING:root:Skipped operation aten::upsample_bilinear2d 22 time(s)
WARNING:root:Skipped operation aten::avg_pool2d 1 time(s)

the flops for hrt_tiny was indeed 1.83G though
So does it mean that the operations in the warnings should not be considered when calculating flops?

What is the inference speed for e.g. semantic segmentation using 1024x1024 (referring to table 5)? Measured on GPU of your choice, just to get a feeling?

Hi, thank you for your great work.

The number of FLOPs and the numbers of parameters are less than Swin Transformer, however the training time of HRFormer is at least 2 times longer than Swin, and 3 times longer than HRNet.
I guess the gradient calculation is very long because of a lot of reshape operations ? Is there a way to optimize that ?

Thank you

Can convert HRFormer backbone into TensorRT? I want to speed it, but the structure is so complex, especially 'class MHA_(MultiheadAttention)', can you give some advice?

the computation cost of HRF-s 256 isd about 2.8G flops. but when i training it, i found that it is significantly slower than the hrnet which have about 7.9 Gflops do you know how to solve it? thanks

Hi，I‘m very interested in your work about the Local Self-Attention and feature fusion in Transformer。But I have a doubt that Because the input image size for the image classification task in the source code is fixed, 224 or 384, in other words, the size is an integer multiple of 32. If the input size is not fixed, for example the detection task, the input is 800x1333, although the feature map can be divided into window size windows by using padding, but for the key_ padding_ mask, how should the mask be handled?

The shape of attention weights map is [bs x H/7 x W/7, 49, 49], default there window size is 7, but the key padding mask shape is
[1, HW], so how can I convert this mask to match the attention weights map。

I sincerely hope you can give me some advice about this question. Thanks !

When I use HRT model for SEG task, I find that the same PID will occupy GPU memory on different GPU.

eg:
PID equal to 13165 takes up memory on both GPU

The paper link on github is not available, and it is not found on arxiv. Is that possible to update the link? Thanks.

Hard to read. I have to open tens of .py and a script files, reading from main.py to hrt.py to xx.py, then i found i have to refer configuration from the script file...

有没有简单的能调用的推理代码。给张图就能跑的？

你现在这个工程，实在是复杂

Hello, I observes in the class PadBlock, the operation you have done is
"n (qh ph) (qw pw) c -> (ph pw) (n qh qw) c"
which you makes the padding group as batch dim. Therefore, it may cause a problem that you consider the pad-group wise attention across all batches. Do you think the permutation should be
"n (qh ph) (qw pw) c -> (n ph pw) (qh qw) c"

the provided pretrained seg model can not load, since the link broke.

Did it provide hrt_ pretrained_ Models? I didn't find it. If it already involved, can anybody post it？Thanks.

I tried to run the test of HRFormer on ImageNet-1k, but the test result was strange. The top-1 accuracy is about 2.0%

Test command

bash run_eval.sh hrt/hrt_tiny ~/Downloads/hrt_tiny_imagenet_pretrained_top1_786.pth  ~/data/imagenet

Test output

[2022-09-06 15:00:15 hrt_tiny](main.py 157): INFO number of params: 8035820
All checkpoints founded in output/hrt_tiny/default: []
[2022-09-06 15:00:15 hrt_tiny](main.py 184): INFO no checkpoint found in output/hrt_tiny/default, ignoring auto resume
[2022-09-06 15:00:15 hrt_tiny](utils.py 21): INFO ==============> Resuming form /home/mzr/Downloads/hrt_tiny_imagenet_pretrained_top1_786.pth....................
[2022-09-06 15:00:15 hrt_tiny](utils.py 31): INFO <All keys matched successfully>
[2022-09-06 15:00:19 hrt_tiny](main.py 389): INFO Test: [0/391]	Time 4.122 (4.122)	Loss 8.9438 (8.9438)	Acc@1 2.344 (2.344)	Acc@5 4.688 (4.688)	Mem 2309MB
[2022-09-06 15:00:29 hrt_tiny](main.py 389): INFO Test: [10/391]	Time 1.028 (1.279)	Loss 9.0749 (9.3455)	Acc@1 5.469 (2.486)	Acc@5 12.500 (7.031)	Mem 2309MB
[2022-09-06 15:00:39 hrt_tiny](main.py 389): INFO Test: [20/391]	Time 1.027 (1.159)	Loss 9.9610 (9.3413)	Acc@1 0.781 (2.269)	Acc@5 4.688 (7.403)	Mem 2309MB
[2022-09-06 15:00:49 hrt_tiny](main.py 389): INFO Test: [30/391]	Time 0.952 (1.103)	Loss 9.1598 (9.3309)	Acc@1 1.562 (2.293)	Acc@5 7.812 (7.359)	Mem 2309MB
[2022-09-06 15:00:59 hrt_tiny](main.py 389): INFO Test: [40/391]	Time 0.951 (1.071)	Loss 9.3239 (9.3605)	Acc@1 0.781 (2.210)	Acc@5 4.688 (7.241)	Mem 2309MB
[2022-09-06 15:01:09 hrt_tiny](main.py 389): INFO Test: [50/391]	Time 0.952 (1.049)	Loss 9.7051 (9.3650)	Acc@1 0.781 (2.191)	Acc@5 3.125 (7.200)	Mem 2309MB
[2022-09-06 15:01:18 hrt_tiny](main.py 389): INFO Test: [60/391]	Time 0.951 (1.035)	Loss 9.5935 (9.3584)	Acc@1 1.562 (2.075)	Acc@5 7.812 (7.095)	Mem 2309MB
...

The environment is brand new according to the install instruction, and the checkpoint is from https://github.com/HRNet/HRFormer/releases/tag/v1.0.0 . The only change is I disabled the amp.

Could you provide some results to demonstrate the performance gain when adding the OCR module on semantic segmentation tasks? Due to the wide usage of self-attention in backbone, is it necessary to use the ocr module which brings extra computation?

Hi,

When I installed mmpose in this repo, I found there is no mmpose/version.py file.

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/chenshoufa/workspace/HRFormer/pose/setup.py", line 105, in <module>
        version=get_version(),
      File "/home/chenshoufa/workspace/HRFormer/pose/setup.py", line 14, in get_version
        with open(version_file, 'r') as f:
    FileNotFoundError: [Errno 2] No such file or directory: 'mmpose/version.py'

Hi, thanks for your work!
I'm trying to train your model in custom config from scratch, but have not found any pre-trained model on ImageNet-1K.
Do you plan to share these models?

Hello, what is the command format during training? I can't load the configuration file. How to write after config in the command line?

`
FutureWarning,
WARNING:torch.distributed.run:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

Traceback (most recent call last):
File "tools/train.py", line 168, in
main()
File "tools/train.py", line 122, in main
env_info_dict = collect_env()
File "/dataset/wh/wh_code/HRFormer-main/pose/mmpose/utils/collect_env.py", line 8, in collect_env
env_info = collect_basic_env()
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/utils/env.py", line 85, in collect_env
from mmcv.ops import get_compiler_version, get_compiling_cuda_version
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/ops/init.py", line 1, in
from .bbox import bbox_overlaps
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/ops/bbox.py", line 3, in
ext_module = ext_loader.load_ext('_ext', ['bbox_overlaps'])
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/utils/ext_loader.py", line 12, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _Z13__THCudaCheck9cudaErrorPKci
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 42674) of binary: /home/celia/anaconda3/envs/open-mmlab/bin/python
Traceback (most recent call last):
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/run.py", line 718, in run
)(*cmd_args)
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/celia/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

tools/train.py FAILED

Failures:
[1]:
time : 2022-10-24_10:03:43
host : omnisky
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 42675)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2022-10-24_10:03:43
host : omnisky
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 42676)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2022-10-24_10:03:43
host : omnisky
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 42677)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
time : 2022-10-24_10:03:43
host : omnisky
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 42674)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================`

Thank you for sharing your work.
I have some problem running it, and I suspect it may be a version inconsistency problem.
So what are the specific version numbers of the torch and MMPose you used?