er-muyue / bemapnet Goto Github PK

Thanks for your interesting work!
I wonder how you split the dataset by different conditions.
Which part of information do you use in the nuScenes dataset?

Hi, @er-muyue, thanks for your great work!

I've noticed that there is a huge performance gap between results of different evaluation protocol, but details of your own evaluation protocol are missing. Could you please provide more information? Thanks a lot!

i can access the npy file after load it, it give me shape = ()
and i can t visualize it

Which nuscene data set I should download?

I have tried to reproduce the Pivot Dynamic Matching algorithm of PivotNet. But I find that it is very slow during training. Could you please give some optimization ideas or have I made some mistakes in my reproduction? Thanks in advance!

When will you plan to open your codes?

When I run the bezier_converter, it said "No such file or directory: './data/nuscenes/maps/expansion/boston-seaport.json". How can I fix it?

For segmentation head, why you set the output channel = 2, which means [b, 2, 400, 200], but then when calculating the loss function you only use dt_masks[:, 1, :, :] ? Why don't you predict only one channel, like [b, 1, 400, 200] ? Thanks

Hi,@er-muyue

Thanks for your greak work! I notice that you push a release version, but it's empty. Would you please open source the code again?

How can I visualize the inference result and generate the video as you showed?

Hi, thanks for the code. Could you help me understand more why you do cross attention first in the instance decoder, instead of self-attention? As some architecture like DETR, they run with self-attention first. Thanks a lot.

@er-muyue Hi, thanks for your great work. PDM elaborates on the point-level matching for PivotNet, but I did not find the details about how to do instance-level matching in the paper.

Specifically, PDM assumes that "We consider the point matching problem between a predicted sequence Sˆ = {vˆn} and a ground truth sequence S^p = {v_n}", but what is your approach to assigning a ground truth sequence to a prediction?

Could you please explain more? Thanks.

Hi, thanks for your greate work, when will you release the codes.
Looking forward to your reply, thanks! @er-muyue

Hello,

Im trying to run training process with the command mentioned in readme.
bash run.sh train bemapnet_nuscenes_swint 30

I completed all of the setup steps and my project directory same as mentioned in readme.

But i have this error

assets/weights/upernet_swin_tiny_patch4_window7_512x512.pth
Backbone missing_keys: []
Backbone unexpected_keys: ['norm0.weight', 'norm0.bias']
Traceback (most recent call last):
File "configs/bemapnet_nuscenes_swint.py", line 314, in
BeMapNetCli(Exp).run()
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/core.py", line 172, in run
self.dispatch(self.executor)
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/core.py", line 169, in dispatch
executor_func()
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/core.py", line 153, in executor
self.get_trainer().train()
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/core.py", line 143, in get_trainer
trainer = Trainer(exp=exp, callbacks=callbacks, logger=logger, evaluator=evaluator)
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/executor.py", line 104, in init
super(Trainer, self).init(exp, callbacks, logger)
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/executor.py", line 62, in init
self._invoke_callback("after_init")
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/executor.py", line 93, in _invoke_callback
func(self, *args, **kwargs)
File "/home/adastec/catkin_bemapnet/BeMapNet/bemapnet/engine/environ.py", line 91, in after_init
ranks = np.arange(self.world_size()).reshape(-1, self.sync_bn)
ValueError: cannot reshape array of size 1 into shape (8)
[Epoch]: 0%| | 0/30 [00:00<?, ?it/s

Python version: Python 3.8.10
CUDA Version: 11.8
Torch version: 2.0.1+cu117 (also tried with 1.10.1+cu111)

Hi, thanks for your great work!
May I please ask when would the source code of pivotnet be released?
Looking forward to your reply! @er-muyue

Hi~ It seems that the model does not converge at 4 gpus? Do you have any ideas about this problem? And can you tell me how the thickness of semantic lane affects the performance?

Thanks for your works!
If I want to set batch size to 2 in run.sh, do I need to set sync_bn to 16 (8*2) ?
Looking forward to your kindly reply!

When will you plan to open your codes?

We use your BeMapNet-SwinT checkpoint to evaluate nuScenes val, but get results which is different from your github results.
Ours: mAP@EASY 70, mAP@HARD 55.3
Yours: mAP@EASY 67, mAP@HARD 49.1
Our test setting and environments
python==3.8, torch==1.9.1, CUDA==11.4, mmcv-full==1.4.0, pillow==8.1.2, numpy==1.21.6, detectron2(build from source)
We use 1*A6000 to test

@er-muyue Hi, thanks for your great work. About function get_embedding in position_encoding.py， why just do xy_world_coords = xy_world_coords * depth.unsqueeze(-1), without add trans_xy

how to visualize the npy file ? can you provide a script for visualization

Hi, your job is really impressive, here is one question I wanna figure out.

From the eval results given in BeMapNet & Pivot Net, it seems that BeMapNet actually have better results than PivotNet, is that true? Why the results from BeMapNet were not compared with pivot net? If there is any different, could you please point out? Thank you

Thank you for opening this to the public. Greate work!

I tried testing the published weight but couldn't get it to work. The error is as follows:

2023-08-23 15:46:08.332 | INFO     | ...engine.callbacks:load_checkpoint:110 - Loading parameters from checkpoint assets/weights/bemapnet_nuscenes_swint_ep110_model.pth
.../site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
assets/weights/upernet_swin_tiny_patch4_window7_512x512.pth
Backbone missing_keys: []
Backbone unexpected_keys: ['norm0.weight', 'norm0.bias']
Traceback (most recent call last):
  File "configs/bemapnet_nuscenes_swint.py", line 314, in <module>
    BeMapNetCli(Exp).run()
  File ".../engine/core.py", line 172, in run
    self.dispatch(self.executor)
  File ".../engine/core.py", line 169, in dispatch
    executor_func()
  File ".../engine/core.py", line 148, in executor
    self.get_evaluator().eval()
  File ".../engine/core.py", line 190, in get_evaluator
    evaluator = BeMapNetEvaluator(exp=exp, callbacks=callbacks, logger=logger)
  File ".../engine/executor.py", line 190, in __init__
    super(BeMapNetEvaluator, self).__init__(exp, callbacks, logger)
  File ".../engine/executor.py", line 61, in __init__
    self._invoke_callback("load_checkpoint")
  File ".../engine/executor.py", line 93, in _invoke_callback
    func(self, *args, **kwargs)
  File ".../engine/callbacks.py", line 134, in load_checkpoint
    trainer.optimizer.load_state_dict(checkpoint["optimizer_state"])
KeyError: 'optimizer_state'

When I change the training batch size to 2

python3 configs/"${CONFIG_NAME}".py -d 0-15 -b 2 -e ${NUM_EPOCHS} --sync_bn 16 --no-clearml

I got the following error

RuntimeError: Boolean value of Tensor with more than one value is ambiguous

Can I train with batch size 2?

It worked fine when I use batch size 1 (-b 1)