gigo-team / bev_lane_det Goto Github PK

Hi,

I'm struggling in understanding the preprocess of openlane dataloader. Would you please give some illustrations of my questions?

1. the meaning of cam_w_extrinsics, which seems like the extrinsic params of the camera;
2. the meaning of maxtrix_camera2camera_w ;
3. difference between cam_extrinsics and cam_extrinsics_persformer;
4. why line 176 uses cam_w_extrinsics once and line 178 also uses cam_extrinsics ?

I used the source code provided by you to train on 4 cards at 1080, and selected the final round of training results as the validation result. The accuracy was only 56.2%. Therefore, I would like to ask you about the training hardware information you used. Thank you very much for your help!

Thanks for your great work! I wonder how fast your model detects the lane while inference. Can FPS be greater than 10?
By the way, looking forward to your code.

In the papar, it just say 'which are derived from the mean value of the in/extrinsic
parameters of the training dataset'.
I want to use Virtual Camera on other dataset with new in/extrinsic parameters. And I can't find corresponding code in this project. I wonder if you can explain the specific calculation method of the intrinsic parameters and extrinsic parameters of the Virtual Camera.

您好！
我看到您在torch 求虚拟相机参数，是现在图像 大概地面位置取四个点，然后根据z=0，再反投到车体坐标系，再接您求h矩阵的后续
但是再c++示例代码里，却是随机取15个高度为0的店，然后再接您后续h矩阵的后续
请问，这两者有区别有什么特别的含义和思考吗？

Is your visualization code consistent with that of openlane? Can you provide your result visualization code, thank you very much!

Thanks for your open-source codes. When I train the model, I find that it seems that you don't use tensorboard to record the loss or F1-score. And the total performance of one epoch isn't recorded either. So I wonder how to choose the best model I have got. Should I run the val_openlane.py on each of the models I trained? Or maybe I miss something important.
Thanks for your reply.

When I try to run the code:"python3 val_apollo.py"
I get a run error:
NameError: name 'Dataset' is not defined

After comparing with val_openlane.py
I think there should be a missing line of code in val_apollo.py which is:
"from torch.utils.data import Dataset,DataLoader"

How can I use this with real time camera video? It is necessary to change codes in virtual camera?

how to use tensorrt to do inference

how can I predict lane class of the openlane

This method uses virtual camera, which is a good idea. However, in the data processing stage, only the image is mapped to the virtual camera, and no corresponding transformation is done to the three-dimensional coordinate points.

您好，

请教一下，看论文包括代码，发现您是通过在真实图像中采样四个点，然后通过一系列转换得到虚拟图像中的四个对应坐标，最终根据四对坐标之间的对应关系求出单应矩阵，我理解这四个点相当于真实图像中所有像素坐标的采样，所以是否有一种方式能够直接根据两个相机的内外参得到对应的单应矩阵那？

感谢

In "val_openlane", theres a line as follows:

cam_extrinsics_persformer[0:2, 3] = 0.0

Why is this the case? Isn't this zeroing the x and y elements of the translation component? Wouldn't it need to do cam_extrinsics_persformer[3, 0:3] = 0.0 in order to zero the last row of the matrix?

Thank you

I am trying to train the model with images from CARLA but the extrinsic matrix that I get from CARLA is used to map from camera coordinates to world coordinates, while the "imageview2ego" method uses vehicle oriented coordinates from what I understood. So calculating the homography matrix between the virtual camera and the actual camera is not working well. Do you have any insights as to how to proceed? Thank you.

When I trained the model at the 25th epoch, it halted for mannual stopping, I rewite the code in train_openlane.py

# TODO template config file.
if __name__ == '__main__':
    import warnings
    warnings.filterwarnings("ignore")
    #openlane_cfg_path= os.getcwd()+'/openlane_config.py'
    openlane_cfg_path= os.path.abspath(os.path.dirname(__file__)) + '/openlane_config.py'
    ckpt_path = '/media/hitbuyi/E8E4A23EE4A20F3E/dataset/openlane1.2/model/BEV_LaneDet/ep025.pth'
    #worker_function('./openlane_config.py', gpu_id=[4, 5, 6, 7])
    #worker_function(openlane_cfg_path, gpu_id=[0])
    worker_function(openlane_cfg_path, gpu_id=[0],checkpoint_path = ckpt_path)

thus

    if checkpoint_path:
        if getattr(configs, "load_optimizer", True):
            resume_training(checkpoint_path, model.module, optimizer, scheduler)
        else:
            load_checkpoint(checkpoint_path, model.module, None)

was called, but the code failed, it prompted the error in load_model.py

def load_checkpoint(checkpoint, net, optimizer=None, map_loc="cuda"):
    sd = torch.load(checkpoint, map_location=map_loc)
    net.load_state_dict(sd['models'])
    if optimizer and sd['optimizer']:
        optimizer.load_state_dict(sd['optimizer'])
    return sd

How to solve this issue?

hi, thanks for your great work, some questions about the generalization of the virtual camera and the STP,
1.did you test the model trained on openlane with other datasets which the camera extrinsics have large difference with openlane camera ?
2. In principle, is it possible for the MLP based STP to generalize to other scenarios or not? why? @qinjian623

Why IPM2ego_matrix is 'M = np.array([[-m_per_pixel, 0, center_x], [0, -m_per_pixel, center_y]])'? Thx

    cam_w_extrinsics = np.array(gt['extrinsic'])
    maxtrix_camera2camera_w = np.array([[0, 0, 1, 0],
                                        [-1, 0, 0, 0],
                                        [0, -1, 0, 0],
                                        [0, 0, 0, 1]], dtype=float)
    cam_extrinsics = cam_w_extrinsics @ maxtrix_camera2camera_w  #
    R_vg = np.array([[0, 1, 0],
                     [-1, 0, 0],
                     [0, 0, 1]], dtype=float)
    R_gc = np.array([[1, 0, 0],
                     [0, 0, 1],
                     [0, -1, 0]], dtype=float)
    cam_extrinsics_persformer = copy.deepcopy(cam_w_extrinsics)
    cam_extrinsics_persformer[:3, :3] = np.matmul(np.matmul(
        np.matmul(np.linalg.inv(R_vg), cam_extrinsics_persformer[:3, :3]),
        R_vg), R_gc)
    cam_extrinsics_persformer[0:2, 3] = 0.0
    matrix_lane2persformer = cam_extrinsics_persformer @ np.linalg.inv(maxtrix_camera2camera_w)

matrix_lane2persformer 这个矩阵代表的是什么

请问 FCTransform 输出的feature大小为啥是(25,5)的，视角转换后 这个 25 和5是怎么设定的？有一定依据吗

Could I use your algorithm in an opensource project? And which license should I follow?

can you give me source code through my email: [email protected]
thank you very much.

Hello all,

I ran the Openlane experiment based on the provided codebase and didn't change any parameters in the config file. I tested multiple models saved at different epochs and only got the best results of 56.3% F-score on Openlane validation set, which is still 2% lower than the number shown in your paper (i.e., 58.4% F-score). Is there anything wrong? Could you please give me some advice?

In the original evaluation scheme of the Apollo Synthetic dataset, ground truth lanes are considered in the lateral (x-) range [-30 m, 30 m]. In the paper I only found that the final grid size used for both datasets is [-10 m, 10 m] lateral and [3 m, 103 m] longitudinal. To my understanding there should be no key-points to represent lanes in the range [-30 m, -10 m] and [10 m, 30 m], but still the detection scores are pretty high and far-range errors fairly low.
How is this possible? Was the grid size enlarged for Apollo or was the Apollo evaluation scheme adapted with ground truth lanes being pruned to [-10 m, 10 m] lateral (as it is the case for OpenLane evaluation)?

Hi, thanks for your great work, did you have any video demos on openlane and ONCE-3DLane?
Looking forward to your reply, thanks! @qinjian623

全部是默认设置，epoch=50. val结果如下所示：
{'f1_score': 0.5584564102804471,
'precision': 0.6719318365019449,
'recall': 0.4777714921062119,
'x_error_close': 0.25970785558032083,
'x_error_far': 0.6890227179538931,
'z_error_close': 0.1996908412880719,
'z_error_far': 0.6315434166948418}