wangyueft / dcp Goto Github PK

When I set "strict" "True" in load_state_dict, I find errors "Unexpected key(s) in state_dict:...". Because in "pointer.model.encoder.layers.0.sublayer.0.norm", there are redundant parameters.
for example:
44 pointer.model.encoder.layers.0.sublayer.0.norm.norm.weight
45 pointer.model.encoder.layers.0.sublayer.0.norm.norm.bias
46 pointer.model.encoder.layers.0.sublayer.0.norm.norm.running_mean
47 pointer.model.encoder.layers.0.sublayer.0.norm.norm.running_var
48 pointer.model.encoder.layers.0.sublayer.0.norm.norm.num_batches_tracked

Hi, sorry to bother. When I test it with a model cloud as a souce and a transformed cloud(by random rotation and translation) as target, it works amazingly.
But when I use a incomplete point cloud(a RGBD projection to 3D, incomplete here is point cloud from one perspective) as source and a complete model in the RGBD, it works poorly. I wonder it is my mistaken operation or the model itself? Will it perform as perfectly as it is on complete ones now if I train it with incomplete point cloud?
Thanks!
Best
Cheng

Not really an issue, just a friendly review: there is a typo in the title of readme: Deep Cloeset Point.

=)

 scores = torch.matmul(src_embedding.transpose(2, 1).contiguous(), tgt_embedding) / math.sqrt(d_k)
 scores = torch.softmax(scores, dim=2)
 src_corr = torch.matmul(tgt, scores.transpose(2, 1).contiguous())

I am confused why muplying target by scores, why not just use target cloud as the src_corr?

File "D:\Project\Python_project\MyProject\dcp-master\main.py", line 214, in train_one_epoch
total_loss += loss.item() * batch_size
^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Hello, I would like to ask what is the reason for the above problem when running the source code? It previously appeared when running to the 70th epoch.

Excuse me, I would like to ask you how to test your own code?

Hello, I used the pre-trained parameters for testing and saved the virtual corresponding points generated by the y linear weighting generated by the network, but unfortunately, these points are distributed in the middle of the point cloud, how to explain it?

Dear DCP authors,

Is there a specific license under which the code is released? The repository currently doesn't specify one.

Thank you,
Andrei

I check your code and I couldn't understand the graph feature with respect to dgcnn paper. It turns out that the original authors used to have the bug in line 72 of your code
feature = torch.cat((feature, x), dim=3).permute(0, 3, 1, 2) and it should be

feature = torch.cat((feature - x, x), dim=3).permute(0, 3, 1, 2)

Thanks for de code, I haven't tested, I'm still checking that it matches the theory

I have ply files and I want to use the ply files for training. I have to make this data available...

Currently, the first point cloud is randomly sampled from the ground-truth point cloud. And the second point cloud is rotated and translated from the first point cloud as indicated by the following line of code. These two point clouds have an exact one-to-one correspondence and the relative point distribution in each local neighborhood is exactly the same.
I have tried to randomly sample the second point cloud from the rotated and tranlated ground-truth point cloud, which is a more practical scenario since we should not expect the point clouds for registration have an exact one-to-one correspondence, and the performance decreases dramatically.
This issue should draw the attention of the authors: this issue in some degree indicates that the network may not learn to do the registration, but learn to recognize the relative local point distribution.

I tested the code with my data. The running time was 4s, the computer was 1070, and the video memory was 8 gigabytes.What is wrong?Excluding the read data time, the network runs in 2. 2 seconds

Thanks for your contribution and sharing your implementation.

However, in this repo, the test dataloader is used for choosing the best model, but the best model is also used for testing. A more strict way for model selection is to split validation dataset from the training part because test dataset should be completely unseen before testing.

Some relative codes are shown as follows:

class ModelNet40(Dataset):
    def __init__(self, num_points, partition='train', gaussian_noise=False, unseen=False, factor=4):
        self.data, self.label = load_data(partition)
        self.num_points = num_points
        self.partition = partition
        self.gaussian_noise = gaussian_noise
        self.unseen = unseen
        self.label = self.label.squeeze()
        self.factor = factor
        if self.unseen:
            ######## simulate testing on first 20 categories while training on last 20 categories
            if self.partition == 'test':
                self.data = self.data[self.label>=20]
                self.label = self.label[self.label>=20]
            elif self.partition == 'train':
                self.data = self.data[self.label<20]
                self.label = self.label[self.label<20]

The partition parameter can only be set as 'test' or 'train' and partition=test is used for evaluating and selecting the best model for training.

Hope for your reply, cheers.

There is an error in generating translation_ba in the code:

where the transpose T is not necessary.

Thanks for sharing the code.

Hi, @WangYueFt ,

I can DCP-v1 without any issues. However, when I run
python main.py --exp_name=dcp_v2 --model=dcp --emb_nn=dgcnn --pointer=transformer --head=svd, I got the following GPU memory error

  File "/root/anaconda3/envs/pytorch1.0/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/code9/dcp/model.py", line 232, in forward
    dropout=self.dropout)
  File "/data/code9/dcp/model.py", line 27, in attention
    scores = torch.matmul(query, key.transpose(-2, -1).contiguous()) / math.sqrt(d_k)
RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 11.91 GiB total capacity; 10.80 GiB already allocated; 5.44 MiB free; 64.97 MiB cached)

How large GPU memory is required to train dcp_v2? According to your paper, GTX 1070 GPU (8GB) is used. But my system uses Titan XP (12GB).

THX!

thank you for your work
i am wondering that what GPU were you using in trainning and testing

i'm using GTX1060 python3.6 PyTorch1.0 Cuda9.0
but i get "RuntimeError: CUDA out of memory." with batch_size=8 while trainning
and get the same error with test_batch_size=4 while testing

for src, target, rotation_ab, translation_ab, rotation_ba, translation_ba, euler_ab, euler_ba in tqdm(test_loader):
For the code above, i get that the src the source point cloud, and the target is the transformed point cloud. The net would calculate a matrix to make those two point cloud can coincide. However, i visuaize the src and the target, and i can see the shape of src while the target is just some scattered points. What i kown the point registation is rigid transformation, is there anything wrong that the target make shape change in the transformation or i don't get the meaning what you want to explain?

Hey,

Thanks for the code. I am trying to visualize the results after testing. Can you please say how to visualize the results of the registration after I have the trained model? Thanks in advance.

Hi @WangYueFt Thanks for your sharing. I have several questions related to the paper:

1. For the performance of ICP reported in your paper, I wonder what is the initial transformation you used for ICP algorithm?
2. It looks like your method does not need an initial transformation as input, so have you ever test your model on larger rotation and translation cases? ( like rotation in [0, 360degree) translation in [-1, 1]?)
3. Will your model handle the cases where there is no 1-to-1 correspondence? I think for the current experiment setting, all the point cloud pairs have 1-to-1 correspondence. How about two point clouds have different number of points?

Best,
Xuyang.

Hi Yue,

Many thanks for the great work. I have a question about the calculation of the MSE. When calculating the MSE, we should have already got the 1-to-1 correspondence between two point cloud sets. But in the following line, the order of point has been permuted.

Bests,
Zheng

Please add a license file to your repo. Thank you!

Hi WangYue:
Thanks your work,I want ask a question:
scores = torch.matmul(src_embedding.transpose(2, 1).contiguous(), tgt_embedding) / math.sqrt(d_k)
scores = torch.softmax(scores, dim=2)
I want to know what the distance metric of features you use,I think it's like the Attention(Q; K; V ) = softmax(QK' /sqrt(pdk) )V in Attention Is All You Need.

I loaded the pre-trained model "dcp_v2.t7" first, and test it with loss 0.000212.
Then I continued training on this basis. But I get a bigger loss about 0.005.
Later I discovered that this was caused by my learning rate setting being too large(0.001). So I set it as 0.00001 which is the value you suggest after 200 epochs in your code.
Strangely, the loss has not continued to decrease.
So i want to ask, Can I train a model as good as the pre-trained model after 250 epoch as your paper says？

I think the implement of pointnet in your codes is different from the original version，your implement is much simpler. Why does this happen?

DCP focuses on finding Rigid Transformations given source and target point cloud. DCP is trained by randomly applying rotation and transformation on the source point cloud to generate a target point cloud, the Network is then used to predict the transformation.
In my scenario, I have two point clouds of the same object taken at different points in time and I have to register these 2 point clouds. As in real-life scenarios, the source and target points are never exactly the same. Even after uniform sampling, we can't guarantee this condition. Can anyone suggest how I can use DCP to solve this issue?

Hi Yue,

Many thanks for the great work. I have a question about the point cloud sampling strategy in your code. If I understand your code correctly, the point cloud sampling is done before creating ground truth, does that mean the point cloud X and point cloud Y have a 1-to-1 correspondence? i.e. if a perfect R and t are computed, X and Y will have an exact alignment?

More specifically, say we have a point cloud before sampling A, from what I understanding, you sample it to get B, and transform B to C using a ground truth transformation T and you train it with B, C, and T.

I was wondering have you ever done any experiment that adding ground truth transformation before sampling and how does the NN perform? That means apply ground truth transformation T over A to have D, sample A to have E, sample D to have F and train the NN with E, F, and T.

Best,
Zirui

Hello,

Thanks for making the code and models public, I was able to reproduce the results of sections 5.1 and 5.3 as they both use the same pretrained model trained on 20 categories.

Could you please upload the pretrained model for section 5.2 - unseen categories as well?

Thanks in advance!

          Yes, just follow the instructions given in the README with the pretrained model. To run inference on the whole dataset - 

python main.py --exp_name=dcp_v2 --model=dcp --emb_nn=dgcnn --pointer=transformer --head=svd --eval --model_path=pretrained/dcp_v2.t7

You have to modify the Dataloaders and inference code to run it on your own point clouds such as STL or PLY

Originally posted by @cy94 in #19 (comment)

Hi, thanks for this fantastic work. In the paper, you said "The methods we describe here extend easily to the M = N case
because DGCNN, Transformer, and Softmax treat inputs as unordered sets. None requires X and Y to have the same length or a bijective matching."

Is there any implementation of the cases when M differs from N? Thanks.