visinf / n3net Goto Github PK

>>> import matmul_cuda
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
>>>

Environment I have tried:
python=3.6
pytorch=1.10.2
cudatoolkit=9.2 or 10.0 or 11.3

I followed the readme to complete the environment configuration and encountered no errors; Then I run the test statement and ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

I can't install pyinn, could you please tell me what is wrong？：

(n3net) C:\Users\lenovo>pip install pyinn
ERROR: Could not find a version that satisfies the requirement pyinn (from versions: none)
ERROR: No matching distribution found for pyinn

Hi,

I'm getting out of memory at test time (16GB on my graphics card). This is true even when I reduce patch size to only 20x20 (x3 channel). However, I was able to train the network at 80x80 and batch size of 32 using only 4.5GB memory.

Do you have any idea what might cause this? I'll look at your code closely in the meantime, but any pointer is greatly appreciated.

I don't have TC installed as I am not using conda. Does TC help you save memory?

Thank you.

Thank you for your excellent work! Your idea is innovative to me, and your code is well-structured. However, on reviewing your code, I have some suggestions that potentially improve your performance.

Your implementation of ssim and psnr are from open-source codes on GitHub. However, I highly recommend you consider choosing the standard implementation in the package skimage. In my research project, I once use the same code as you are now. But after I switch to skimage for evaluation, the performance experiences an improvement. Although I do not know the reason for this, I suggest you try using metrics in skimage, which may reveal a better performance.

The current euclidean distance calculation produces non-symmetric distance matrices. I suspect this is a numeric issue (although it persists even when using float64):

    out = -2 * torch.matmul(x, y)
    out += (x ** 2).sum(dim=-1, keepdim=True)
    out += (y ** 2).sum(dim=-2, keepdim=True)
    a = out.t() - out

tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.1054e-15,
         -7.1054e-15,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  7.1054e-15,
          0.0000e+00,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ..., -7.1054e-15,
          0.0000e+00, -7.1054e-15],
        ...,
        [ 7.1054e-15, -7.1054e-15,  7.1054e-15,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 7.1054e-15,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  7.1054e-15,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00]], dtype=torch.float64)

Apparently sklearn.metric.pairwise_distance also has this issue...

BSDS, DIV2K and Waterloo datasets are used in the paper for real image denoising. I want to know how many patches were extracted for training. Did you use the same principle as in Gaussian denoising (512 80x80 patches from each image)?

What about the number of epochs and lr decay? I couldn't find those on the paper, they are given only for Gaussian denoising.

You said the input "I" is "Indexing tensor defining O potential neighbors for each query item". How can i construct this input? My xe is a tensor of 4204812 and ye is a tensor of 4204812, so the "I" is a tensor of 41212, am i right? What are the elements of "I"?

I find training is unstable when using n3net in correspondences experiments, the training loss increases suddenly and the valid accuracy drops simultaneously. It falls into bad local minima.

So, has anyone encountered this problem? I use the default config for training.

I'm trying to evaluate NNN against conventional KNN on a simple test case.
The test case is to find the 5 nearest neighbour for a permutation of indices (for easy intuitive verification).
The problem is that the aggregate output is outputting the same value for all 5 neighbours.

Problem setup:

import torch
import non_local
import numpy as np

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

N = 50
nn = non_local.N3AggregationBase(5, temp_opt={"external_temp": False})

x = torch.tensor(np.random.permutation(list(range(N))), dtype=torch.float, requires_grad=True)
x = x.reshape(1, N, 1).to(device)
xe = x
ye = xe
I = torch.tensor(list(range(N)), dtype=torch.long).repeat(N, 1).reshape(1, N, N).to(device)

z = nn(x, xe, ye, I)

Where the aggregate output z is

tensor([[[[10.0001, 10.0001, 10.0001, 10.0001, 10.0001]],
         [[42.0001, 42.0001, 42.0001, 42.0001, 42.0001]],
         [[22.0001, 22.0001, 22.0001, 22.0001, 22.0001]],
...

Is this supposed to be the case and I'm interpreting the result wrong? If so then what is the aggregate output z supposed to represent?

Evaluation mode
Checkpoints from ../results_gaussian_denoising/pretrained_sigma70
Writing to ../results_gaussian_denoising/pretrained_sigma70/test04/_data.pkl
Epoch 51
Testing on Set12
Traceback (most recent call last):
File "main.py", line 281, in
main()
File "main.py", line 257, in main
run(my_experiment)
File "main.py", line 239, in run
evaluate(experiment)
File "main.py", line 160, in evaluate
test_epoch(epoch, experiment)
File "main.py", line 119, in test_epoch
pred = net(inputs)
File "/home/tuqiang/anaconda3/envs/pt_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/media/tuqiang/16af747f-50d9-46bf-82c1-788861185451/workspace/denoise_new/deeplearning_denoise/n3net/src_denoising/models/n3net.py", line 231, in forward
x = self.nlsi
File "/home/tuqiang/anaconda3/envs/pt_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/media/tuqiang/16af747f-50d9-46bf-82c1-788861185451/workspace/denoise_new/deeplearning_denoise/n3net/src_denoising/models/n3net.py", line 184, in forward
x = self.n3aggregation(xg,xe,ye,log_temp=log_temp)
File "/home/tuqiang/anaconda3/envs/pt_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/media/tuqiang/16af747f-50d9-46bf-82c1-788861185451/workspace/denoise_new/deeplearning_denoise/n3net/src_denoising/models/non_local.py", line 274, in forward
x_patch, padding = ops.im2patch(x, self.patchsize, self.stride, None, returnpadding=True)
File "/media/tuqiang/16af747f-50d9-46bf-82c1-788861185451/workspace/denoise_new/deeplearning_denoise/n3net/src_denoising/ops.py", line 121, in im2patch
x2col = pyinn.im2col(xpad, [patchsize]*2, [stride]*2, [0,0])
NameError: name 'pyinn' is not defined

Hi,
I read your paper and really appreciate the novelty of the differentiable KNN selection network.

In this git repo,

1. The bash for the datasets downloads for BSDS500 is not working and no val set for DIV2K.
2. Could you please provide some example commands for your experiments so that readers can run experiments in your paper more conveniently?

Thx and looking forward to your response.

It is reported in your paper that

To this end, we evaluate on the recent Darmstadt Noise Dataset [34], consisting of 50 noisy images shot with four different cameras at varying ISO levels. Realistic noise can be well explained by a Poisson-Gaussian distribution which, in turn, can be well approximated by a Gaussian distribution where the variance depends on the image intensity via a linear noise level function [12].

So the results_poissongaussian_denoising/pretrained is the model you used on DND dataset?

In your paper, you have reported the data of denoised result of dnd srgb images. But I can only find mode "raw" and "raw_full" in function evalate_dnd of file main_pg.py. Can the same model from results_poissongaussian_denoising/pretrained be used to denoise srgb images? And b.t.w what is the difference of "raw" and "raw_full"? Sorry for the cumbersome questions.

Hi I found the main.py can only evaluate on the datasets specified, do you plan to provide an API to directly denoise one image to get visual results?

Firstly, I run the command
pip install git+https://github.com/szagoruyko/pyinn.git@master
After running that in my Mac OS Mojave system, I got th error like this:
Modules: cuda : No -> Include files not found: ['cublas_v2.h', 'cuda.h', 'cuda_profiler_api.h', 'cuda_runtime.h', 'cufft.h', 'curand.h', 'cusparse.h', 'nvrtc.h'] -> Check your CFLAGS environment variable.
what should i do to fix this?

Thanks for your impressive work! But I have a problem about testing on dnd. In "main_pg.py" the input has 3 channels and the output has 1 channels, I want to know how to get the RGB results or what's the meaning of output. Looking forward to your reply.

This code

N = 7
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
nn = N3AggregationBase(5, temp_opt={"external_temp": False})
nn.cuda()
nn.nnn.log_temp_bias = -50 # decrease temperature -> NNN acts more like hard kNN

# x = torch.tensor(np.random.permutation(list(range(N))), dtype=torch.float, requires_grad=True)
x = torch.tensor(list(range(N)), dtype=torch.float, requires_grad=True)
n = torch.zeros_like(x).normal_() * 0.0001
x = x+n
x = x.reshape(1, N, 1).to(device)
xe = x
ye = xe
I = torch.tensor(list(range(N)), dtype=torch.long).repeat(N, 1).reshape(1, N, N).to(device)

z = nn(x, xe, ye, I)

produces output z with shape torch.Size([1, 7, 1, 5]).
If input embeddings have more than 1 feature, third dimension of output would be changed.

How to get global indexes of nearest neighbors?
For example, if we have XE with shape [1, 10, 5], and YE with shape [1, 3, 5], i want to have output indexes with shape [1, 3], just as indexes of nearest neighbors in KNeighborsClassifier.

when I run this code "python setup.py install " in lib file. it successfully install matmul_cuda, but when I run 'import matmul_cuda' in terminal, it raise an error, like this:
"Traceback (most recent call last):
File "", line 1, in
ImportError: /home/tong2018234172/anaconda3/lib/python3.6/site-packages/matmul-0.0.0-py3.6-linux-x86_64.egg/matmul_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E
"
and my python version is 3.6, torch version is 0.4.1, cuda version is 8.0.61

Please add a link in the readme to https://arxiv.org/abs/1810.12575