Comments (6)
I think perhaps I have solved it.In my option,he wants to use convolution in single dimension.So he use a list[0, k_x.shape[3] // 2] in an image.the answer is to use 2d conv not 1d conv.And you should solve an inplace operation in network,which caused BP of network failed.
Can you show us how you solved it? Cause I did change it to 2d conv, but I got this issue: 0%| | 0/400 [00:00<?, ?it/s]C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\plugins\pillow.py:320: UserWarning: Loading 16-bit (uint16) PNG as int32 due to limitations in pillow's PNG decoder. This will be fixed in a future version of pillow which will make this warning dissapear. warnings.warn( torch.Size([1, 8, 202, 202]) C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd__init__.py:251: UserWarning: Error detected in MeanBackward1. Traceback of forward call that caused the error: File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 309, in main(config) File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 295, in main trainAndGetBestModel(fusion_model, regis_model, optimizer, dataloaders, baseline_cpsnrs, config) File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 177, in trainAndGetBestModel shifts = register_batch(regis_model, File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 40, in register_batch theta = shiftNet(torch.cat([reference, lrs[:, i : i + 1]], 1)) File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1527, in call_impl return forward_call(*args, **kwargs) File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\DeepNetworks\ShiftNet.py", line 66, in forward x[:, 1] = x[:, 1] - torch.mean(x[:, 1], dim=(1, 2)).view(-1, 1, 1) (Triggered internally at ..\torch\csrc\autograd\python_anomaly_mode.cpp:119.) Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 0%| | 0/131 [00:10<?, ?it/s] 0%| | 0/400 [00:10<?, ?it/s] Traceback (most recent call last): File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 309, in main(config) File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 295, in main trainAndGetBestModel(fusion_model, regis_model, optimizer, dataloaders, baseline_cpsnrs, config) File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 190, in trainAndGetBestModel loss.backward() File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensor.py", line 492, in backward torch.autograd.backward( File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd__init.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8, 128, 128]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
This is because of inplace operation in pytorch.I remembered it appeared in shiftnet.py, you can try to fix it.But I get a bad result,so I gave up.
from highres-net.
This code was picked up in different projects available on GitHub (eg in RobustMFSRforEO) and I still faced the same error.
I modified the code so that the input data corresponds to what is expected for the conv1d
function, hoping not to have introduced an error... Changes are commented with ##
.
def lanczos_shift(img, shift, p=3, a=3):
'''
Shifts an image by convolving it with a Lanczos kernel.
Lanczos interpolation is an approximation to ideal sinc interpolation,
by windowing a sinc kernel with another sinc function extending up to a
few number of its lobes (typically a=3).
Args:
img : tensor (batch_size, channels, height, width), the images to be shifted
shift : tensor (batch_size, 2) of translation parameters (dy, dx)
p : int, padding width prior to convolution (default=3)
a : int, number of lobes in the Lanczos interpolation kernel (default=3)
Returns:
I_s: tensor (batch_size, channels, height, width), shifted images
'''
## These lines are added from RobustMFSRforEO
B, C, H, W = img.shape
# Because examples and channels are interleaved in dim 1.
shift = shift.repeat(C, 1).reshape(B * C, 2)
img = img.view(1, B * C, H, W)
dtype = img.dtype
if len(img.shape) == 2:
img = img[None, None].repeat(1, shift.shape[0], 1, 1) # batch of one image
elif len(img.shape) == 3: # one image per shift
assert img.shape[0] == shift.shape[0]
img = img[None, ]
# Apply padding
padder = torch.nn.ReflectionPad2d(p) # reflect pre-padding
I_padded = padder(img)
I_padded_reshape = I_padded.view(I_padded.shape[0],I_padded.shape[1],-1) ## image reshaped to work with conv1d
# Create 1D shifting kernels
y_shift = shift[:, [0]]
x_shift = shift[:, [1]]
k_y = (lanczos_kernel(y_shift, a=a, N=None, dtype=dtype)
.flip(1) # flip axis of convolution
)[:, None, :, None].squeeze(3) ## expand dims to get shape (batch, channels, y_kernel) instead of (batch, channels, y_kernel, 1)
k_x = (lanczos_kernel(x_shift, a=a, N=None, dtype=dtype)
.flip(1)
)[:, None, None, :].squeeze(2) ## shape (batch, channels, x_kernel) instead of (batch, channels, 1, x_kernel)
# Apply kernels
I_s = torch.conv1d(I_padded_reshape,
groups=k_y.shape[0],
weight=k_y,
padding=k_y.shape[2] // 2) ## previously : [k_y.shape[2] // 2, 0]
I_s = torch.conv1d(I_s,
groups=k_x.shape[0],
weight=k_x,
padding=k_x.shape[2] // 2) ## previously : [0, k_x.shape[3] // 2]
I_s = I_s.view(B, C, H+2*p, W+2*p) ## result reshaped in image format
I_s = I_s[..., p:-p, p:-p] # remove padding
return I_s
Plus : in DeepNetworks/ShiftNet.py I removed the .transpose(0, 1)
whose format does not correspond to the documentation
from highres-net.
Hi, have you tested your solution since then? I encountered the same error, I'm currently running the training process with your changes, but there's nothing to compare the results with, since there's no pretrained model, sadly
from highres-net.
Indeed, this repository does not contain pre-trained models. After a quick review, I see that this competing architecture also uses ShiftNet, with the same code for lanzcos_shift, this time with pre-trained models that can be used for comparison: https://github.com/rarefin/MISR-GRU
I had tested my proposal, but the results were worse using the ShiftNet than without... And below the values reported in the publication... I had given up hoping for some interaction on this forum to help me move forward!
On rereading my proposal, it seems obvious that the result of the shift cannot be applied twice, for x and y, to the same flattened matrix.
I therefore complete my previous proposal with a slight modification between the two 1d convolutions. The image is reconstructed after the x-correction to be flattened again, this time to accommodate the y-correction.
... Unfortunately the results are not improved, some confusion must be introduced somewhere!
def lanczos_shift(img, shift, p=3, a=3):
'''
Shifts an image by convolving it with a Lanczos kernel.
Lanczos interpolation is an approximation to ideal sinc interpolation,
by windowing a sinc kernel with another sinc function extending up to a
few number of its lobes (typically a=3).
Args:
img : tensor (batch_size, channels, height, width), the images to be shifted
shift : tensor (batch_size, 2) of translation parameters (dy, dx)
p : int, padding width prior to convolution (default=3)
a : int, number of lobes in the Lanczos interpolation kernel (default=3)
Returns:
I_s: tensor (batch_size, channels, height, width), shifted images
'''
B, C, H, W = img.shape
## Because examples and channels are interleaved in dim 1.
shift = shift.repeat(C, 1).reshape(B * C, 2)
img = img.view(1, B * C, H, W)
dtype = img.dtype
if len(img.shape) == 2:
img = img[None, None].repeat(1, shift.shape[0], 1, 1) # batch of one image
elif len(img.shape) == 3: # one image per shift
assert img.shape[0] == shift.shape[0]
img = img[None, ]
# Apply padding
padder = torch.nn.ReflectionPad2d(p) # reflect pre-padding
I_padded = padder(img)
I_padded_reshapeX = I_padded.view(I_padded.shape[0],I_padded.shape[1],-1) ## The images are flattened
# Create 1D shifting kernels
y_shift = shift[:, [0]]
x_shift = shift[:, [1]]
k_y = (lanczos_kernel(y_shift, a=a, N=None, dtype=dtype)
.flip(1) # flip axis of convolution
)[:, None, :, None].squeeze(3) # expand dims to get shape (batch, channels, y_kernel, 1)
k_x = (lanczos_kernel(x_shift, a=a, N=None, dtype=dtype)
.flip(1)
)[:, None, None, :].squeeze(2) # shape (batch, channels, 1, x_kernel)
# Apply kernels
I_s_reshapeX = torch.conv1d(I_padded_reshapeX, #.permute(1, 0, 2, 3)
groups=k_y.shape[0],
weight=k_y,
padding=k_y.shape[2] // 2) # same padding # [k_y.shape[2] // 2, 0]
I_s = I_s_reshapeX.view(1, B*C, H + 2 * p, W + 2 * p) ## Reconstruction of the padded image
I_s_reshapeY = I_s_reshapeY = I_s.transpose(2,3).reshape(I_s.shape[0],I_s.shape[1],-1) ## The images are flattened after inversion of the width and heigth
I_s_reshapeY = torch.conv1d(I_s_reshapeY,
groups=k_x.shape[0],
weight=k_x,
padding=k_x.shape[2] // 2) #[0, k_x.shape[3] // 2]
I_s = I_s_reshapeY.reshape(B, C, W+2*p, H+2*p).transpose(2,3) ## Reconstruction of the image and re-inversion of width and height
I_s = I_s[..., p:-p, p:-p] # remove padding
return I_s # , k.squeeze()
from highres-net.
I think perhaps I have solved it.In my option,he wants to use convolution in single dimension.So he use a list[0, k_x.shape[3] // 2] in an image.the answer is to use 2d conv not 1d conv.And you should solve an inplace operation in network,which caused BP of network failed.
from highres-net.
I think perhaps I have solved it.In my option,he wants to use convolution in single dimension.So he use a list[0, k_x.shape[3] // 2] in an image.the answer is to use 2d conv not 1d conv.And you should solve an inplace operation in network,which caused BP of network failed.
Can you show us how you solved it? Cause I did change it to 2d conv, but I got this issue:
0%| | 0/400 [00:00<?, ?it/s]C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\imageio\plugins\pillow.py:320: UserWarning: Loading 16-bit (uint16) PNG as int32 due to limitations in pillow's PNG decoder. This will be fixed in a future version of pillow which will make this warning dissapear. warnings.warn(
torch.Size([1, 8, 202, 202])
C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd_init_.py:251: UserWarning: Error detected in MeanBackward1. Traceback of forward call that caused the error:
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 309, in
main(config)
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 295, in main
trainAndGetBestModel(fusion_model, regis_model, optimizer, dataloaders, baseline_cpsnrs, config)
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 177, in trainAndGetBestModel
shifts = register_batch(regis_model,
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 40, in register_batch
theta = shiftNet(torch.cat([reference, lrs[:, i : i + 1]], 1))
File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\DeepNetworks\ShiftNet.py",
line 66, in forward
x[:, 1] = x[:, 1] - torch.mean(x[:, 1], dim=(1, 2)).view(-1, 1, 1)
(Triggered internally at ..\torch\csrc\autograd\python_anomaly_mode.cpp:119.)
Variable.execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
0%| | 0/131 [00:10<?, ?it/s]
0%| | 0/400 [00:10<?, ?it/s]
Traceback (most recent call last):
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 309, in
main(config)
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 295, in main
trainAndGetBestModel(fusion_model, regis_model, optimizer, dataloaders, baseline_cpsnrs, config)
File "C:\Users\Юнсок\Desktop\Research\MISR\HighRes-net-master\src\train.py", line 190, in trainAndGetBestModel
loss.backward()
File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch_tensor.py", line 492, in backward
torch.autograd.backward(
File "C:\Users\Юнсок\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\autograd_init.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8, 128, 128]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
from highres-net.
Related Issues (10)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from highres-net.