Hello,

This SSIM implementation is a great addiction to the differentiable functions of PyTorch. However it is lacking the Multi-Scale Structural Similarity, a more commonly used metric in image assessment. I think it would be beneficial and relatively simple to implement and add this metric to this project.

I wrote a basic working MS-SSIM implementation based on SSIM from this project and using the Tensorflow implementation as reference. Would you consider adding the MS-SSIM metric if a pull request was sent?

Thank you,

import pytorch_ssim
import torch
from torch.autograd import Variable
from torch import optim
import cv2
import numpy as np

npImg1 = cv2.imread("einstein.png")

img1 = torch.from_numpy(np.rollaxis(npImg1, 2)).float().unsqueeze(0)/255.0
img2 = torch.rand(img1.size())

if torch.cuda.is_available():
img1 = img1.cuda()
img2 = img2.cuda()

img1 = Variable( img1, requires_grad=False)
img2 = Variable( img2, requires_grad = True)

Functional: pytorch_ssim.ssim(img1, img2, window_size = 11, size_average = True)

print(img1.shape,img2.shape)
ssim_value = pytorch_ssim.ssim(img1, img2).item()
print("Initial ssim:", ssim_value)

Module: pytorch_ssim.SSIM(window_size = 11, size_average = True)

ssim_loss = pytorch_ssim.SSIM()

optimizer = optim.Adam([img2], lr=0.01)

while ssim_value < 0.95:
optimizer.zero_grad()
ssim_out = -ssim_loss(img1, img2)
ssim_value = - ssim_out.item()
print(ssim_value)
ssim_out.backward()
optimizer.step()
ii = np.transpose(img2[0].detach().cpu().numpy(), (1,2,0))
cv2.imshow("pre", ii255)
print(np.max(ii255))
cv2.waitKey(100)

How to deal with the problem of nan loss during training?

I tried using your code. However, it returns RuntimeError of window being non contiguous. More explicitly

RuntimeError: cuDNN requires contiguous weight tensor

I put a check of window.is_contiguous() which returns 'False'. Therefore, I tried making it contiguous by window.contiguous() but then I get a different error stating

RuntimeError: tensors are on different GPUs

However, I have only one GPU.

while ssim_value < 0.95:
optimizer.zero_grad()
ssim_out = -ssim_loss(img1, img2)
ssim_value = - ssim_out.item()
print(ssim_value)
ssim_out.backward()
optimizer.step()
ii = np.transpose(img2[0].detach().cpu().numpy(), (1,2,0))
cv2.imshow("pre", ii255)
print(np.max(ii255))
cv2.waitKey(100)

Hi. The MS-SSIM loss is really helpful and I wonder will you consider adding cuda support?

Traceback (most recent call last):
  File "run_model.py", line 143, in <module>
    loss_ssim = -ssim_loss(noriginalimage, outputimage)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pytorch_ssim/__init__.py", line 57, in forward
    return _ssim(img1, img2, window, self.window_size, channel, self.size_average)
  File "/usr/local/lib/python3.6/site-packages/pytorch_ssim/__init__.py", line 18, in _ssim
    mu1 = F.conv2d(img1, window, padding = window_size//2, groups = channel)
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

It is throwing this error. I have to convert both the variables to CPU to run which kills the speed of the computation. Can someone help out in resolving this?

I use SSIM as a loss, but it seems always = 1. I try to print ssim_map. It's all one. I make sure my two img is not the same. Their L1 distance isn't zero. I don't know where the error is. Could you please help me?

Quick question on the output range. Does the SSIM implementation output number of range [0;1] or [-1;1] instead as described in the original SSIM paper http://www.cns.nyu.edu/pub/lcv/wang03-reprint.pdf ?

why there is no parameter PI in gaussian function?

Tried to run sample code but the error occurred at this line:
--> ssim_value = SSIM(img1, img2).data[0]

Any idea what happened? Thanks.

Hello,

I am using your SSIM implementation as a part of my total objective for performing denoising. Unfortunately, after a few iterations, I started getting NaN in the objective (this does not happen if I remove the SSIM from the loss).

I am wondering if there is any occasion where your implementation may divide by zero, or something that may cause NaN. By just looking at the code, I don't know why this could happen (since C1 and C2 are used to avoid zero-divisions).

Any thoughs?

It would be great if there was a flag to make the results identical to the output of compare_ssim from scikit image.
Their implementation uses no padding, but there's still a slight difference in my results, that I haven't found.

If you want to use this code to calculate loss, I recommend you to change the L (maximum brightness) from 255 to 1.

From the original paper, C1=(KL)², but I didn't see L=255 just K1 = 0.01 in the specific implementation of the code. Can the item L be discarded?Does the code mean K1=0.01?

Hi,

I am not able to run the example script. Could you please let me know what is missing?

basic usage script :

import pytorch_ssim
import torch
from torch.autograd import Variable

img1 = Variable(torch.rand(1, 1, 256, 256))
img2 = Variable(torch.rand(1, 1, 256, 256))

if torch.cuda.is_available():
img1 = img1.cuda()
img2 = img2.cuda()

print(pytorch_ssim.ssim(img1, img2))

ssim_loss = pytorch_ssim.SSIM(window_size=4)

print(ssim_loss(img1, img2))

here is the traceback of the code:

Traceback (most recent call last):
File "/home/thesis_2/ssim_test.py", line 12, in
print(pytorch_ssim.ssim(img1, img2))
File "/opt/conda/envs/venv/lib/python3.6/site-packages/pytorch_ssim/init.py", line 62, in ssim
return _ssim(img1, img2, window, window_size, channel, size_average)
File "/opt/conda/envs/venv/lib/python3.6/site-packages/pytorch_ssim/init.py", line 18, in _ssim
mu1 = F.conv2d(img1, window, padding = window_size/2, groups = channel)
TypeError: conv2d(): argument 'padding' must be tuple of ints, not float

Hi. It supports n, c, h, w-dimensional images？Thanks

whty do we need set the requires_grad = True in the img2 = Variable( img2, requires_grad = True) ,while the value in the img1 is False

import pytorch_ssim
import torch
from torch.autograd import Variable
from torch import optim
import cv2
import numpy as np

npImg1 = cv2.imread("einstein.png")

img1 = torch.from_numpy(np.rollaxis(npImg1, 2)).float().unsqueeze(0)/255.0
img2 = torch.rand(img1.size())

if torch.cuda.is_available():
    img1 = img1.cuda()
    img2 = img2.cuda()


img1 = Variable( img1,  requires_grad=False)
img2 = Variable( img2, requires_grad = True)


# Functional: pytorch_ssim.ssim(img1, img2, window_size = 11, size_average = True)
ssim_value = pytorch_ssim.ssim(img1, img2).data[0]
print("Initial ssim:", ssim_value)

# Module: pytorch_ssim.SSIM(window_size = 11, size_average = True)
ssim_loss = pytorch_ssim.SSIM()

optimizer = optim.Adam([img2], lr=0.01)

while ssim_value < 0.95:
    optimizer.zero_grad()
    ssim_out = -ssim_loss(img1, img2)
    ssim_value = - ssim_out.data[0]
    print(ssim_value)
    ssim_out.backward()
    optimizer.step()

I have a question about how to compute the variance.

So, following wiki, I wanted to define luminance ( l ), contrast ( c ) and structure ( s), for that I need the standard deviation: square root of the variance (sigma1_ below). But I also tested the formula $E((X-E(X))^2)$ and it gives me two different results.

Below is the code for $sigma1_= (sigma1_sq)^.5=E(X^2)-E(X)^2$ and $sigma1=E((X-E(X))^2)$ that can be added to here.

    mu1 = F.conv2d(img1, window, padding = window_size//2, groups = channel)
    mu2 = F.conv2d(img2, window, padding = window_size//2, groups = channel)
    mu1_sq = mu1.pow(2)
    mu2_sq = mu2.pow(2)
    mu1_mu2 = mu1*mu2
    sigma1_sq = F.conv2d(img1*img1, window, padding = window_size//2, groups = channel) - mu1_sq
    sigma2_sq = F.conv2d(img2*img2, window, padding = window_size//2, groups = channel) - mu2_sq
    sigma12 = F.conv2d(img1*img2, window, padding = window_size//2, groups = channel) - mu1_mu2
    C1 = 0.01**2
    C2 = 0.03**2
    sigma1 = F.conv2d((img1-mu1).pow(2), window, padding = window_size//2, groups = channel).pow(.5)
    sigma2 = F.conv2d((img2-mu2).pow(2), window, padding = window_size//2, groups = channel).pow(.5)
    sigma1_= sigma1_sq.pow(.5)
    sigma2_= sigma2_sq.pow(.5)
    assert  torch.all(sigma2.eq(sigma2_)) #this returns false
    assert  torch.all(sigma1.eq(sigma1_))

Is there a problem with my code?
Is there any reason to choose one of the two formulas? I assume it has to do with numerical stability.

My model consumes data in the range of 0 to1 instead of 0 to 255. Can the same function be used in that case or should I convert it ?

When computing the ssim of the output of a CNN (batch size 8, image resolution is 192*192), the consumed GPU memory are doubled. It seems that the ssim should require little GPU memory. I am curious about this problem. Thx~

Most implementation such as this http://www.cns.nyu.edu/~lcv/ssim/ssim_index.m also have L, but I understand L scales with bit sizes so that probably won't do much difference for float images?

Hello, what's wrong with code that comes out with negative values？

0.1 has window_size / 2, but python 3 requires window_size // 2, you have already modified the code. Can you please release a new version?

When I try to run your code I get the following error:

Traceback (most recent call last):
File "pytorch_NN4.py", line 144, in
loss = loss_fn(y_pred, y)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 224, in __call__
result = self.forward(*input, **kwargs)
File "/media/brenco/Shared Partition/Neural Network/pytorch_ssim.py", line 57, in forward
return _ssim(img1, img2, window, self.window_size, channel, self.size_average)
File "/media/brenco/Shared Partition/Neural Network/pytorch_ssim.py", line 18, in _ssim
mu1 = F.conv2d(img1, window, padding = window_size/2, groups = channel)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py", line 51, in conv2d
_pair(0), groups, torch.backends.cudnn.benchmark, torch.backends.cudnn.enabled)
RuntimeError: argument 1 (padding) must be tuple of int but got tuple of (float, float)

I downloaded the python-ssim.py file and just made a call to that instead of installing it, but I don't think that should be a problem. Any suggestions?

when executing (_, channel, _, _) = img1.size()

It would be good for both the users as well as for @Po-Hsun-Su , the owner of this package

Or even N-dimensional SSIM!

Hi
This code contains the same error as skimage, and pytorch-ssim you can read full description here: scikit-image/scikit-image#5192

Shortly, when used for estimation of perceptual quality, authors of original paper proposed to downsample images first to make SSIM focus on major differences between reference and distorted inputs.

So what?
If you are using this implementation as a loss function for CNN, you're likely leading it in the wrong direction.

Alternatives
You can find correct implementation of SSIM, MS-SSIM and some other metrics here:
https://github.com/photosynthesis-team/piq

Hello, I want to use SSIM as a loss function in my own project. How can I reference your code?

*** RuntimeError: tensors are on different GPUs error when input cuda input is not on gpu 0

Why do you not pull a request to pytorch to have it among the standard cost functions?