photosynthesis-team / piq Goto Github PK

Is your feature request related to a problem? Please describe.
Now these two measure have different high-level interfaces and work only with passed extracted features. Make it better.

Describe the solution you'd like
Create interfaces that allows to specify feature extractor, accept different types of collections of images etc.

Describe alternatives you've considered
None

Additional context
None

Is your feature request related to a problem? Please describe.
This metric showed to be comparable in quality of predictions with FID and KID scores.

Describe the solution you'd like
Take code from this repo https://github.com/xgfs/msid/tree/master/msid and potentially refactor it to make all FID, KID, MSID metrics interface look a like.

Hi, thank you for your awesome work. I do like the idea of combining many metrics in one place.

In the future, it will be great to see implementations of some image quality metrics from https://github.com/dsoellinger/blind_image_quality_toolbox

So, I hope they can be ported to PyTorch and I will not have to run Matlab :)

In SSIM documentation it is stated that one can specify the max_val argument.
However the current interface of the forward() function does not support it - it seems that max_val has been moved to initialization. The documentation (see link below) should reflect the current state.

https://github.com/photosynthesis-team/photosynthesis.metrics/blob/e17c4c1ddc19f282b12b232e4332f27b3291ea49/photosynthesis_metrics/ssim.py#L134

The same is true for MSSIM:
https://github.com/photosynthesis-team/photosynthesis.metrics/blob/e17c4c1ddc19f282b12b232e4332f27b3291ea49/photosynthesis_metrics/ssim.py#L317

Is your feature request related to a problem? Please describe.
Set up Continuous Integration with linting and running tests.

Describe the solution you'd like
Provide this functionality via Github Actions.

Describe alternatives you've considered
Travis CI or Drone CI if Github Actions will not work for some reason.

Additional context

Is your feature request related to a problem? Please describe.
It's been used in a couple of Style Transfer articles.
https://www.wikiwand.com/en/Total_variation_denoising

Describe the solution you'd like
Implement anisotropic version of total variation.

Describe the bug
When use PyTorch 1.4.0, it fails to import in the CI testing pipeline. Example pipeline is here. The error started to occur in the day of PyTorch 1.4.0. release.

To Reproduce
Steps to reproduce the behavior:

1. Go to requirements.txt
2. Change required PyTorch version to 1.4.0
3. Run $ pytest from the project folder
4. See error

Expected behavior
No error expected to occur.

Screenshots

SSIM and MSSIM currently compute the similarity index which is between 0 and 1, if arguments are given within correct ranges.

However, usually people minimize losses in Pytorch. So, unless they use the negative SSIM and MSSIM, they will end up with images that are dissimilar to their targets. I would suggest writing this word of advice somewhere in SSIM/MSSIM documentation or examples, just in case, for beginners, etc., e.g.:

If you want to train a neural network to produce images that are similar to your target, you should use SSIM or MSSIM losses with a negative sign.

Is your feature request related to a problem? Please describe.
All files with metrics are named as their abbreviation except inception_score.py

Also rename Minimal examples in README.md

Describe the solution you'd like
Rename to isc.py or ins.py.

Additional context
It is not possible to name file is.py because then it will be impossible to import from it because is is a Python keyword.

Is your feature request related to a problem? Please describe.
Implement metric described in the paper.

Describe the solution you'd like
Fix existing implementation or add a custom one.

Is your feature request related to a problem? Please describe.
First implementation of SSIM and MS-SSIM losses will be basic, treating each element of the batch equally. However, a user may want to weight items within batch as it can be done in BCELoss and some other losses in PyTorch.

Describe the solution you'd like
First, implement _WeightedLoss interface. Keep it protected since it is not supposed to be publicly accessed and only supposed to be used for the inheritance reasons. Take _WeightedLoss implementation in PyTorch as a reference

Describe alternatives you've considered

Additional context

Is your feature request related to a problem? Please describe.
MS-SSIM implementation need to be tested in order to show its correctness and provide a better environment for the refactoring.

Is your feature request related to a problem? Please describe.
Currently is low resolution image/set of images is passed to the MultiScaleSSIMLoss it crashes with the default PyTorch stack trace, which can be hard to interpret

Describe the solution you'd like
Find out the minimal allowed resolution and introduce an assertion/check of image resolution before it is passed to the MS-SSIM computation code.

Is your feature request related to a problem? Please describe.
Now PRs are unstructured, which makes ideas behind them less clear and harder to understand.

Describe the solution you'd like
Adjust one of the common templates to meet the needs of our project.

Is your feature request related to a problem? Please describe.
Need to add two types of perceptual scores: content score and style score. Both scores might be needed both and measures and as losses.

Describe the solution you'd like
Add both measures based on implementation by @zakajd. Improvements to be made:

• Readable layer names. They are needed for better understanding of what layers are used.
• Normalization of values in layers if several layers are used.

Is your feature request related to a problem? Please describe.
It is convenient to have our lib to be available both in pip and conda.

Describe the solution you'd like
Add new CD file, provide required configurations.

Is your feature request related to a problem? Please describe.
Inception score is a popular measure in the GAN world, hence need to be added.

Describe the solution you'd like
Implement from scratch or adapt external implementation. Support BaseFeatureMetric interface.

Background
Current implementations of VIF and SSIM/MS-SSIM losses share the same functionality - 2D convolution with Gaussian kernel.

Idea
Optimise the code by moving the mentioned methods to utils.py.

Outcomes

• No duplicates of code
• Better readability

What is the purpose of defining the TV loss as the absolute difference between the Total Variations of predicted tensor and the target one? Usually, TV used as a regularizer applied only to the predicted tensor.

I can make a pull request with the correction.

Is your feature request related to a problem? Please describe.
Now we have total_variation metric and TVLoss, which are exactly the same things.

Describe the solution you'd like
One of entities needs to removed. Another, probably, renamed.

Is your feature request related to a problem? Please describe.
Currently some implementations use libraries for computations on CPU like scipy.

Describe the solution you'd like
Refactor the code in such a way that it will work without them.

Is your feature request related to a problem? Please describe.
Need to add image-to-image metric used in this challenge.

Describe the solution you'd like
Fit existing implementation or create a custom one

Is your feature request related to a problem? Please describe.
Currently SSIM/MS-SSIM losses try to work with 2, 3, 4-dimensional tensors, but corresponding measures support only 4D. Unify that.

Describe the solution you'd like
Provide interface for the unification.

Describe alternatives you've considered
None

Additional context
None

Update README.md with the new interfaces, metrics and losses.

Is your feature request related to a problem? Please describe.
The library need to be published to pypi package manager in order to become easier to be installed.

Describe the solution you'd like
Decide on internal package name. Now it is called metrics but it might be changed to something less common in order to avoid collisions.

Describe alternatives you've considered
None

Additional context
None

Problem
Current implementation (after #58) uses not optimised validation of input tensors and their dimension adjustments as different number of inputs can be provided.

Solution
Unify the interface for input validations:

• Introduce validation method for a single input;
• Change validation of method of 2 inputs using methods mentioned above;
• Do the same for dimension adjustments.

Describe the bug
Reductions at TVLoss return the same value, when default parameters are used to initialise TVLoss()

To Reproduce

from photosynthesis_metrics import TVLoss
import torch
torch.manual_seed(42)
a = torch.rand(3,3,256,256)
print(f'mean reduction: {TVLoss(reduction='mean')(a)}')
print(f'sum reduction: {TVLoss(reduction='sum')(a)}')

Output

>>> mean reduction: 255.3401
>>> sum reduction: 255.3401

Expected behavior
Output

>>> mean reduction: 255.3401
>>> sum reduction: 766.0204

Solution

• Get rid of size_average parameter at TVLoss;
• Call total_variation with size_average=False at TVLoss().forward()

Is your feature request related to a problem? Please describe.
TV metric and loss were meant to be unified (#76). It meant that function will be removed so it was not updated to work with 2/3/4D tensors.
However, it was decided to not to do that and now TV needs an update.

Describe the solution you'd like

• Move validate_input and adjust_dimensions to the total_variation function
• Call total_variation with all required checks from TVLoss

Is your feature request related to a problem? Please describe.
Official KID implementation is in numpy which may make it slower then it could be. We also would like to avoid additional dependency.

Describe the solution you'd like
Replace numpy and sklearn functions with PyTorch and perform other required changes.

Describe alternatives you've considered
None

Additional context
None

Describe the bug
If PR is introduced from a fork, the results of workflows are not visible in the PR.

To Reproduce
Steps to reproduce the behavior:

1. Fork the repo
2. Introduce your changes
3. Make PR
4. See that no workflows were run even though they were

Expected behavior
Workflows are running the same way as they do in case of PRs from branches in the repo

Is your feature request related to a problem? Please describe.
Currently, not feature-based methods perform two steps before actual computation of metrics: adjustment of dimensions (which is meant to make processing of tensors with different number of dimensions possible) and validation of input (which is meant to validate correctness of user's input). However, their existing implementations contradict to each other: the validation function validates that user's input is 4-dimensional while adjust_dimention changes user's input to be 4-dimensional. This behaviour is redundant and introduces extra confusion.

Describe the solution you'd like

• Remove check that user's input is 4-dimensional
• Make sure that adjust_dimentions functionality works as expected (introduce extra tests)
• Implement all above for all not feature-based metrics

Additional context
It is important to keep in mind #6. We would like to work not only with 4-dimensional tensors because in some applications like medical imaging 5D tensors are used. It is also sometimes convenient for a user to pass 2D or 3D tensors and not perform adjustment themselves.

Describe the bug
multi_scale_ssim and MultiScaleLoss appear to estimate wrong values compared to results obtained from official TensorFlow implementation.

To Reproduce

from photosynthesis_metrics import multi_scale_ssim, MultiScaleSSIMLoss
import torch
import tensorflow as tf
from time import time
torch.manual_seed(42)
for _ in range(2):
    a = torch.rand(3, 3, 256, 256)
    b = torch.rand(3, 3, 256, 256)
    start = time()
    ps_ms_ssim = multi_scale_ssim(a,b, kernel_size=11, kernel_sigma=1.5, size_average=False, data_range=1.).numpy()
    ps_time = time()-start
    start = time()
    ps_ms_ssim_loss = MultiScaleSSIMLoss( kernel_size=11, kernel_sigma=1.5, data_range=1.)(a, b).numpy()
    ps_loss_time = time() - start
    tf_a = tf.convert_to_tensor(a.permute(0, 2, 3, 1).numpy())
    tf_b = tf.convert_to_tensor(b.permute(0, 2, 3, 1).numpy())
    start = time()
    tf_ms_ssim = tf.image.ssim_multiscale(tf_a, tf_b, max_val=1., filter_size=11, filter_sigma=1.5).numpy()
    tf_time = time() - start
    print(f'photosynthesis result: {ps_ms_ssim}, time: {ps_time}s')
    print(f'tensorflow result: {tf_ms_ssim}, time: {tf_time}s\n')
    print(f'photosynthesis loss result: {ps_ms_ssim_loss}, time: {ps_loss_time}s')
    print(f'tensorflow result: {tf_ms_ssim.mean()}, time: {tf_time}s\n')

Output

>>>photosynthesis result: [0.06354865 0.06629971 0.07052894], time: 0.20274901390075684s
>>>tensorflow result: [0.05706326 0.08324481 0.08343431], time: 1.2265629768371582s

>>>photosynthesis loss result: 0.06679243594408035, time: 0.18335914611816406s
>>>tensorflow result: 0.07458078861236572, time: 1.2265629768371582s

>>>photosynthesis result: [0.06800205 0.07029662 0.07833724], time: 0.19111299514770508s
>>>tensorflow result: [0.08063222 0.08336634 0.09324924], time: 1.2148618698120117s

>>>photosynthesis loss result: 0.07221197336912155, time: 0.20091605186462402s
>>>tensorflow result: 0.08574926853179932, time: 1.2148618698120117s

Expected behavior
Same values are expected.

Additional context
The bug was found during inspection of issue #13 . I assume the problem is located at _compute_multi_scale_ssim. Possible fail points:

1. In the TensorFlow implementation of MS-SSIM, SSIM appears to return tensor of (batch, channels), while in current implementation tensor has only (batch) dimension.
2. First, the product should be estimated. Then, the result should be averaged along channel dimension. Current implementation performs it vice versa.

Add tests before more significant refactoring

Is your feature request related to a problem? Please describe.
Right now for all metrics, that require features, there is exist a function to compute it AND a class, which both have almost the same functionality. The only difference is that inside a class there is also an option to precompute features from an image.

Describe the solution you'd like
Idea is to move everything related to metric computation into compute_metric class method,
make a unified forward call which takes torch.Tensors and return torch.Tensor, moves them from GPU device to CPU and detaches if needed.

Describe alternatives you've considered
Alternatively, we could add an option to automatically turn images into features inside a class forward call, but as discussed with @snk4tr, initially proposed solution is better.

Is your feature request related to a problem? Please describe.
Create a separate external hosted documentation site.

Describe the solution you'd like
Use recommendations from here - Sphinx as a documentation generator and Read the Docs as a free hosting engine.

Describe alternatives you've considered
No other options under consideration at the moment.

Additional context
None

Is your feature request related to a problem? Please describe.
Add LPIPS measure, which is somewhat similar to Perceptual score.
Differences:

• Predictions are made on image patches
• Weights for different feature maps are learnable
• Feature maps are normalized to have same scale.

Describe the solution you'd like
Adjust code from official implementation repo: https://github.com/richzhang/PerceptualSimilarity

This should be closed before #26 , because then Perceptual score can be inherited from this score with some parameters fixed.

Is your feature request related to a problem? Please describe.
Currently there is no way to understand which interfaces are available and how to use them without reading code.

Describe the solution you'd like
Update readme with API description and examples.

Is your feature request related to a problem? Please describe.
Current implementation supports only 2, 3, and 4 dimensional tensors. However, in some domains like medical imaging it may be useful to be able to work with 5D tensors out of the box.

Describe the solution you'd like
Fix the _compute_ssim and _compute_multi_scale_ssim functions so they can handle 5D.

Describe alternatives you've considered
Fix top-level functions to make then iterate over the extra dimension and then reduce.

Additional context

Is your feature request related to a problem? Please describe.
There is no documentation for GS in Readme

Describe the solution you'd like
Add documentation for GS in Readme as it is done for other metrics

Additional context
Note that documentation has to be added in the same PR as the new metric itself

Is your feature request related to a problem? Please describe.
Implement BRISQUE metric or adapt existing implementation

Is your feature request related to a problem? Please describe.
With minimal examples added in #24 there is a risk that some of them may not work after some API changes or due to other problems.

Describe the solution you'd like
Add automatic tests that will verify that minimal examples work correctly.

Problem.
SSIM and MS-SSIM use set of parameters (size_average, reduction, reduce), which introduce same functionality.
TVLoss uses size_average, but this parameter is not needed.

Solution.
Remove redundant parameters and perform required optimisation of code.

Raised at #55 by @zakajd.

Is your feature request related to a problem? Please describe.
New metrics to measure full-reference image quality based on gradients similarity.
It was shown to improve Super-Resolution quality compared to usage of only MSE loss.

Describe the solution you'd like
Add Gradient magnitude similarity deviation (GMSD) https://arxiv.org/pdf/1308.3052.pdf
And Multiscale GMSD https://ieeexplore.ieee.org/document/7952357

Is your feature request related to a problem? Please describe.
As stated in #41 GS is computationally intense and this problem is solved now with CPU multiprocessing. Migration to PyTorch may help to solve this problem. It will also make code simpler due to the lack of error-prone multiprocessing.

Now only metric is implemented. Add implementation of differentiable PyTorch loss that can be used for training.

Is your feature request related to a problem? Please describe.
Currently there is no proof that our SSIM and MS-SSIM metrics are correct and give values consistent with official CPU implementations. Such tests need to be added.

Describe the solution you'd like
Take official and our implementations and run them on several different objects and problems. Compare results. Results need to be identical up to floating point operations error or another understandable error gap.

Describe alternatives you've considered
None

Additional context
None

Currently, linter check within the CI pipeline performs only very basic checks like line length being <120 symbols. PR review process shows that the majority of feedback in related to code style and formatting issues, which can be automated. Make checks more strict and descriptive.

Is your feature request related to a problem? Please describe.
It's used in several articles to compare GAN quality. Authors of MSID score states, that it's better than GS.

Describe the solution you'd like
https://github.com/KhrulkovV/geometry-score
https://arxiv.org/pdf/1802.02664.pdf

Describe the bug
Model init for InceptionV3 model both from this repo and from torchvision.models take very long time. Usually ~180-200 seconds, while other models init in 2-3 sec.
Problem arose a couple of month ago and discussed here

To Reproduce

from torchvision.models import inception_v3  
import photosynthesis_metrics as pm
import scipy
print(scipy.__version__)

model = inception_v3(pretrained=True, transform_input=True, progress=True)
pm_model = pm.feature_extractors.fid_inception.InceptionV3(
    resize_input=True, 
    normalize_input=True,
    requires_grad=False,
    use_fid_inception=True,
)

This could should take around 6 minutes to complete (3 minutes per model).

Expected behaviour
Fast model initialisation, not longer than a couple of seconds.

Fix
Problem comes from scipy package with version 1.4.1 defined in requirements.txt.
Uninstalling newer version and installing 1.3.3 fixed the issue for me. The loading time is much faster now. Just had to do pip install --upgrade scipy==1.3.3

Proposed solution
Change version in requirements.txt
This will increase testing speed 10x (now all tests run in 10 minutes, because of 3 model inits, after fix they will run in 1 minute).

Add MS-SSIM to the library