lidq92 / sfa Goto Github PK

[official] No reference image quality assessment based Semantic Feature Aggregation, published in ACM MM 2017, TMM 2019

License: MIT License

MATLAB 59.73% Python 40.27%

image-quality-assessment blind-image-quality-assessment camera-images resnet-50 image-content-variation plsr-model caffe

sfa's Introduction

Which Has Better Visual Quality: The Clear Blue Sky or a Blurry Animal?

Description

SFA code for the following papers:

Li, Dingquan, Tingting Jiang, Weisi Lin, and Ming Jiang. "Which Has Better Visual Quality: The Clear Blue Sky or a Blurry Animal?." IEEE Transactions on Multimedia, vol. 21, no. 5, pp. 1221-1234, May 2019.
Li, Dingquan, Tingting Jiang, and Ming Jiang. "Exploiting High-Level Semantics for No-Reference Image Quality Assessment of Realistic Blur Images." Proceedings of the 2017 ACM on Multimedia Conference. ACM, 2017.

Requirement

Framework: Caffe 1.0 + MATLAB 2016b Interface

The PLSR model uesd in the test code is trained on LIVE gblur images with DMOS (the larger the worse). w and best_layer in the journal extension are determined by five cross-validation (See TMMinter.m).

The ResNet-50-model.caffemodel is downloaded from KaimingHe/deep-residual-networks and it should be pasted into the directory models/ before you run the code! It's about 100MB which is too large to upload to this repo. If you have difficulty, you can also download the ResNet-50-model.caffemodel in my sharing on BaiduNetDisk with password u8sd.

New! We provide the PyTorch implementation of the method in SFA-pytorch

Notes

Note for training

All we need to train is a PLSR model, where the training function is plsregress in MATLAB. The features are extracted from the DCNN models pre-trained on the image classification task.

Update: remember to change the value of "im_dir" and "im_lists" in data info.

Note for datasets

You can download the datasets used in the papers from their owners for research purpose. If you have difficulty, you can refer to my sharing on BaiduNetDisk with password cu9j. We only consider the blur related images in this work.

Note for cross dataset evaluation

The reported Spearman correlation (SROCC) is multiplied by -1 when the training and testing datasets have different forms of subjective scores, i.e., one is MOS and the other is DMOS. This is to make sure that the prediction monotonicity is better when SROCC is closer to 1.

Citation

Please cite our papers if it helps your research:

@arcticle{li2018which,
  title={Which Has Better Visual Quality: The Clear Blue Sky or a Blurry Animal?},
  author={Li, Dingquan and Jiang, Tingting and Lin, Weisi and Jiang, Ming},
  booktitle={IEEE Transactions on Multimedia},
  volume={21}, 
  number={5}, 
  pages={1221-1234},  
  month={May},
  year={2019}, 
  doi={10.1109/TMM.2018.2875354}
}

[Paper]

@inproceedings{li2017exploiting,
  title={Exploiting High-Level Semantics for No-Reference Image Quality Assessment of Realistic Blur Images},
  author={Li, Dingquan and Jiang, Tingting and Jiang, Ming},
  booktitle={Proceedings of the 2017 ACM on Multimedia Conference},
  pages={378--386},
  year={2017},
  organization={ACM}
}

[Paper] [Poster]

Contact

Dingquan Li, dingquanli AT pku DOT edu DOT cn.

sfa's People

Contributors

Stargazers

Watchers

Forkers

dumpling426 vie-pku xwang0415 hxj525279 you-to-be zjh1124 lywww greitzmann sandy4321 zhangyingyue dale610 cv-ip flt19940317

sfa's Issues

test own picture

When I am researching the test_demo.py you giving found that the model exists in the VSFA,CNNIQA,WaDIQaM you gave, however this code does not save this model,How should I test blur images of myself without subjective scoring ?
Does the sfa file you give is a training code or a test code?

Test my own pictures by SFA-pytorch

If I want to test the pictures other than your dataset with SFA-pytorch (such as pictures I took myself——a dataset with about 6,000 images, only pictures with nothing else ), how should I do. I just want to test, no training.
If you can, could you give me an example, thank you for your help

Could you please write a pytorch version of test_cross_date.py or test_demo.py ,so that I can understand better

https://github.com/lidq92/CNNIQA/blob/master/test_cross_dataset.py
https://github.com/lidq92/CNNIQA/blob/master/test_cross_dataset.py
just like the CNNIQA project.
I am also a little confused where did you save the model ?

waiting for your reply~

Writing an Application layer

Hi,
The Pytorch code works and I am able to get some values. I have two questions:

Is their way to test the blurriness of a single image rather than preparing test dataset at run time with indexes ( i.e currently done by reading the matlab file )
How can I train the model again? Can you please elaborate, how to utilise .Mat files better?

Do you know where I can download the BID dataset?

Will the `PLCC be a negative number` during the `cross database` train and test?

Dear Dingquan,

Your work is awesome! It's truly great!!

As for the notation here:

Note for cross dataset evaluation
The reported Spearman correlation (SROCC) is multiplied by -1 when the training and testing datasets have different forms of subjective scores, i.e., one is MOS and the other is DMOS. This is to make sure that the prediction monotonicity is better when SROCC is closer to 1.

So, for cross-validation, if two databases used MOS and DMOS respectively, the SRCC will be a negative number. What about the PLCC? Will the PLCC be a negative number too?

I'm looking forward to your early reply and have a nice day!

Best,

Shuyue

Pytorch

I can't really find a pytorch version of this code.
Can anyone share with me? @lidq92

Error when testing

Hi, when test using your code, i have met these problems. Could you give me some suggestions?

F1113 20:51:42.569744 38502 upgrade_proto.cpp:95] Check failed: ReadProtoFromBinaryFile(param_file, param) Failed to parse NetParameter file: /home/liang/caffe/matlab/SFA/models/ResNet-50-model.caffemodel
This error was detected while a MEX-file was running. If the MEX-file
is not an official MathWorks function, please examine its source code
for errors. Please consult the External Interfaces Guide for information
on debugging MEX-files.
** This crash report has been saved to disk as /home/liang/matlab_crash_dump.38344-1 *

When i search in google, some people says that the reason of Question1 is due to the error of caffemodel. But i download the caffemodel on the https://github.com/KaimingHe/deep-residual-networks.

Others say that the error is due to multi-gpu. But i use caffe.set_device(0), this problem still exists. How do you think?

Moreover, could you provide the dataset of BID and CLIVE, it is difficult to get on the internet.

demo

你好，能提供一下python语言的demo吗？

关于pytorch程序

请问Which Has Better Visual Quality: The Clear Blue Sky or a Blurry Animal?论文的pytorch程序什么时候会公布，谢谢。

about SFA_pytorch how to use adaptive layer selection procedure

hello,about SFA_pytorch how to use adaptive layer selection procedure, on the readme.md 'Add adaptive layer selection procedure used in the journal version'.Is it me who needs to rewrite this file? Or can the author provide the appropriate version code?

The link of "my sharing on BaiduNetDisk" is gone!!

Hi, it seems downloading these database is difficult, can you update your database link? Thanks!

how to great data.mat

Hello, I would like to consult with you,
How is your data.mat folder built, What does each variable mean, and what does it contain? Especially how your index data is created? I want to test the new datasets with your model, but I don t know how to create the datasets.mat (like KonIQ.mat) document. Hope for your help.
Thank you very much for your help!
Thank you.