Hi,
I am tring to test the modle,but when I have finished to install the numpy/scipy/matplotlib/pandas packages and run ./unittest,the err ocurrs.

Do u know why?

I heard we can print ms-ssim values while running run_vmaf, is it true? How can I enable it? Did not find documentation when running ./run_vmaf --help.

$ ./run_vmaf --help
usage: run_vmaf fmt width height ref_path dis_path [--model model_path] [--out-fmt out_fmt]

fmt:
	yuv420p
	yuv422p
	yuv444p
	yuv420p10le
	yuv422p10le
	yuv444p10le

out_fmt:
	text (default)
	xml
	json

I'll try to explain what I did and hopefully it will make sense. I took a bunch of the netflix-provided "4K" files from xiph's website (http://media.xiph.org/video/derf/). For the purposes of this discussion, I'll focus on the example where I took DinnerScene and FoodMarket2. I downsampled them to 360p 8bit 100frames using ffmpeg, and then encoded them using my own VP9 encoder (http://www.twoorioles.com/eve-vp9-video-encoder) at fixed base quantizers (DinnerScene: 40, 90, 140, 190, and FoodMarket2 any even quantizer between 40 and 254). "Base quantizer" here means that "normal" P-frames are encoded using that quantizer (or some value roughly around it), and "ARF frames" are encoded at a lower quantizer defined by all kind of stuff in the encoder. I don't think that's very relevant, nor is it relevant that this is VP9. You could probably reproduce this using x264 or libvpx also. An example of file sizes for these 100-frame, 360p sequences:

21523 res/DinnerScene.q140.ivf
9510 res/DinnerScene.q190.ivf
118710 res/DinnerScene.q40.ivf
46032 res/DinnerScene.q90.ivf
362298 res/FoodMarket2.q140.ivf
123366 res/FoodMarket2.q190.ivf
2222365 res/FoodMarket2.q40.ivf
922392 res/FoodMarket2.q90.ivf

As for quantizers-in-P-frames, I confirmed that the quantizer of the second frame in each sequence above is 39,90,141,190 for DinnerScene and 40,90,141,193 for FoodMarket2 in the sequences above. I hope that explains my input data.

Next, I tried to simply concatenate the decoded sequences of any combination of DinnerScene+FoodMarket2 (where the base quantizer of FoodMarket2 is equal to or greater than the base quantizer in DinnerScene) and measure the sequence VMAF (the last line "Aggregate: [..] VMAF_score:([0-9.]+)" from your tools) and PSNR ("geometric" as well as "arithmetic" - although I use libvpx terminology, i.e. "average" avg_psnr=avg(sse_to_psnr(frame_sse)) and "global" glb_psnr=sse_to_psnr(avg(sse_frame)), where sse_to_psnr is the typical formula from wikipedia. As file size, I simply used the sum of the two filesizes of the input IVF files. This is not exact (since there's two IVF file headers) but it's close enough. Together, this allows me to plot a typical bitrate vs. quality curve. Note that in my plots, I plot VMAF logaritmic also (basically 10*log10(100.0/vmaf_score)), and I understand that this is probably strange, but it slightly increases resolution near the 100.0 score in the graphs.

So, in these graphs, each dot is a combination. It's easiest to look at the graphs as 4 lines, where each line is a combination of DinnerScene at q=40/90/140/190 and FoodMarket2=dinnerscene_q+factor, where factor is an even number between 0 and 120. The leftmost dot in each line ("largest file") has factor=0 and the rightmost dot per line ("smallest file") has factor=120. The factor is the delta quantizer between the two files. Let's first look at glb PSNR and avg PSNR. It's no surprise that glb PSNR biases towards a higher relative weight for hard content, and avg PSNR biases towards a higher relative weight for easy content. This is well-established and is mathematically easy to demonstrate. I'm not making any statements on which one is better, they are just there to make a point.

I could, for example, make a statement that according to glb PSNR, the x value for the top line's factor=56 point is almost identical to the x value for the second line's factor=0 point, but the y value for the second line's is higher, so at the same combined average bitrate (1.16 mbps), glb PSNR (slightly) prefers dinnerscene at q=90 and foodmarket2 at q=90 to dinnerscene at q=40 and foodmarket2 at q=96.

Likewise, I can make a statement that according to avg PSNR, the top line's x value is identical (at factor=120) to the second line's x value at factor=58 and the third line at factor=4. However, the top point's y value is slightly better than the second point's y value, and both are much much better than the third point's y value. Therefore, at the same combined average bitrate (426/431 kbps), avg PSNR prefers dinnerscene at q=40 and foodmarket2 at q=160 to dinnerscene at q=90 and foodmarket2 at q=148, and both are much preferred to dinnerscene at q=140 and foodmarket2 at q=144.

Now, let's look at the third graph, lVMAF (logarithmic VMAF). This graph looks a lot more like avg than glb PSNR, but not exactly, because we at least have intersection points, which the other graph didn't have. This should allow us to compare the same sequence (but with fragments encoded at different base quantizers) which have the same file size as well as the same aggregate VMAF score. I chose the point of the top line at factor=116 (dinnerscene q=40 and foodmarket2 q=158) and second line at factor=54 (dinnerscene q=90 and foodmarket2 q=144), which gives a combined bitrate of 454-457 kbps, and have a nearly identical VMAF score of 94.317 and 94.34 (logarithmic VMAF=12.454/12.472). I'd like to attach these files here but each of them (losslessly re-compressed) is 20MB and the bug tracker doesn't like that (10 MB/file limit). I'm suspecting you can easily re-create these files and the experiment yourself or I can upload them somewhere else where 20MB per file for 2 files is not an issue.

The point is that I visually look at the differences between the two files, and I don't think I agree. Here's the last frame of dinnerscene at q=40 and q=90:

And here's the 4th frame of foodmarket2 at q=158 and q=144:

To me, visually, the differences between the second set of images (q=144 and q=158) is much greater than between the first set of images (at q=40 and q=90), and thus the combination of q=90+q=144 should be preferred (possibly by a pretty big margin) over the combination of q=40+q=158. The fact that it brings the relative quality of the two sequences less far apart is a bonus (to me, psychologically, it would seem that "good" quality in one segment does not erase my recent memory of "horrible" quality in another) but I'm not enough of an expert in this field to be able to make a case over whether that's relevant or not.

This is an example, obviously. When you go to extreme cases (try ToddlerFountain+DinnerScene), you get seemingly crazy peaks where the scores claim that DinnerScene at q=40 and ToddlerFountain at q=180 (factor=140) is about the same as DinnerScene at q=90 and ToddlerFountain at q=176 (factor=86), which is again a lot better than DinnerScene at q=140 and ToddlerFountain at q=174 (factor=34), all at the same combined bitrate of about 900 kbps. At some level this seems almost obvious, since what could possibly be the visual difference of ToddlerFountain at q=180, 176 or 174? Whereas a visual difference at q=140 and q=40 should be pretty significant, right? But it seems to call for seemingly absurd (?) quality variations between easy and hard content. Optimal factor (i.e. the one giving optimal VMAF scores) in this case would likely be around 115 (or at least somewhere relatively in the middle of 140 and 86), which means effective delta quantizer would be around 10, since the effective quantizer in vp9 increases by 1.02 per index, and pow(1.02, 115)=9.75.

Example plot for logarithmic VMAF scores for dinnerscene at q=40,90,140,190 and toddlerfountain at q=dinnerscene_q+factor, where factor is all even numbers between 0 and 200:

I guess I have an intrinsic bias in me that wants to believe that larger relative quality differences would have a negative impact on the overall viewing experience, but I'm not an expert in this field so I don't know if that makes sense or not. Beyond the examples given here, I've looked at some other examples, and I feel that what I presented here for one example is generally applicable to VMAF, i.e. that it may be biasing a little too much towards higher overall scores because of high frame scores on easy content regardless of the scores on hard content. I hope that makes sense as a "potential bug" report.

zli@zhi-ubuntu14-vm:~/Projects/stash/MCE/transcoder/vmaf_oss/vmaf/feature$ ./ssim yuv420p /home/zli/Desktop/debug_ssim/AttackOnTitan_1026_2.yuv /home/zli/Desktop/debug_ssim/AttackOnTitan_5v_1026_2.yuv 1920 1080
ssim: 0 -0.016983
ssim_l: 0 -0.046559
ssim_c: 0 0.665225
ssim_s: 0 0.508533
ssim: 1 -0.018055
ssim_l: 1 -0.046968
ssim_c: 1 0.664024
ssim_s: 1 0.508169

Hey there, after I ran make I wanted to run unittest to verify the result, but it does not seem to be working. Here is the trace:

.test libsvmnusvr train and predict...
.test random forest train and predict...
.test train, save, load and predict...
.test libsvmnusvr train, save, load and predict...
.......
======================================================================
FAIL: test_to_json (result_test.ResultFormattingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jyx/code/vmaf/python/test/result_test.py", line 147, in test_to_json
    '{\n    "executorId": "SSIM_V1.0", \n    "asset": {\n        "identifier": "test_0_0_checkerboard_1920_1080_10_3_0_0_1920x1080_vs_checkerboard_1920_1080_10_3_1_0_1920x1080_q_1920x1080"\n    }, \n    "frames": [\n        {\n            "frameNum": 0, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935802, \n            "SSIM_score": 0.933353\n        }, \n        {\n            "frameNum": 1, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935802, \n            "SSIM_score": 0.933353\n        }, \n        {\n            "frameNum": 2, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935803, \n            "SSIM_score": 0.933354\n        }\n    ], \n    "aggregate": {\n        "SSIM_feature_ssim_c_score": 0.99740399999999996, \n        "SSIM_feature_ssim_l_score": 0.99998299999999996, \n        "SSIM_feature_ssim_s_score": 0.93580233333333329, \n        "SSIM_score": 0.93335333333333337, \n        "method": "mean"\n    }\n}')
AssertionError: '{\n    "executorId": "SSIM_V1.0", \n    "asset": {\n        "identifier": "test_0_0_checkerboard_1920_1080_10_3_0_0_1920x1080_vs_checkerboard_1920_1080_10_3_1_0_1920x1080_q_1920x1080"\n    }, \n    "frames": [\n        {\n            "frameNum": 0, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935802, \n            "SSIM_score": 0.933353\n        }, \n        {\n            "frameNum": 1, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935802, \n            "SSIM_score": 0.933353\n        }, \n        {\n            "frameNum": 2, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935803, \n            "SSIM_score": 0.933354\n        }\n    ], \n    "aggregate": {\n        "SSIM_feature_ssim_c_score": 0.997404, \n        "SSIM_feature_ssim_l_score": 0.999983, \n        "SSIM_feature_ssim_s_score": 0.9358023333333333, \n        "SSIM_score": 0.9333533333333334, \n        "method": "mean"\n    }\n}' != '{\n    "executorId": "SSIM_V1.0", \n    "asset": {\n        "identifier": "test_0_0_checkerboard_1920_1080_10_3_0_0_1920x1080_vs_checkerboard_1920_1080_10_3_1_0_1920x1080_q_1920x1080"\n    }, \n    "frames": [\n        {\n            "frameNum": 0, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935802, \n            "SSIM_score": 0.933353\n        }, \n        {\n            "frameNum": 1, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935802, \n            "SSIM_score": 0.933353\n        }, \n        {\n            "frameNum": 2, \n            "SSIM_feature_ssim_c_score": 0.997404, \n            "SSIM_feature_ssim_l_score": 0.999983, \n            "SSIM_feature_ssim_s_score": 0.935803, \n            "SSIM_score": 0.933354\n        }\n    ], \n    "aggregate": {\n        "SSIM_feature_ssim_c_score": 0.99740399999999996, \n        "SSIM_feature_ssim_l_score": 0.99998299999999996, \n        "SSIM_feature_ssim_s_score": 0.93580233333333329, \n        "SSIM_score": 0.93335333333333337, \n        "method": "mean"\n    }\n}'

----------------------------------------------------------------------
Ran 234 tests in 372.974s

Does anyone know why?

At the moment videos have to be read in raw YUV format, which requires creating huge intermediary files when attempting to compare the output of different encodes. It would be extremely more useful to be able to read encoded H264/VP9/H265,etc. video directly, without having to decode to disk for every video file to compare.

This can be done withffmpeg or libav, but doing so at the moment would involve changing both the Python code as well as the C code in several places. So perhaps this needs to wait for a refactoring in issue #42. It would be nice if as part of that refactoring an interface was developed that made it possible to swap in an alternate way of reading frames.

ptools folder is the project's folder, it is always there.
error information:
/wrapper/obj/main.o /home/cqy/vmaf/wrapper/obj/pugixml.o -L/home/cqy/vmaf/wrapper/../ptools -lm -lpthread -lptools -Wl,-rpath=/home/cqy/vmaf/wrapper/../ptools
/usr/bin/ld: cannot find -lptools
collect2: error: ld returned 1 exit status
make[1]: *** [vmafossexec] Error 1

anyone can help?

Hello; i have installed VMAF on an OpenSuse server (version 13.1); I am seeing errors when running the unittest suite as described in the Readme @ github.com/Netflix/VMAF .

some score assertions have differences up to 4 places. for example:
FAIL: test_all (feature_test.FeatureTestYuv422p10le)

Traceback (most recent call last):
File "/home/vqeg/git/vqegjeg.git/netflix/vmaf/feature/test/feature_test.py", line 429, in test_all
self.assertAlmostEquals(score, 6899.24648475, places=4)
AssertionError: 6899.246359479167 != 6899.24648475 within 4 places

Are errors of this type okay and safely ignored? any information is appreciated.
Thank you very much for making this tool available!

I have attached the otput file to this posting:
unittest.txt

Regards,
steves

Hello Everyone,

I am using 'run_vmaf' to compare an original video with a distorted video, yet it is really taking long time in the order of minutes. Is that normal? Is there something that I am missing?

This is my Python code for the vmaf call:

sp.call( (path_to_vmaf_tool + ' ' + pix_fmt + ' ' + width + ' ' + ' ' + height + ' ' + original_video + ' ' + encoded_video + ' --out-fmt text > ' + output_path + 'vmaf.txt'), shell=True)

Thanks!
Hossam

Hi,

I did a quick comparison of the SSIM calculations using

vmaf_psnr: feature/psnr
vmaf_ssim: feature/ssim, using the ssim: \d+ value
ycbcr_{psnr,ssim}: https://github.com/figgis/yuv-tools
ffmpeg: ffmpeg -s SIZE -i REC -s SIZE -i DISTORTED -lavfi "ssim="stats_file=stats_ssim.log";[0:v][1:v]"psnr="stats_psnr.log" -f null -

As you can see the SSIM values differs quite alot compared to other tools (I tried some more versions of ssim that I had available and the vmaf version differs compared to them as well).

Do you have an explanation for this?

Hi All,

I run into some errors when do unittest.
I am using OSX 10.11.6. g++ has been upgrade to 4.8.1

[ 3:42PM ] [ jack@jacksmb15retina:~/NetFlix/vmaf(master✗) ]
$ g++ --version
g++-4.8 (Homebrew GCC 4.8.5) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

All other package have been upgrade to required versions. I have attached the unittest log in below.
Thanks ~~~~~~~~

[ 3:35PM ] [ jack@jacksmb15retina:/NetFlix/vmaf(master✗) ]
$ ls
CHANGELOG.md VERSION feature resource
CONTRIBUTING.md Xcode ffmpeg2vmaf run_psnr
Dockerfile build.gradle gradle run_testing
Dockerfile.dev check-environment gradlew run_vmaf
FAQ.md de1577a0-abe9-4035-8dd8-a1c90d911105 gradlew.bat run_vmaf_in_batch
LICENSE docker-bash libsvm run_vmaf_training
Makefile example_batch_input matlab settings.gradle
NOTICE.md example_dataset.py model unittest
OSSMETADATA example_raw_dataset.py ptools workspace
README.md extratest python wrapper
[ 3:35PM ] [ jack@jacksmb15retina:/NetFlix/vmaf(master✗) ]
$ ls
CHANGELOG.md VERSION feature resource
CONTRIBUTING.md Xcode ffmpeg2vmaf run_psnr
Dockerfile build.gradle gradle run_testing
Dockerfile.dev check-environment gradlew run_vmaf
FAQ.md de1577a0-abe9-4035-8dd8-a1c90d911105 gradlew.bat run_vmaf_in_batch
LICENSE docker-bash libsvm run_vmaf_training
Makefile example_batch_input matlab settings.gradle
NOTICE.md example_dataset.py model unittest
OSSMETADATA example_raw_dataset.py ptools workspace
README.md extratest python wrapper
[ 3:35PM ] [ jack@jacksmb15retina:/NetFlix/vmaf(master✗) ]
$ cd python/src
[ 3:35PM ] [ jack@jacksmb15retina:/NetFlix/vmaf/python/src(master✗) ]
$ ls
vmaf
[ 3:35PM ] [ jack@jacksmb15retina:/NetFlix/vmaf/python/src(master✗) ]
$ pwd
/Users/jack/NetFlix/vmaf/python/src
[ 3:35PM ] [ jack@jacksmb15retina:/NetFlix/vmaf/python/src(master✗) ]
$ vim /.bash_profile
[ 3:36PM ] [ jack@jacksmb15retina:/NetFlix/vmaf/python/src(master✗) ]
$ source /.bash_profile
[ 3:36PM ] [ jack@jacksmb15retina:/NetFlix/vmaf/python/src(master✗) ]
$ cd ../..
[ 3:36PM ] [ jack@jacksmb15retina:/NetFlix/vmaf(master✗) ]
$ ls
CHANGELOG.md VERSION feature resource
CONTRIBUTING.md Xcode ffmpeg2vmaf run_psnr
Dockerfile build.gradle gradle run_testing
Dockerfile.dev check-environment gradlew run_vmaf
FAQ.md de1577a0-abe9-4035-8dd8-a1c90d911105 gradlew.bat run_vmaf_in_batch
LICENSE docker-bash libsvm run_vmaf_training
Makefile example_batch_input matlab settings.gradle
NOTICE.md example_dataset.py model unittest
OSSMETADATA example_raw_dataset.py ptools workspace
README.md extratest python wrapper
[ 3:36PM ] [ jack@jacksmb15retina:/NetFlix/vmaf(master✗) ]
$ make
for dir in ptools feature libsvm wrapper; do
cd ${dir}; /Applications/Xcode.app/Contents/Developer/usr/bin/make; cd ..;
done
g++ -pthread -Wall -Wextra -fpic -O -fno-strict-aliasing -DLINUX_ -DOC_NEW_STYLE_INCLUDES -Wno-deprecated -I./opencontainers_1_8_4/include -pthread -D_REENTRANT m2pythontools.o valpython.o midassocket.o valprotocol2.o m2ser.o m2streamdataenc.o m2convertrep.o timeconv.o -dynamiclib -install_name /Users/jack/NetFlix/vmaf/ptools/libptools.dylib -o libptools.dylib
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
make[1]: Nothing to be done for all'. make[1]: Nothing to be done for all'.
make[1]: Nothing to be done for `all'.
cd libsvm; /Applications/Xcode.app/Contents/Developer/usr/bin/make lib; cd ..;
if [ "Darwin" = "Darwin" ]; then
SHARED_LIB_FLAG="-dynamiclib -Wl,-install_name,libsvm.so.2";
else
SHARED_LIB_FLAG="-shared -Wl,-soname,libsvm.so.2";
fi;
c++ ${SHARED_LIB_FLAG} svm.o -o libsvm.so.2
[ 3:36PM ] [ jack@jacksmb15retina:~/NetFlix/vmaf(master✗) ]
$ ./unittest
...................Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/run_testing", line 10, in
from vmaf.core.quality_runner import QualityRunner, VmafQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays
FFTraceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/run_testing", line 10, in
from vmaf.core.quality_runner import QualityRunner, VmafQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays
FTraceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/run_testing", line 10, in
from vmaf.core.quality_runner import QualityRunner, VmafQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays
FTraceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/run_testing", line 10, in
from vmaf.core.quality_runner import QualityRunner, VmafQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays
FFFTraceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/run_vmaf_training", line 10, in
from vmaf.routine import print_matplotlib_warning, train_test_vmaf_on_dataset
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/routine.py", line 10, in
from vmaf.core.quality_runner import VmafQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays
FTraceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/run_vmaf_training", line 10, in
from vmaf.routine import print_matplotlib_warning, train_test_vmaf_on_dataset
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/routine.py", line 10, in
from vmaf.core.quality_runner import VmafQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays
FE...........................test on feature assembler with selected atom features...
.test on feature assembler with whole feature...
....test on running Moment feature extractor...
.test on running MS-SSIM feature extractor...
.test on running PSNR feature extractor...
.test on running SSIM feature extractor...
.test on running VMAF feature extractor...
.test on running VMAF feature extractor on checkerboard...

.test on running VMAF feature extractor on flat pattern...
.test on running VMAF feature extractor with result store...
running for the first time with fresh calculation...
running for the second time with stored results...
.test on running Moment feature extractor in parallel...
.test on running MS-SSIM feature extractor in parallel...
.test on running SSIM feature extractor in parallel...
.test on running VMAF feature extractor with result store in parallel...
running for the first time with fresh calculation...
running for the second time with stored results...
.test on running VMAF feature extractor in parallel...
.test on checkerboard pattern identical...
.test on checkerboard pattern opposite...
.test on checkerboard pattern shifted by 1...
.test on flat pattern identical...
.test on flat pattern of value 10...
.test 2nd moment...
.test adm...
.test all...
.test ansnr...
.test motion...
.test ms_ssim...
.test psnr...
.test ssim...
.test vif...
.test adm on yuv422p10le...
.test all on yuv422p10le...
.test ansnr on yuv422p10le...
.test motion on yuv422p10le...
.test ms_ssim on yuv422p10le...
.test psnr on yuv422p10le...
.test ssim on yuv422p10le...
.test vif on yuv422p10le...
.Etest on running Moment noref feature extractor on Assets...
.test on running Moment noref feature extractor on NorefAssets...
.test on running Moment noref feature extractor on NorefAssets in parallel...
.test on running Moment noref feature extractor on Assets...
.test on running Moment noref feature extractor on NorefAssets...
.............Etest on running dis YUV raw video extractor...
.test on running dis YUV raw video extractor...
.test on running dis YUV raw video extractor in parallel (disabled)...
.test on running asset extractor...
........EE..../Users/jack/anaconda/lib/python2.7/site-packages/scipy/linalg/basic.py:1018: RuntimeWarning: internal gelsd driver lwork query error, required iwork dimension not returned. This is likely the result of LAPACK bug 0038, fixed in LAPACK 3.2.2 (released July 21, 2010). Falling back to 'gelss' driver.
warnings.warn(mesg, RuntimeWarning)
............=== Belief Propagation ===
Iteration 254: change 9.91555463312e-09, mean q_e 3.54828453357, mean b_s -0.00349388118573, mean sigma_s 0.603144852442
.=== Belief Propagation ===
Iteration 307: change 9.78896915482e-09, mean q_e 3.55730737817, mean b_s -0.0125167257814, mean sigma_s 0.809166336017
.=== Belief Propagation ===
Iteration 260: change 9.81422873693e-09, mean q_e 3.54366775631, mean b_s -0.00711692999394, mean sigma_s 0.596951479671
=== Belief Propagation ===
Iteration 465: change 9.95001902283e-09, mean q_e 3.55279757432, mean b_s 0.0022204564692, mean sigma_s 0.5767783932585
..=== Belief Propagation ===
Iteration 233: change 9.65017600353e-09, mean q_e 2.99728217893, mean b_s -0.0346687009615, mean sigma_s 0.490088018146
.=== Belief Propagation ===
Iteration 252: change 9.77726429001e-09, mean q_e 3.54796102141, mean b_s -0.00317036901959, mean sigma_s 0.603144852444
.=== Belief Propagation ===
Iteration 254: change 9.91555466308e-09, mean q_e 3.77661948975, mean b_s -0.00349388118573, mean sigma_s 0.603144852442
.=== Belief Propagation ===
Iteration 263: change 9.58227168354e-09, mean q_e -1.03011832538e-15, mean b_s -6.96592160669e-17, mean sigma_s 0.447249964541
.=== Belief Propagation ===
Iteration 262: change 9.91501519175e-09, mean q_e -2.52962208142e-16, mean b_s 2.012306542e-16, mean sigma_s 0.44493098698827
.=== Belief Propagation ===
Iteration 2144: change 9.96715706524e-09, mean q_e 3.65276492988, mean b_s 0.120360678571, mean mu_c 0.0, mean sigma_s 0.3909487505, mean delta_c 0.43303199468899
.=== Belief Propagation ===
Iteration 2163: change 9.9673970506e-09, mean q_e 3.54693345078, mean b_s -0.00214279851342, mean mu_c 0.0, mean sigma_s 0.390948750454, mean delta_c 0.4330319947236
.=== Belief Propagation ===
Iteration 1152: change 9.95409015337e-09, mean q_e 3.54500609959, mean b_s -0.00842433540402, mean mu_c 0.0, mean sigma_s 0.377572857903, mean delta_c 0.434491608589
=== Belief Propagation ===
Iteration 2606: change 9.95227409239e-09, mean q_e 3.54809647554, mean b_s 0.0087044170798, mean mu_c 0.0, mean sigma_s 0.473766057786, mean delta_c 0.2924269824185
.=== Belief Propagation ===
Iteration 254: change 9.91555476002e-09, mean q_e 3.54828453357, mean b_s -0.00349388118573, mean mu_c 0.0, mean sigma_s 0.60314485244, mean delta_c 0.00
.=== Belief Propagation ===
Iteration 1: change 1.2161883889e-15, mean q_e 3.54479065239, mean b_s 0.0, mean mu_c 0.0, mean sigma_s 0.0, mean delta_c 0.674424698149
........E

ERROR: cross_validation_test (unittest.loader.ModuleImportFailure)

ImportError: Failed to import test module: cross_validation_test
Traceback (most recent call last):
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/jack/NetFlix/vmaf/python/test/cross_validation_test.py", line 6, in
from vmaf.core.train_test_model import SklearnRandomForestTrainTestModel, LibsvmNusvrTrainTestModel,
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays

======================================================================
ERROR: local_explainer_test (unittest.loader.ModuleImportFailure)

ImportError: Failed to import test module: local_explainer_test
Traceback (most recent call last):
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/jack/NetFlix/vmaf/python/test/local_explainer_test.py", line 9, in
from vmaf.core.local_explainer import LocalExplainer
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/local_explainer.py", line 5, in
import sklearn.metrics
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays

======================================================================
ERROR: quality_runner_test (unittest.loader.ModuleImportFailure)

ImportError: Failed to import test module: quality_runner_test
Traceback (most recent call last):
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/jack/NetFlix/vmaf/python/test/quality_runner_test.py", line 11, in
from vmaf.core.quality_runner import VmafLegacyQualityRunner, VmafQualityRunner,
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays

======================================================================
ERROR: result_test (unittest.loader.ModuleImportFailure)

ImportError: Failed to import test module: result_test
Traceback (most recent call last):
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/jack/NetFlix/vmaf/python/test/result_test.py", line 14, in
from vmaf.core.quality_runner import VmafLegacyQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays

======================================================================
ERROR: routine_test (unittest.loader.ModuleImportFailure)

ImportError: Failed to import test module: routine_test
Traceback (most recent call last):
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/jack/NetFlix/vmaf/python/test/routine_test.py", line 5, in
from vmaf.routine import train_test_vmaf_on_dataset, read_dataset, run_test_on_dataset, generate_dataset_from_raw
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/routine.py", line 10, in
from vmaf.core.quality_runner import VmafQualityRunner
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/quality_runner.py", line 11, in
from vmaf.core.train_test_model import TrainTestModel
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays

======================================================================
ERROR: train_test_model_test (unittest.loader.ModuleImportFailure)

ImportError: Failed to import test module: train_test_model_test
Traceback (most recent call last):
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/Users/jack/anaconda/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
import(name)
File "/Users/jack/NetFlix/vmaf/python/test/train_test_model_test.py", line 7, in
from vmaf.core.train_test_model import TrainTestModel,
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/core/train_test_model.py", line 5, in
from sklearn.metrics import f1_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/init.py", line 33, in
from . import cluster
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/init.py", line 23, in
from .bicluster import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/init.py", line 1, in
from .bicluster_metrics import consensus_score
File "/Users/jack/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/bicluster/bicluster_metrics.py", line 6, in
from sklearn.utils.validation import check_arrays
ImportError: cannot import name check_arrays

======================================================================
FAIL: test_run_cleaning_cache_psnr (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 102, in test_run_cleaning_cache_psnr
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_testing PSNR /Users/jack/NetFlix/vmaf/python/test/resource/example_dataset.py --parallelize --cache-result --suppress-plot

======================================================================
FAIL: test_run_psnr (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 95, in test_run_psnr
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_psnr yuv420p 576 324 /Users/jack/NetFlix/vmaf/python/test/resource/yuv/src01_hrc00_576x324.yuv /Users/jack/NetFlix/vmaf/python/test/resource/yuv/src01_hrc01_576x324.yuv >/dev/null 2>&1

======================================================================
FAIL: test_run_testing_psnr (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 45, in test_run_testing_psnr
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_testing PSNR /Users/jack/NetFlix/vmaf/python/test/resource/example_dataset.py --parallelize --suppress-plot

======================================================================
FAIL: test_run_testing_vmaf (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 31, in test_run_testing_vmaf
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_testing VMAF /Users/jack/NetFlix/vmaf/python/test/resource/example_dataset.py --parallelize --suppress-plot

======================================================================
FAIL: test_run_testing_vmaf_raw_dataset (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 38, in test_run_testing_vmaf_raw_dataset
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_testing VMAF /Users/jack/NetFlix/vmaf/python/test/resource/example_raw_dataset.py --parallelize --suppress-plot

======================================================================
FAIL: test_run_vmaf (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 87, in test_run_vmaf
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_vmaf yuv420p 576 324 /Users/jack/NetFlix/vmaf/python/test/resource/yuv/src01_hrc00_576x324.yuv /Users/jack/NetFlix/vmaf/python/test/resource/yuv/src01_hrc01_576x324.yuv >/dev/null 2>&1

======================================================================
FAIL: test_run_vmaf_in_batch (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 79, in test_run_vmaf_in_batch
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_vmaf_in_batch /Users/jack/NetFlix/vmaf/workspace/workdir/test_batch_input --parallelize >/dev/null 2>&1

======================================================================
FAIL: test_run_vmaf_training (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 55, in test_run_vmaf_training
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_vmaf_training /Users/jack/NetFlix/vmaf/python/test/resource/example_dataset.py /Users/jack/NetFlix/vmaf/python/test/resource/vmaf_v4.py /Users/jack/NetFlix/vmaf/python/test/resource/vmaf_v4.py /Users/jack/NetFlix/vmaf/workspace/workdir/tmp.pkl --parallelize --suppress-plot

======================================================================
FAIL: test_run_vmaf_training_raw_dataset (command_line_test.CommandLineTest)

Traceback (most recent call last):
File "/Users/jack/NetFlix/vmaf/python/test/command_line_test.py", line 65, in test_run_vmaf_training_raw_dataset
ret = run_process(cmd, shell=True)
File "/Users/jack/NetFlix/vmaf/python/src/vmaf/tools/misc.py", line 357, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 1, cmd: /Users/jack/NetFlix/vmaf/run_vmaf_training /Users/jack/NetFlix/vmaf/python/test/resource/example_raw_dataset.py /Users/jack/NetFlix/vmaf/python/test/resource/vmaf_v4.py /Users/jack/NetFlix/vmaf/python/test/resource/vmaf_v4.py /Users/jack/NetFlix/vmaf/workspace/workdir/tmp.pkl --parallelize --suppress-plot

Ran 166 tests in 138.016s

FAILED (failures=9, errors=6)

When I run:

./run_vmaf yuv420p 576 324 python/test/resource/yuv/src01_hrc00_576x324.yuv python/test/resource/yuv/src01_hrc01_576x324.yuv --out-fmt json

I get an AsserionError:

Illegal instruction
Traceback (most recent call last):
File "./run_vmaf", line 143, in
ret = main()
File "./run_vmaf", line 107, in main
runner.run()
File "/home/ubuntu/vmaf/python/src/vmaf/core/executor.py", line 120, in run
self.results = map(self._run_on_asset, self.assets)
File "/home/ubuntu/vmaf/python/src/vmaf/core/quality_runner.py", line 311, in _run_on_asset
vmaf_fassembler.run()
File "/home/ubuntu/vmaf/python/src/vmaf/core/feature_assembler.py", line 82, in run
runner.run(parallelize=self.parallelize)
File "/home/ubuntu/vmaf/python/src/vmaf/core/executor.py", line 120, in run
self.results = map(self._run_on_asset, self.assets)
File "/home/ubuntu/vmaf/python/src/vmaf/core/executor.py", line 286, in _run_on_asset
self._generate_result(asset)
File "/home/ubuntu/vmaf/python/src/vmaf/core/feature_extractor.py", line 162, in _generate_result
run_process(vmaf_feature_cmd, shell=True)
File "/home/ubuntu/vmaf/python/src/vmaf/tools/misc.py", line 374, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 132, cmd: /home/ubuntu/vmaf/feature/vmaf all yuv420p python/test/resource/yuv/src01_hrc00_576x324.yuv python/test/resource/yuv/src01_hrc01_576x324.yuv 576 324 >> /home/ubuntu/vmaf/workspace/workdir/c40e5bb6-9542-45af-92cf-f6e0f37f1d58/VMAF_feature_V0.2.4b_ef6af54084b6d2f6fd4ad0c6c2301830c53a9abf

What could be the problem?

Thanks

I'm using Netflix public data and using below commands to train model parameters

./run_vmaf_training resource/dataset/NFLX_dataset_public.py resource/feature_param/vmaf_feature_v3.py resource/model_param/libsvmnusvr_v3.py workspace/model/test_model.pkl --parallelize

But I got different parameter by comparing with all files under resource/model. Also if I use new trained model and got different scores by running

./run_vmaf yuv420p 576 324 resource/yuv/src01_hrc00_576x324.yuv resource/yuv/src01_hrc01_576x324.yuv --model workspace/model/test_model.pkl

This should be understandable because of different trained parameters. So can we list out which dataset are used to training each model files under resource/model?

I tried to build it in Cygwin latest version.
All the targets except "wrapper" directory compiled successfully.

That is, vmaf-master/wrapper/Makefile doesn't seem to work in Cygwin.
I added "-D_GNU_SOURCE" to CXXFLAGS in Makefile to let several C++ source file compile successfully.
ld.exe complained "cannot find -lptools" and the linking was not successful.

Does anyone know how to make it right?

Thank you.

Whenever I do a ./run_vmaf on an original source and transcoded output, I only get a small subset of frames reported. For example, using a video trailer and a transcoded version with 4496 frames each (verified with ffprobe) at the same geometry and framerate, only 34 frames are reported from the run_vmaf command.

Does VMAF simply sample spots of a video or should this not be happening?

Is this a bug or is something wrong with the model? I've used both ffmpegvmaf solution and even directly converted the inputs to the raw yuv format and then using run_vmaf but several frames in a row get a score of 0.

Here is a snippet

        {
            "frameNum": 3149, 
            "VMAF_feature_adm2_score": 0.18777403146762484, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.04195884384335988, 
            "VMAF_feature_vif_scale1_score": 0.08855364552799247, 
            "VMAF_feature_vif_scale2_score": 0.10226445090778727, 
            "VMAF_feature_vif_scale3_score": 0.10459395357252557, 
            "VMAF_score": 0.0
        }, 
        {
            "frameNum": 3150, 
            "VMAF_feature_adm2_score": 0.1914710687993693, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.04211828274754784, 
            "VMAF_feature_vif_scale1_score": 0.08896359219122792, 
            "VMAF_feature_vif_scale2_score": 0.10259623131730722, 
            "VMAF_feature_vif_scale3_score": 0.1043777529108215, 
            "VMAF_score": 0.0
        }, 
        {
            "frameNum": 3151, 
            "VMAF_feature_adm2_score": 0.1907573786299611, 
            "VMAF_feature_motion2_score": 0.000379, 
            "VMAF_feature_vif_scale0_score": 0.041465745476344525, 
            "VMAF_feature_vif_scale1_score": 0.08775040008985106, 
            "VMAF_feature_vif_scale2_score": 0.10113313232219975, 
            "VMAF_feature_vif_scale3_score": 0.10239778087801354, 
            "VMAF_score": 0.0
        }, 
        {
            "frameNum": 3152, 
            "VMAF_feature_adm2_score": 0.19076707800293727, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.04144712716460506, 
            "VMAF_feature_vif_scale1_score": 0.08775434542624778, 
            "VMAF_feature_vif_scale2_score": 0.1011435197553379, 
            "VMAF_feature_vif_scale3_score": 0.10241793897120459, 
            "VMAF_score": 0.0
        }, 
        {
            "frameNum": 3153, 
            "VMAF_feature_adm2_score": 0.19137527176270244, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.04007674001856412, 
            "VMAF_feature_vif_scale1_score": 0.08486254121296907, 
            "VMAF_feature_vif_scale2_score": 0.09723600047904274, 
            "VMAF_feature_vif_scale3_score": 0.0970543968664229, 
            "VMAF_score": 0.0
        }, 
        {
            "frameNum": 3154, 
            "VMAF_feature_adm2_score": 0.18194278306189662, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.03926503200457012, 
            "VMAF_feature_vif_scale1_score": 0.08368120872642014, 
            "VMAF_feature_vif_scale2_score": 0.09566259901336759, 
            "VMAF_feature_vif_scale3_score": 0.09500753193635625, 
            "VMAF_score": 0.0
        }, 
        {
            "frameNum": 3155, 
            "VMAF_feature_adm2_score": 0.18207186988752105, 
            "VMAF_feature_motion2_score": 0.00188, 
            "VMAF_feature_vif_scale0_score": 0.040422883990877456, 
            "VMAF_feature_vif_scale1_score": 0.08505649637560102, 
            "VMAF_feature_vif_scale2_score": 0.09738503430924197, 
            "VMAF_feature_vif_scale3_score": 0.096331832964267, 
            "VMAF_score": 0.0
        }, 
        {
            "frameNum": 3156, 
            "VMAF_feature_adm2_score": 0.1800941061800006, 
            "VMAF_feature_motion2_score": 0.00188, 
            "VMAF_feature_vif_scale0_score": 0.04219385620393975, 
            "VMAF_feature_vif_scale1_score": 0.08726038599806475, 
            "VMAF_feature_vif_scale2_score": 0.10007589482740874, 
            "VMAF_feature_vif_scale3_score": 0.10042091310260892, 
            "VMAF_score": 0.0
        }
    ], 
    "aggregate": {
        "VMAF_feature_adm2_score": 0.47931637531728954, 
        "VMAF_feature_motion2_score": 7.6866576050047515, 
        "VMAF_feature_vif_scale0_score": 0.11513697124678288, 
        "VMAF_feature_vif_scale1_score": 0.16862615752710952, 
        "VMAF_feature_vif_scale2_score": 0.1866348668537868, 
        "VMAF_feature_vif_scale3_score": 0.19312532668741608, 
        "VMAF_score": 9.83583753845461, 
        "method": "mean"
    }

This is super weird. If anyone has gotten this error before, please let me know. This is the result of a comparison between the AWS elastic transcoder and it's source.
When I transcode the source video using ffmpeg with exactly the same settings as the AWS elastic transcode , I get this which result which is closer to what I'm expecting for the AWS elastic transcoder.

        {
            "frameNum": 3149, 
            "VMAF_feature_adm2_score": 1.0128811514641758, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.7720661307371304, 
            "VMAF_feature_vif_scale1_score": 0.9810677514600266, 
            "VMAF_feature_vif_scale2_score": 0.9934652376657016, 
            "VMAF_feature_vif_scale3_score": 0.9973464945079024, 
            "VMAF_score": 99.64428919878337
        }, 
        {
            "frameNum": 3150, 
            "VMAF_feature_adm2_score": 1.0128800895873187, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.7725760590946087, 
            "VMAF_feature_vif_scale1_score": 0.9811771366777354, 
            "VMAF_feature_vif_scale2_score": 0.9934907720962023, 
            "VMAF_feature_vif_scale3_score": 0.9973505692210137, 
            "VMAF_score": 99.64573605045972
        }, 
        {
            "frameNum": 3151, 
            "VMAF_feature_adm2_score": 1.0128727440083964, 
            "VMAF_feature_motion2_score": 0.000345, 
            "VMAF_feature_vif_scale0_score": 0.7720576309200105, 
            "VMAF_feature_vif_scale1_score": 0.9810409469981245, 
            "VMAF_feature_vif_scale2_score": 0.9934206356769336, 
            "VMAF_feature_vif_scale3_score": 0.9972995810097431, 
            "VMAF_score": 99.63703341692192
        }, 
        {
            "frameNum": 3152, 
            "VMAF_feature_adm2_score": 1.0128815301284628, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.7727966527447488, 
            "VMAF_feature_vif_scale1_score": 0.9812253762177078, 
            "VMAF_feature_vif_scale2_score": 0.9934841401397166, 
            "VMAF_feature_vif_scale3_score": 0.9973250224798983, 
            "VMAF_score": 99.64379803049617
        }, 
        {
            "frameNum": 3153, 
            "VMAF_feature_adm2_score": 1.01288146780093, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.7727568230434121, 
            "VMAF_feature_vif_scale1_score": 0.9812171104962774, 
            "VMAF_feature_vif_scale2_score": 0.9934816324176424, 
            "VMAF_feature_vif_scale3_score": 0.9973248022496637, 
            "VMAF_score": 99.64363944360852
        }, 
        {
            "frameNum": 3154, 
            "VMAF_feature_adm2_score": 1.0128813800806988, 
            "VMAF_feature_motion2_score": 0.0, 
            "VMAF_feature_vif_scale0_score": 0.772749534618683, 
            "VMAF_feature_vif_scale1_score": 0.9812174501834594, 
            "VMAF_feature_vif_scale2_score": 0.9934831142534135, 
            "VMAF_feature_vif_scale3_score": 0.9973272249232007, 
            "VMAF_score": 99.64388729285774
        }, 
        {
            "frameNum": 3155, 
            "VMAF_feature_adm2_score": 1.0128957377127137, 
            "VMAF_feature_motion2_score": 0.001834, 
            "VMAF_feature_vif_scale0_score": 0.7727156311356612, 
            "VMAF_feature_vif_scale1_score": 0.9812366417607263, 
            "VMAF_feature_vif_scale2_score": 0.9934923531644574, 
            "VMAF_feature_vif_scale3_score": 0.9973343893148658, 
            "VMAF_score": 99.6505785504818
        }, 
        {
            "frameNum": 3156, 
            "VMAF_feature_adm2_score": 1.0129057792169165, 
            "VMAF_feature_motion2_score": 0.001834, 
            "VMAF_feature_vif_scale0_score": 0.7727289710781209, 
            "VMAF_feature_vif_scale1_score": 0.981253638178074, 
            "VMAF_feature_vif_scale2_score": 0.9934977468833491, 
            "VMAF_feature_vif_scale3_score": 0.9973351748479444, 
            "VMAF_score": 99.65314555031664
        }
    ], 
    "aggregate": {
        "VMAF_feature_adm2_score": 0.9865332602629746, 
        "VMAF_feature_motion2_score": 7.649734763066202, 
        "VMAF_feature_vif_scale0_score": 0.7232064448804095, 
        "VMAF_feature_vif_scale1_score": 0.9471186673991454, 
        "VMAF_feature_vif_scale2_score": 0.9730892576880159, 
        "VMAF_feature_vif_scale3_score": 0.9841267786144647, 
        "VMAF_score": 97.80357431902264, 
        "method": "mean"
    }

While I was browsing through the code I could see how compute_psnr works:

// function signature
int compute_psnr(const number_t *ref, const number_t *dis, int w, int h, int ref_stride, int dis_stride, double *score, double peak, double psnr_max)

// function usage
ret = compute_psnr(ref_buf, dis_buf, w, h, stride, stride, &score, 255.0, 60.0);

Since ret is always zero, I was wondering if we could change this function to return the score itself or is it a code convention?

I found that these files were referring to compute_psnr function.

feature/src/psnr.c:int compute_psnr(const number_t *ref, const number_t *dis, int w, int h, int ref_stride, int dis_stride, double *score, double peak, double psnr_max)
feature/src/psnr.c:            ret = compute_psnr(ref_buf, dis_buf, w, h, stride, stride, &score, 255.0, 60.0);
feature/src/psnr.c:            ret = compute_psnr(ref_buf, dis_buf, w, h, stride, stride, &score, 255.75, 72.0);
feature/src/psnr.c:            printf("error: compute_psnr failed.\n");
wrapper/src/combo.c:int compute_psnr(const float *ref, const float *dis, int w, int h, int ref_stride, int dis_stride, double *score, double peak, double psnr_max);
wrapper/src/combo.c:                ret = compute_psnr(ref_buf, dis_buf, w, h, stride, stride, &score, 255.0, 60.0);
wrapper/src/combo.c:                ret = compute_psnr(ref_buf, dis_buf, w, h, stride, stride, &score, 255.75, 72.0);
wrapper/src/combo.c:                sprintf(errmsg, "compute_psnr failed.\n");

Trying to open the generated json data after batch processing

Using:
./run_vmaf_in_batch run.batch --out-fmt json --parallelize > data.json

In [6]:  with open('data.json') as json_data:
   ...:     d = json.load(json_data)
   ...:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-889de346a784> in <module>()
      1 with open('data.json') as json_data:
----> 2    d = json.load(json_data)
      3

/home/fredpl/anaconda2/lib/python2.7/json/__init__.pyc in load(fp, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    289         parse_float=parse_float, parse_int=parse_int,
    290         parse_constant=parse_constant, object_pairs_hook=object_pairs_hook,
--> 291         **kw)
    292
    293

/home/fredpl/anaconda2/lib/python2.7/json/__init__.pyc in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    337             parse_int is None and parse_float is None and
    338             parse_constant is None and object_pairs_hook is None and not kw):
--> 339         return _default_decoder.decode(s)
    340     if cls is None:
    341         cls = JSONDecoder

/home/fredpl/anaconda2/lib/python2.7/json/decoder.pyc in decode(self, s, _w)
    365         end = _w(s, end).end()
    366         if end != len(s):
--> 367             raise ValueError(errmsg("Extra data", s, end, len(s)))
    368         return obj
    369

ValueError: Extra data: line 6729 column 1 - line 262213 column 1 (char 310201 - 12282608)

Is ansnr in the source code equal to PSNR-HVS referred in blog? Also is ADM equal to DLM method?

Seems like both ansnr and motion feature is not used in VmafFeatureExtractor. From blog, seems like motion should be used. right?

Seems like the more metrics, the more accurate result may get. If that is true, do you have any plan to add more metrics? for example http://vq.kt.agh.edu.pl//metrics.html has 15 more metrics.

We're attempting to compare VMAF's result across multiple bitrate encodes for a single asset. The original asset is a 29.97fps, 1080 MPEG-2 file and close to CBR (13.5 mbps average bitrate). We first transcode it to an MP4 lossless "reference" using Handbrake. The resultant file is VBR, 720, 24fps, and with an average bitrate of 17.5 mbps. We then use Handbrake to transcode the reference file to 720, 24fps versions from 1 -> 17 mbps. Lastly we use VMAF to grade these assets against the MP4 reference we created using the following command:

./run_vmaf yuv420p 1280 720 [source] [compare target] --out-fmt json

Here is a table of the results we obtained:

avg, mbps vmaf
1 18.9453
2 19.2293
3 19.3395
4 19.3839
5 19.4464
6 19.4368
7 19.4528
8 19.4759
9 19.4794
10 19.4921
11 19.4942
12 19.5033
13 19.5175
14 19.5053
15 19.515
16 19.5202
17 19.5212
reference 99.9911

We had expected to see results in the 20s from 1->3 mbps, with values scaling up towards 99 as bitrate increased. File size and visual inspection would indicate more of a change in encoded quality than VMAF is indicating.

Any suggestions on what we're doing wrong, or where our process is incorrect?

One last thing to mention: the length of the asset is 4316 frames, or ~3 minutes. Wanted to mention this in case there is an issue with VMAF and relatively short video lengths compared to your use case.

Hello!
I read the FAQ note describing why a 480p video has much higher VMAF scores. What you're saying is VMAF treats the 1080p display at 3H viewing distance as the standard for computing a quality score. Assuming display size directly proportional to display resolution, if we now use a 480p display, the quality scores should be computed with a viewing distance that is closer, not farther away, but preserving the same 3H ratio (i.e. 3 * 480/1080 = 1.33H). This is reflective of the fact that as screen sizes get smaller, we need to sit closer, not farther away.

Completely agreed with what is described in the two bullet points.

Adding y4m support is ending up a complex undertaking because there is a lot of duplicate code in both .c and _main.c files for reading yuv files and parsing the CLI. It would be a lot nicer to refactor all of this duplicate code into one place.

One place to start would be to drop the individual binaries and always use the main vmaf binary. Are the individual ones needed for anything?

I get an error when execute ./unittest:

**AssertionError: Process returned 132, cmd: /home/ubuntu/vmaf/feature/ssim yuv420p /home/ubuntu/vmaf/python/test/resource/yuv/src01_hrc00_576x324.yuv /home/ubuntu/vmaf/python/test/resource/yuv/src01_hrc00_576x324.yuv 576 324 >> /home/ubuntu/vmaf/workspace/workdir/1912e6a0-5fdd-4824-a5ad-d8407b3e5eb4/SSIM_feature_V1.1_522f1dc24a70f694b7b4f7cdbdae1eb3ce2c6b0e
Illegal instruction
Process Process-100:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, self._kwargs)
File "/home/ubuntu/vmaf/python/src/vmaf/tools/misc.py", line 258, in func_wrapper
executor = func(args)
File "/home/ubuntu/vmaf/python/src/vmaf/core/executor.py", line 114, in _run
result = self._run_on_asset(asset)
File "/home/ubuntu/vmaf/python/src/vmaf/core/quality_runner.py", line 574, in _run_on_asset
vmaf_fassembler.run()
File "/home/ubuntu/vmaf/python/src/vmaf/core/feature_assembler.py", line 82, in run
runner.run(parallelize=self.parallelize)
File "/home/ubuntu/vmaf/python/src/vmaf/core/executor.py", line 120, in run
self.results = map(self._run_on_asset, self.assets)
File "/home/ubuntu/vmaf/python/src/vmaf/core/executor.py", line 286, in _run_on_asset
self._generate_result(asset)
File "/home/ubuntu/vmaf/python/src/vmaf/core/feature_extractor.py", line 473, in _generate_result
run_process(ssim_cmd, shell=True)
File "/home/ubuntu/vmaf/python/src/vmaf/tools/misc.py", line 374, in run_process
assert ret == 0, 'Process returned {ret}, cmd: {cmd}'.format(ret=ret, cmd=cmd)
AssertionError: Process returned 132, cmd: /home/ubuntu/vmaf/feature/ssim yuv420p /home/ubuntu/vmaf/python/test/resource/yuv/src01_hrc00_576x324.yuv /home/ubuntu/vmaf/python/test/resource/yuv/src01_hrc01_576x324.yuv 576 324 >> /home/ubuntu/vmaf/workspace/workdir/b41d1ce1-72bf-49f0-b669-3bf504d35617/SSIM_feature_V1.1_83453789ae2c5a4e39f5cce9d1c7c8a1c09a62ac

Somebody knows what could be the problem?

Thanks in advance

I encoded all "4K" clips (provided by netflix) from xiph (http://media.xiph.org/video/derf/) after converting them to 360p 8bit 30fps. I used default 1-pass CRF settings at --preset=veryslow and the following additional settings:

[nothing]
--tune=psnr
--aq-mode=0
--no-psy
--psy-rd=0.0
--aq-mode=0 --no-psy

Then I scored the VMAF score and used a slightly modified version of Google's visual-metrics.py tool to calculate bitrate improvement (https://chromium.googlesource.com/webm/contributor-guide/+/master/scripts/).

The interesting thing in the results was that --tune=psnr improved the VMAF score by about 5%. That shouldn't happen. --tune=psnr basically does two things: it sets --no-psy (which is nearly identical to --psy-rd=0.0, i.e. it tunes energy retention in the rate distortion loop) and it disables adaptive quant (--aq-mode=0). Results for --aq-mode=0 --no-psy and --tune=psnr are identical, which demonstrates this point. Additionally, results for --psy-rd=0.0 and --no-psy are nearly identical.

When looking at the results for only --no-psy/--psy-rd=0.0 or --aq-mode=0, I see very big differences, though. --aq-mode=0 increases VMAF score by about 9% (bitrate improvement), whereas --no-psy/--psy-rd=0.0 decrease VMAF score by about 9%. In other words, AQ hurts VMAF but psy-rd helps VMAF. This is strange. AQ is supposed to help visual quality, right?

Image with results from libvpx' contributors-guide script (baseline is x264 --tune=psnr; "vis" means I didn't use --tune=psnr, i.e. visual tuning; aq0 means I used --aq-mode=0; nopsy means I used --no-psy; psyrd0 means I used --psy-rd=0.0):

I've done (unrelated) experiments in my own VP9 encoder (http://www.twoorioles.com/eve-vp9-video-encoder) and have seen similar issues, where AQ (e.g. based on better quality for dark/low-contrast portions of frames) hurts VMAF scores by quite a bit, which I found surprising. I haven't done a lot of analysis on this issue yet and am planning to do more, but figured a bug report might be worth it, maybe you guys have more insight in this and can help me get started trying to address this. I've also done experiments where I used the same code to assign different baseline quantizers based on whole-frame dark/low-contrast scores, and that helped VMAF score as expected. This makes me wonder if certain settings (like contrast, motion, or other things) are per-frame instead of per-block/section values in VMAF, which would mean AQ based on these attributes doesn't increase VMAF score even though it improves visual quality (IMO). (I know, you probably want samples for this, I'll try to make something that's scientifically sound.)

Such as:

• run_vmaf
• run_cleaning_cache
• run_vmaf_training
• run_vmaf_in_batch
• run_testing
• run_psnr
• vmafossexec

FAIL: test_run_kfold_cross_validation_with_list_input (cross_validation_test.FeatureCrossValidationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thomas/sandbox/vmaf/python/test/cross_validation_test.py", line 92, in test_run_kfold_cross_validation_with_list_input
    self.assertAlmostEquals(output['aggr_stats']['SRCC'], 0.90636761259756715, places=4)
AssertionError: 0.90647090523758211 != 0.9063676125975672 within 4 places

The value is off just a little too much for places=4. How much variance is expected here?

Just throwing this out there to see if anybody has been successful in getting this to run in CentOS. I have been attempting in CentOS 6.9, using a virtualenv to maintain Python 2.7 as requirements state. I initially tried through docker and have been unsuccessful in my attempts.
I plan to attempt getting this to run on CentOS 7, but I wanted to throw something out here to see if anyone has evidence or recommendations to go a different route than CentOS.
Any help or advice would be great, thanks!

EDIT: Successful in running vmaf through docker. CentOS 6.9 -> Virtualenv (w/ Python 2.7.13) -> docker -> vmaf. It works!

Hello,

When we launch the run_testing script for VMAF (./run_testing VMAF dataset.py), we get the following error with new version 1.1.16:
Error: Must have one and only one subclass of QualityRunner with type VMAF, but got 2

It was working fine with the version 1.1.13.

Do you know what could be the problem ?

Thanks!

I have been testing a short video which is a series of still images with medium motion between them (unfortunately I cannot provide it due to copyright) and I noticed perhaps motion is weighted too highly? This is the output of VMAF for the video:

Frame 0: VMAF_feature_adm2_score:0.983355, VMAF_feature_motion_score:0.000000, VMAF_feature_vif_scale0_score:0.797342, VMAF_feature_vif_scale1_score:0.972974, VMAF_feature_vif_scale2_score:0.987024, VMAF_feature_vif_scale3_score:0.992817, VMAF_score:93.100445
Frame 1: VMAF_feature_adm2_score:0.985505, VMAF_feature_motion_score:37.661541, VMAF_feature_vif_scale0_score:0.797752, VMAF_feature_vif_scale1_score:0.972760, VMAF_feature_vif_scale2_score:0.986680, VMAF_feature_vif_scale3_score:0.992413, VMAF_score:100.000000
Frame 2: VMAF_feature_adm2_score:0.976141, VMAF_feature_motion_score:40.074627, VMAF_feature_vif_scale0_score:0.708371, VMAF_feature_vif_scale1_score:0.945214, VMAF_feature_vif_scale2_score:0.972793, VMAF_feature_vif_scale3_score:0.984663, VMAF_score:100.000000
Frame 3: VMAF_feature_adm2_score:0.974181, VMAF_feature_motion_score:42.222538, VMAF_feature_vif_scale0_score:0.672937, VMAF_feature_vif_scale1_score:0.947456, VMAF_feature_vif_scale2_score:0.975278, VMAF_feature_vif_scale3_score:0.985465, VMAF_score:100.000000
Frame 4: VMAF_feature_adm2_score:0.978275, VMAF_feature_motion_score:39.471184, VMAF_feature_vif_scale0_score:0.713338, VMAF_feature_vif_scale1_score:0.947480, VMAF_feature_vif_scale2_score:0.973790, VMAF_feature_vif_scale3_score:0.983780, VMAF_score:100.000000
Frame 5: VMAF_feature_adm2_score:0.976263, VMAF_feature_motion_score:40.529591, VMAF_feature_vif_scale0_score:0.692000, VMAF_feature_vif_scale1_score:0.956538, VMAF_feature_vif_scale2_score:0.980027, VMAF_feature_vif_scale3_score:0.989142, VMAF_score:100.000000
Frame 6: VMAF_feature_adm2_score:0.972154, VMAF_feature_motion_score:35.454933, VMAF_feature_vif_scale0_score:0.688905, VMAF_feature_vif_scale1_score:0.943008, VMAF_feature_vif_scale2_score:0.972188, VMAF_feature_vif_scale3_score:0.984484, VMAF_score:100.000000
Frame 7: VMAF_feature_adm2_score:0.972674, VMAF_feature_motion_score:40.848309, VMAF_feature_vif_scale0_score:0.679250, VMAF_feature_vif_scale1_score:0.950740, VMAF_feature_vif_scale2_score:0.976935, VMAF_feature_vif_scale3_score:0.987387, VMAF_score:100.000000
Frame 8: VMAF_feature_adm2_score:0.969504, VMAF_feature_motion_score:38.406681, VMAF_feature_vif_scale0_score:0.642949, VMAF_feature_vif_scale1_score:0.931827, VMAF_feature_vif_scale2_score:0.967498, VMAF_feature_vif_scale3_score:0.982788, VMAF_score:100.000000
Aggregate: VMAF_feature_adm2_score:0.976450, VMAF_feature_motion_score:34.963267, VMAF_feature_vif_scale0_score:0.710316, VMAF_feature_vif_scale1_score:0.952000, VMAF_feature_vif_scale2_score:0.976913, VMAF_feature_vif_scale3_score:0.986993, VMAF_score:99.233383

As you can see the first frame with 0 motion has a more accurate score (93), but the following frames which have a motion score of above 30 (I noticed the README mentioned a normal range is 20) are score as 100? The other elementary metrics are very similar to each other. I'm not sure of the model, but could the 100 scores be due to the high motion? I have inspected the frames and I believe they should be closer to the high 80's, low 90s.

"aggregate": {
"VMAF_feature_adm2_score": 0.93458780728708746,
"VMAF_feature_motion2_score": 3.8953518541666665,
"VMAF_feature_vif_scale0_score": 0.36342081156994926,
"VMAF_feature_vif_scale1_score": 0.76664738784617292,
"VMAF_feature_vif_scale2_score": 0.86285338927816291,
"VMAF_feature_vif_scale3_score": 0.91597186913930484,
"VMAF_score": 76.699271272486044,
"method": "mean"
}

What is the range for VMAF_score ? .I am particularly interested to know the best , average and worst range

Hi,
I have noticed in multiple cases that when comparing the reference file with itself the score is not 100. Sometimes it can be 99.99 but in other cases, say, 98.7, or 99.2, while SSIM in all cases gives exactly 1. So shouldn't VMAF give exactly 100 in those cases? If not, is that related to the perceptual component in the data fusion? And if so, would that be some rough level of uncertainty between VMAF and DMOS? (say, 1% for what should be absolutely lossless and some other number for other resolutions). Is this a known feature? If so, are there details between statistical deviations between VMAF and DMOS for the training sets used as provided? Any thoughts on this?

Btw, vmaf_feature_motion seems to be causing this.

Hi,
I get an import error when run ./run_vmaf:

Traceback (most recent call last):
File "./run_vmaf", line 8, in
from vmaf.config import VmafConfig
ImportError: No module named vmaf.config

What could be the problem?

Thanks

I pull the commit '876d844', and make under ./vmaf , I got the error msg as:

cd ptools; /Applications/Xcode.app/Contents/Developer/usr/bin/make libptools.so; cd ..;
g++ -Wall -Wextra -fpic -O -fno-strict-aliasing -DLINUX_ -DOC_NEW_STYLE_INCLUDES -Wno-deprecated -I./opencontainers_1_8_4/include -pthread -D_REENTRANT -c m2pythontools.cc
In file included from m2pythontools.cc:2:
In file included from ./m2pythontools.h:8:
In file included from ./opencontainers_1_8_4/include/ocarray.h:28:
In file included from ./opencontainers_1_8_4/include/ocstreamingpool.h:23:
./opencontainers_1_8_4/include/ocsynchronizer.h:96:1: warning: 'CondVar' defined as a struct here but previously declared as a class [-Wmismatched-tags]
struct CondVar {
^
./opencontainers_1_8_4/include/ocsynchronizer.h:20:1: note: did you mean struct here?
class CondVar; // Forward
^~~~~
struct
./opencontainers_1_8_4/include/ocsynchronizer.h:204:21: error: use of undeclared identifier 'CLOCK_REALTIME'
clock_gettime(CLOCK_REALTIME, &current_time);
^
In file included from m2pythontools.cc:2:
In file included from ./m2pythontools.h:8:
In file included from ./opencontainers_1_8_4/include/ocarray.h:27:
./opencontainers_1_8_4/include/ocport.h:793:20: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
INT z = INT(-1)<<((sizeof(INT)<<3)-1); // Largest negative int
~~~~~~~^
./opencontainers_1_8_4/include/ocport.h:814:60: note: in instantiation of function template specialization 'OC::StringizeInt' requested here
inline string Stringize<int_2> (const int_2& n) { return StringizeInt(n); }
^
./opencontainers_1_8_4/include/ocport.h:793:20: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
INT z = INT(-1)<<((sizeof(INT)<<3)-1); // Largest negative int
~~~~~~~^
./opencontainers_1_8_4/include/ocport.h:816:60: note: in instantiation of function template specialization 'OC::StringizeInt' requested here
inline string Stringize<int_4> (const int_4& n) { return StringizeInt(n); }
^
./opencontainers_1_8_4/include/ocport.h:793:20: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
INT z = INT(-1)<<((sizeof(INT)<<3)-1); // Largest negative int
~~~~~~~^
./opencontainers_1_8_4/include/ocport.h:818:60: note: in instantiation of function template specialization 'OC::StringizeInt' requested here
inline string Stringize<int_8> (const int_8& n) { return StringizeInt(n); }
^
4 warnings and 1 error generated.
make[1]: *** [m2pythontools.o] Error 1
for dir in feature libsvm wrapper; do
cd ${dir}; /Applications/Xcode.app/Contents/Developer/usr/bin/make; cd ..;
done
make[1]: Nothing to be done for all'. make[1]: Nothing to be done forall'.
c++ -c -o /Users/abc/Netflix_VMAF/vmaf/wrapper/obj/vmaf.o -std=c++11 -g -O3 -fPIC -w -Wextra -pedantic
-DOC_NEW_STYLE_INCLUDES -I /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools
-I /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/opencontainers_1_8_4/include /Users/abc/Netflix_VMAF/vmaf/wrapper/src/vmaf.cpp
In file included from /Users/abc/Netflix_VMAF/vmaf/wrapper/src/vmaf.cpp:34:
In file included from /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/chooseser.h:40:
In file included from /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/valpython.h:6:
In file included from /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/opencontainers_1_8_4/include/ocval.h:38:
In file included from /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/opencontainers_1_8_4/include/ocavlhasht.h:38:
In file included from /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/opencontainers_1_8_4/include/ocarray.h:28:
In file included from /Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/opencontainers_1_8_4/include/ocstreamingpool.h:23:
/Users/abc/Netflix_VMAF/vmaf/wrapper/../ptools/opencontainers_1_8_4/include/ocsynchronizer.h:204:21: error: use of undeclared identifier 'CLOCK_REALTIME'
clock_gettime(CLOCK_REALTIME, &current_time);
^
/Users/abc/Netflix_VMAF/vmaf/wrapper/src/vmaf.cpp:696:16: error: use of undeclared identifier 'pow10'
size_t m = pow10(digit);
^
2 errors generated.
make[1]: *** [/Users/abc/Netflix_VMAF/vmaf/wrapper/obj/vmaf.o] Error 1
cd libsvm; /Applications/Xcode.app/Contents/Developer/usr/bin/make lib; cd ..;
if [ "Darwin" = "Darwin" ]; then
SHARED_LIB_FLAG="-dynamiclib -Wl,-install_name,libsvm.so.2";
else
SHARED_LIB_FLAG="-shared -Wl,-soname,libsvm.so.2";
fi;
c++ ${SHARED_LIB_FLAG} svm.o -o libsvm.so.2

I would like to use the cross validation, but there is no command ./run_vmaf_cv available. Is this a future feature or am I doing something wrong?

Hi，
Now I have succeed buiding the project.The version of the python and packages are:

numpy:1.11.0
scipy:0.13.3
matplotlib:1.3.1
pandas:0.13.1
sklearn:0.18.0

some packages can not update to newest version.

now I run :./unittest got 3 errs:

ERROR: test_run_parallel_vamf_runner_with_model (quality_runner_test.ParallelQualityRunnerTest)

Traceback (most recent call last):
File "/home/xbf/Desktop/vmaf-master/python/test/quality_runner_test.py", line 534, in test_run_parallel_vamf_runner_with_model
'model_filepath':config.ROOT + "/resource/model/nflx_vmaff_rf_v1.pkl",
File "/home/xbf/Desktop/vmaf-master/python/core/executor.py", line 415, in run_executors_in_parallel
results = [executor.results[0] for executor in executors]
AttributeError: 'NoneType' object has no attribute 'results'

ERROR: test_run_vmaf_runner_with_rf_model (quality_runner_test.QualityRunnerTest)

Traceback (most recent call last):
File "/home/xbf/Desktop/vmaf-master/python/test/quality_runner_test.py", line 393, in test_run_vmaf_runner_with_rf_model
self.runner.run()
File "/home/xbf/Desktop/vmaf-master/python/core/executor.py", line 63, in run
self.results = map(self._run_on_asset, self.assets)
File "/home/xbf/Desktop/vmaf-master/python/core/quality_runner.py", line 276, in _run_on_asset
vmaf_fassembler = self._get_vmaf_feature_assembler_instance(asset)
File "/home/xbf/Desktop/vmaf-master/python/core/quality_runner.py", line 253, in _get_vmaf_feature_assembler_instance
model = self._load_model()
File "/home/xbf/Desktop/vmaf-master/python/core/quality_runner.py", line 356, in _load_model
model = TrainTestModel.from_file(model_filepath, self.logger)
File "/home/xbf/Desktop/vmaf-master/python/core/train_test_model.py", line 144, in from_file
info_loaded = pickle.load(file)
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey
File "/usr/lib/python2.7/pickle.py", line 1217, in load_build
setstate(state)
File "sklearn/tree/_tree.pyx", line 629, in sklearn.tree._tree.Tree.setstate (sklearn/tree/_tree.c:7698)
KeyError: 'max_depth'

ERROR: test_run_vmaf_runner_with_rf_model (quality_runner_test.QualityRunnerTest)

Traceback (most recent call last):
File "/home/xbf/Desktop/vmaf-master/python/test/quality_runner_test.py", line 19, in tearDown
self.runner.remove_results()
File "/home/xbf/Desktop/vmaf-master/python/core/executor.py", line 72, in remove_results
self._remove_result(asset)
File "/home/xbf/Desktop/vmaf-master/python/core/quality_runner.py", line 363, in _remove_result
vmaf_fassembler = self._get_vmaf_feature_assembler_instance(asset)
File "/home/xbf/Desktop/vmaf-master/python/core/quality_runner.py", line 253, in _get_vmaf_feature_assembler_instance
model = self._load_model()
File "/home/xbf/Desktop/vmaf-master/python/core/quality_runner.py", line 356, in _load_model
model = TrainTestModel.from_file(model_filepath, self.logger)
File "/home/xbf/Desktop/vmaf-master/python/core/train_test_model.py", line 144, in from_file
info_loaded = pickle.load(file)
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey
File "/usr/lib/python2.7/pickle.py", line 1217, in load_build
setstate(state)
File "sklearn/tree/_tree.pyx", line 629, in sklearn.tree._tree.Tree.setstate (sklearn/tree/_tree.c:7698)
KeyError: 'max_depth'

Ran 69 tests in 147.923s

FAILED (errors=3)

ps.python version：2.7.6

Why and how can I solve this problems. I almost go creazy！ Thx！

Hi,
is it possible to show a progressbar or another status for a single vmaf process? It will be long to finshed one vmaf computation on my computer.

I cannot make the project on Ubuntu (gcc (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4).

I get always the same error:

make[1]: Entering directory /home/ubuntu/vmaf/ptools' g++ -pthread -Wall -Wextra -fpic -O -fno-strict-aliasing -DLINUX_ -DOC_NEW_STYLE_INCLUDES -Wno-deprecated -I./opencontainers_1_8_4/include -pthread -D_REENTRANT m2pythontools.o valpython.o midassocket.o valprotocol2.o m2ser.o m2streamdataenc.o m2convertrep.o timeconv.o -shared -o libptools.so make[1]: Leaving directory /home/ubuntu/vmaf/ptools'
make[1]: Entering directory /home/ubuntu/vmaf/feature' cc -c -o /home/ubuntu/vmaf/feature/obj/adm_tools.o -std=c99 -g -O3 -fPIC -w -Wextra -pedantic /home/ubuntu/vmaf/feature/src/adm_tools.c In file included from /usr/lib/gcc/i686-linux-gnu/4.9/include/emmintrin.h:31:0, from /home/ubuntu/vmaf/feature/src/adm_tools.c:32: /home/ubuntu/vmaf/feature/src/adm_tools.c: In function ârcp_sâ: /usr/lib/gcc/i686-linux-gnu/4.9/include/xmmintrin.h:902:1: error: inlining failed in call to always_inline â_mm_load_ss.isra.0â: target specific option mismatch _mm_load_ss (float const *__P) ^ /home/ubuntu/vmaf/feature/src/adm_tools.c:36:16: error: called from here float xi = _mm_cvtss_f32(_mm_rcp_ss(_mm_load_ss(&x))); ^ In file included from /usr/lib/gcc/i686-linux-gnu/4.9/include/emmintrin.h:31:0, from /home/ubuntu/vmaf/feature/src/adm_tools.c:32: /usr/lib/gcc/i686-linux-gnu/4.9/include/xmmintrin.h:155:1: error: inlining failed in call to always_inline â_mm_rcp_ssâ: target specific option mismatch _mm_rcp_ss (__m128 __A) ^ /home/ubuntu/vmaf/feature/src/adm_tools.c:36:16: error: called from here float xi = _mm_cvtss_f32(_mm_rcp_ss(_mm_load_ss(&x))); ^ In file included from /usr/lib/gcc/i686-linux-gnu/4.9/include/emmintrin.h:31:0, from /home/ubuntu/vmaf/feature/src/adm_tools.c:32: /usr/lib/gcc/i686-linux-gnu/4.9/include/xmmintrin.h:964:1: error: inlining failed in call to always_inline â_mm_cvtss_f32â: target specific option mismatch _mm_cvtss_f32 (__m128 __A) ^ /home/ubuntu/vmaf/feature/src/adm_tools.c:36:11: error: called from here float xi = _mm_cvtss_f32(_mm_rcp_ss(_mm_load_ss(&x))); ^ make[1]: *** [/home/ubuntu/vmaf/feature/obj/adm_tools.o] Error 1 make[1]: Leaving directory /home/ubuntu/vmaf/feature'
make[1]: Entering directory /home/ubuntu/vmaf/libsvm' make[1]: Nothing to be done for all'.
make[1]: Leaving directory /home/ubuntu/vmaf/libsvm' make[1]: Entering directory /home/ubuntu/vmaf/wrapper'
cc -c -o /home/ubuntu/vmaf/wrapper/obj/adm_tools.o -std=c99 -g -O3 -fPIC -w -Wextra -pedantic /home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c
In file included from /usr/lib/gcc/i686-linux-gnu/4.9/include/emmintrin.h:31:0,
from /home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c:32:
/home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c: In function ârcp_sâ:
/usr/lib/gcc/i686-linux-gnu/4.9/include/xmmintrin.h:902:1: error: inlining failed in call to always_inline â_mm_load_ss.isra.0â: target specific option mismatch
_mm_load_ss (float const *__P)
^
/home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c:36:16: error: called from here
float xi = _mm_cvtss_f32(_mm_rcp_ss(_mm_load_ss(&x)));
^
In file included from /usr/lib/gcc/i686-linux-gnu/4.9/include/emmintrin.h:31:0,
from /home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c:32:
/usr/lib/gcc/i686-linux-gnu/4.9/include/xmmintrin.h:155:1: error: inlining failed in call to always_inline â_mm_rcp_ssâ: target specific option mismatch
_mm_rcp_ss (__m128 __A)
^
/home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c:36:16: error: called from here
float xi = _mm_cvtss_f32(_mm_rcp_ss(_mm_load_ss(&x)));
^
In file included from /usr/lib/gcc/i686-linux-gnu/4.9/include/emmintrin.h:31:0,
from /home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c:32:
/usr/lib/gcc/i686-linux-gnu/4.9/include/xmmintrin.h:964:1: error: inlining failed in call to always_inline â_mm_cvtss_f32â: target specific option mismatch
_mm_cvtss_f32 (__m128 __A)
^
/home/ubuntu/vmaf/wrapper/../feature/src/adm_tools.c:36:11: error: called from here
float xi = _mm_cvtss_f32(_mm_rcp_ss(_mm_load_ss(&x)));
^
make[1]: *** [/home/ubuntu/vmaf/wrapper/obj/adm_tools.o] Error 1
make[1]: Leaving directory /home/ubuntu/vmaf/wrapper' cd libsvm; make lib; cd ..; make[1]: Entering directory /home/ubuntu/vmaf/libsvm'
if [ "Linux" = "Darwin" ]; then
SHARED_LIB_FLAG="-dynamiclib -Wl,-install_name,libsvm.so.2";
else
SHARED_LIB_FLAG="-shared -Wl,-soname,libsvm.so.2";
fi;
g++ ${SHARED_LIB_FLAG} svm.o -o libsvm.so.2
make[1]: Leaving directory `/home/ubuntu/vmaf/libsvm'

Thanks

With some errors after 'make' (ticket #26), I tried ./unittest. I got the following errors:

Etest on running dis YUV raw video extractor...
.test on running dis YUV raw video extractor in parallel (disabled)...
.test on running asset extractor...
..........test on file system result store save and load...
.test on result aggregate scores...
.test on result to/from dataframe...
....Stats on training data: (SRCC: 0.949, PCC: 0.919, RMSE: 9.776)
..test extra trees train and predict...
.test libsvmnusvr train and predict...
.test random forest train and predict...
.test train, save, load and predict...
.test libsvmnusvr train, save, load and predict...

.......

ERROR: test_run_vmafossexec_runner (quality_runner_test.QualityRunnerTest)

Traceback (most recent call last):
File "/Users/abc/Netflix_VMAF/vmaf/python/test/quality_runner_test.py", line 510, in test_run_vmafossexec_runner
self.runner.run()
File "/Users/abc/Netflix_VMAF/vmaf/python/core/executor.py", line 81, in run
self.results = map(self._run_on_asset, self.assets)
File "/Users/abc/Netflix_VMAF/vmaf/python/core/executor.py", line 228, in _run_on_asset
result = self._read_result(asset)
File "/Users/abc/Netflix_VMAF/vmaf/python/core/quality_runner.py", line 60, in _read_result
result.update(self._get_quality_scores(asset))
File "/Users/abc/Netflix_VMAF/vmaf/python/core/quality_runner.py", line 416, in _get_quality_scores
tree = ElementTree.parse(log_file_path)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
tree.parse(source, parser)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse
parser.feed(data)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
self._raiseerror(v)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
ParseError: syntax error: line 1, column 0

ERROR: test_run_vmafossexec_runner_norm_type_none (quality_runner_test.QualityRunnerTest)

Traceback (most recent call last):
File "/Users/abc/Netflix_VMAF/vmaf/python/test/quality_runner_test.py", line 561, in test_run_vmafossexec_runner_norm_type_none
self.runner.run()
File "/Users/abc/Netflix_VMAF/vmaf/python/core/executor.py", line 81, in run
self.results = map(self._run_on_asset, self.assets)
File "/Users/abc/Netflix_VMAF/vmaf/python/core/executor.py", line 228, in _run_on_asset
result = self._read_result(asset)
File "/Users/abc/Netflix_VMAF/vmaf/python/core/quality_runner.py", line 60, in _read_result
result.update(self._get_quality_scores(asset))
File "/Users/abc/Netflix_VMAF/vmaf/python/core/quality_runner.py", line 416, in _get_quality_scores
tree = ElementTree.parse(log_file_path)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
tree.parse(source, parser)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse
parser.feed(data)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
self._raiseerror(v)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
ParseError: syntax error: line 1, column 0

ERROR: test_run_vmafossexec_runner_yuv422p10le (quality_runner_test.QualityRunnerTest)

Traceback (most recent call last):
File "/Users/abc/Netflix_VMAF/vmaf/python/test/quality_runner_test.py", line 611, in test_run_vmafossexec_runner_yuv422p10le
self.runner.run()
File "/Users/abc/Netflix_VMAF/vmaf/python/core/executor.py", line 81, in run
self.results = map(self._run_on_asset, self.assets)
File "/Users/abc/Netflix_VMAF/vmaf/python/core/executor.py", line 228, in _run_on_asset
result = self._read_result(asset)
File "/Users/abc/Netflix_VMAF/vmaf/python/core/quality_runner.py", line 60, in _read_result
result.update(self._get_quality_scores(asset))
File "/Users/abc/Netflix_VMAF/vmaf/python/core/quality_runner.py", line 416, in _get_quality_scores
tree = ElementTree.parse(log_file_path)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1182, in parse
tree.parse(source, parser)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 656, in parse
parser.feed(data)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1642, in feed
self._raiseerror(v)
File "/Users/abc/software/anaconda2/lib/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
raise err
ParseError: syntax error: line 1, column 0

Ran 104 tests in 215.684s

FAILED (errors=3)

Training usually takes longer time. Suggest to print start/end time in the code as well as total consuming time. It would be very helpful.

Hi guys,

I've been playing a bit with the software and I have few questions about it.

The first one, reading the your article, you made a comparison for live sources but I don't see exactly the way you can do it, as you need the sources frame aligned and also yuv files. (unless you record the live streams).

The second question is: mostly all the sources (live and VOD) are interlaced, and in the OTT world all the streams are progressive. What is the behavior of vmaf when compares these two files? Are you de-interlacing them previously the comparison?

If all the previous steps have to be done, must it be done by hand or is there an automated way already?

Thanks,

Hi,
if I want to compile this script with make, I get an error:

/usr/bin/ld: cannot find -lptools
collect2: error: ld returned 1 exit status
Makefile:116: recipe for target 'vmafossexec' failed
make[1]: *** [vmafossexec] Error 1

Which libary is missing on my linux machine?

Thank your for your help!

regards
Mathias

which tool do you use to convert video into .yuv file?

Models are being installed in $(PREFIX)/share directory after running make install.

This type of files are usually installed in $(PREFIX)/share/<name> directory.
Shouldn't it be installed in something else like $(PREFIX)/share/vmaf or $(PREFIX)/share/model?

======================================================================
ERROR: test_run_vamf_legacy_runner_10le (quality_runner_test.QualityRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/cvley/vmaf/python/test/quality_runner_test.py", line 94, in test_run_vamf_legacy_runner_10le
    self.runner.run()
  File "/home/cvley/vmaf/python/core/executor.py", line 62, in run
    self.results = map(self._run_on_asset, self.assets)
  File "/home/cvley/vmaf/python/core/quality_runner.py", line 172, in _run_on_asset
    vmaf_fassembler.run()
  File "/home/cvley/vmaf/python/core/feature_assembler.py", line 70, in run
    optional_dict=self.optional_dict,
  File "/home/cvley/vmaf/python/core/executor.py", line 406, in run_executors_in_parallel
    executors = map(run_executor, list_args)
  File "/home/cvley/vmaf/python/core/executor.py", line 381, in run_executor
    executor.run()
  File "/home/cvley/vmaf/python/core/executor.py", line 62, in run
    self.results = map(self._run_on_asset, self.assets)
  File "/home/cvley/vmaf/python/core/executor.py", line 188, in _run_on_asset
    self._prepare_log_file(asset)
  File "/home/cvley/vmaf/python/core/executor.py", line 115, in _prepare_log_file
    with open(log_file_path, 'wt') as log_file:
IOError: [Errno 36] File name too long: '/home/cvley/vmaf/workspace/workdir/664701be-de8b-46c3-a0fc-ba685bd9652e/VMAF_feature_V0.2.1_test_0_0_src01_hrc00_576x324.yuv422p10le_576x324_yuv422p10le_vs_src01_hrc01_576x324.yuv422p10le_576x324_yuv422p10le_q_576x324'

With some tests and google, I think it was caused by ecryptfs which file length maximum number is less than 144. Hope you can deal with it, thanks 😃

Hi Li，
I want to kown how to get the Video Database. Can I download the Video Database through Network？ Thx

Sometimes depending on content I will generate two points that hit the 99-100 score range, meaning visually lossless. This behavior is mostly fine, but problematic for generating BD-Rate because the cubic fit will have a negative slope. It's pretty frequent on AWCY on a couple of "easy" clips, see for example:

https://arewecompressedyet.com/?job=fix1_pvq_with_daala_dist_30f%402017-01-20T21%3A03%3A04.007Z&job=av1_pvq_30f_Jan12_try3%402017-01-18T09%3A14%3A59.302Z

I'd like to define a standard way to handle it. One obvious way is just to fudge the values such that they are always monotonically increasing.