powturbo / turbobench Goto Github PK
View Code? Open in Web Editor NEWCompression Benchmark
Compression Benchmark
I tried to print to cvs or markdown (with `-p7') but it didn't work. Actually any format makes it to exit without any action.
I tried to fix it but didn't get far. All I got to is that `printfile()' exits at if(!k) return;
as `plugread()' returns 0.
As part of my efforts to fuzz compression libraries, I recently tested pithy and found some vulnerabilities. The author has not responded to email about the issue for over a month, nor the pull request filed against the project, so AFAICT it's abandoned.
I'm not sure whether or not it is appropriate for this project to keep supporting pithy or not. It obviously shouldn't be used in production code, but turbobench isn't really intended for production code… I guess the distinction should be whether turbobench is intended to help people choose a compression library for their code (in which case it would be better not to include pithy), or as a tool for people writing compression libraries (in which case it would be better to include it).
Does TurboBench support the ZSTD dictionary compression performance test? I want to try to compare the performance of ZSTD compression use the dictionary with other algorithms in small data mode.
Would be interesting to see results:
https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/
Thanks!
I got Error message as follow when make project:
In file included from lzham_codec_devel/lzhamdecomp/lzham_core.h:296:
lzham_codec_devel/lzhamdecomp/lzham_platform.h:27:25: error: unrecognized instruction mnemonic
__asm__ __volatile__("pause");
^
<inline asm>:1:2: note: instantiated into assembly here
pause
^....
Then I modified the makefile, set "LZHAM=0" and try again:
Error message as follow:
Turbo-Range-Coder/include_/sse_neon.h:232:85: error: invalid conversion between vector type 'uint64x2_t' (vector of 2 'uint64_t' values) and 'uint8x8_t' (vector of 8 'uint8_t' values) of different size
static ALWAYS_INLINE uint64_t mm_movemask4_epu8(__m128i v) { return vgetq_lane_u64((uint64x2_t)vshrn_n_u16((uint8x16_t)v, 4), 0); } //uint8x16_t
Thanks Hamid for sharing your binaries.
A quick run on i7-3630QM @3.4GHz 16GB DDR3 @1600MHz:
F:\S>turbobench_build_19_Feb_2020.exe "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" -eOPTIMAL/lzo1b,999/bsc,0:e2/lzfse/zlib,9
C Size ratio% C MB/s D MB/s Name File
5315936 13.2 6.11 10.68 bsc 0:e2 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
5446644 8.06 ? bwtturbo -59 -t0 -b1024 "%1" "%1.zv"
6942920 17.2 1.28 120.86 lzma 9 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
6996677 17.4 0.92 278.81 lzham 4 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
7022275 17.4 0.36 312.06 brotli 11 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
7038628 17.5 1.40 703.90 zstd 22 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
8061821 20.0 1.40 815.14 lizard 49 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
8851573 1.38 ? BriefLZ_130_Intel_v19_64bit.exe --optimal -b400m "%1" "%1.blz"
9177782 0.05 930 Satanichi_GCC730_64bit.exe "%1" "%1.Nakamichi" 24 4000 i
10060030 25.0 0.25 365.81 zopfli Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
10771354 26.7 4.74 325.28 zlib 9 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
11728605 29.1 5.02 651.98 lzo1b 999 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
12073794 30.0 9.23 2629.03 lz4 9 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
12239442 30.4 38.69 435.75 lzfse Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
40303104 100.0 9216.35 9159.80 memcpy Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
40303104/774=52,071 B/s
40303104/29=1,389,762 B/s @jibsen
40303104/5=8,060,620 B/s
F:\S>timer64.exe Satanichi_GCC730_64bit.exe "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.Nakamichi" 24 4000 i
...
Leprechaun: Total Searches-n-Inserts Per Second: 41,639,014 SNIPS = 26,898,803,684 keys in 646 seconds
Leprechaun: RAM needed to house B-trees (relative to the file being ripped): 51N = 1,994MB
Compressing 40,303,104 bytes ...
-; Each rotation means 64KB are encoded; Speed: 0,338,681 B/s; Done 100%; Compression Ratio: 4.39:1; Matches(16/24/48): 752,724/380,713/20,519; 128[+] long matches: 0; ETA: 0.00 hours
NumberOfFullLiterals (lower-the-better): 1294
Tsuyo_HEURISTIC_APPLIED_thrice_back-to-back: 0
NumberOf(Tiny)Matches[Micro]Window (4)[16B]: 9672
NumberOfMatches[Bheema]Window [128GB window]: 11405
RAM-to-RAM performance: 338681 B/s.
Compressed to 9,177,782 bytes.
Source-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xcd46,fbc3
Target-file-Hash(FNV1A_YoshimitsuTRIAD) = 0xb7ac,545f
Decompressing 9,177,782 (being the compressed stream) bytes ...
RAM-to-RAM performance: 930 MB/s.
Verification (input and output sizes match) OK.
Verification (input and output blocks match) OK.
Kernel Time = 1.390 = 0%
User Time = 772.437 = 99%
Process Time = 773.828 = 99% Virtual Memory = 5952 MB
Global Time = 774.604 = 100% Physical Memory = 3935 MB
F:\S>timer64.exe BriefLZ_130_Intel_v19_64bit.exe --optimal -b400m "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.blz"
Kernel Time = 0.343 = 1%
User Time = 29.000 = 98%
Process Time = 29.343 = 99% Virtual Memory = 8869 MB
Global Time = 29.352 = 100% Physical Memory = 1180 MB
F:\S>TIMER64 bwtturbo -59 -t0 -b1024 "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar" "Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.zv"
Kernel Time = 0.359 = 7%
User Time = 4.484 = 92%
Process Time = 4.843 = 99% Virtual Memory = 2564 MB
Global Time = 4.850 = 100% Physical Memory = 1219 MB
09/15/2018 04:41 231,424 BriefLZ_130_Intel_v19_64bit.exe
02/07/2020 01:38 555,806 bwtturbo.exe
02/19/2020 13:42 193,517 Satanichi_GCC730_64bit.exe
02/15/2020 22:24 6,144 timer64.exe
01/09/2020 12:51 1,079 turbobench.ini
02/19/2020 18:39 4,536,493 turbobench_build_19_Feb_2020.exe
06/30/2018 20:35 5,446,644 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.zv
06/30/2018 20:32 8,851,573 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.blz
9,177,782 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar.Nakamichi
07/28/2018 07:22 40,303,104 Complete_works_of_Fyodor_Dostoyevsky_in_15_volumes_(Russian).tar
Unlike lzbench, TurboBench is quite useless. Doesn't have an easy way to test all possible codecs to see which one compresses best or decompresses faster.
It absolutely HAS TO HAVE -eall
mode like lzbench. I absolutely wasn't able to even test lzturbo.
Hi. zstd and brotli both produce incorrect output for file https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-12.0.0-amd64-DVD-1.iso (3.7 G). Here is output:
# ./turbobench -D -B4G -ebrotli,1/zstd,1 /tmp/debian-12.0.0-amd64-DVD-1.iso
0 0.0 -0.00 -1023.18 brotli 1 debian-12.0.0-amd64-DVD-1.iso
ERROR at 868478448:3f, 66 file=debian-12.0.0-amd64-DVD-1.iso
3677323496 93.5 -240.61 -689.08 zstd 1 debian-12.0.0-amd64-DVD-1.iso
ERROR at 869007360:34, cb file=debian-12.0.0-amd64-DVD-1.iso
TurboBench: - Sun Jun 11 10:13:25 2023
C Size ratio% C MB/s D MB/s SCORE Name File
0 0.0 0.00 11053.89 167423219872854250310480057943336997924438274955232624522333477440436613541123027690304032184897808517832736441413296818951444073477260938629943455293343883926095357701695786860068785110851124808262911167419640293628520552910140667265102788183953006076438190605626183719130397942773345747264478380032.00 ?brotli 1 debian-12.0.0-amd64-DVD-1.iso
3677323496 93.5 2599.46 7444.48 3679.89 ?zstd 1 debian-12.0.0-amd64-DVD-1.iso
Test was performed in docker container with debian sid in it. Using gcc 12.2.0 from debian. Using turbobench commit 1d17683 .
Here is full reproducer for docker:
FROM debian:sid-20230522
ENV LC_ALL C.UTF-8
RUN apt-get update && apt-get install -y apt-utils whiptail
RUN apt-get update && apt-get install -y git make gcc g++ wget ca-certificates
RUN git clone --recurse https://github.com/powturbo/TurboBench
WORKDIR TurboBench
RUN git checkout 1d17683531537ca953ae7200fb36198ad5ab2ae6
RUN make
RUN wget 'https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/debian-12.0.0-amd64-DVD-1.iso'
RUN ./turbobench -D -B4G -ebrotli,1/zstd,1 debian-12.0.0-amd64-DVD-1.iso
I would like to know if you could use your tool to evaluate genomic file compression metrics such as compression rate, speed, and especially data loss?
Could I adapt your script to parse specific genomics compression tools?
Tnx
Are you interested in adding new codecs?
I've been testing lately a lot of compressors specific for short strings, some suggestions
If you are interested in more of these I can dig my archives.
Turbobench is marvellous testing tool, many thanks for creating it!
I have a i3-4005U cpu, running ubuntu 16.04. I just tried to clone and run this program, with the command ./turbobench -p2 -S2 file.tbb &> error_output.txt
then the program aborts with exit code 255 and the output of error_output.txt is <86> <AD><FB>
(less
puts <
and >
with hexcode instead of showing raw characters). file.tbb
was not written
When compiling from a pristine clone I'm getting the following error:
plugins.o: In function `codcomp':
plugins.cc:(.text+0x1ce1): undefined reference to `NakaCompress'
plugins.o: In function `coddecomp':
plugins.cc:(.text+0x32aa): undefined reference to `NakaDecompress'
collect2: error: ld returned 1 exit status
make: *** [makefile:436: turbobench] Error 1
Hi, just find this project, And not sure the really difference between turbobench and lzbench. Thanks
Dear Developer,
I've tried to compile with gcc 7.1.1 and got this error:
zpaq/libzpaq.o: In function `libzpaq::Array<int>::resize(unsigned long, int) [clone .constprop.129]':
libzpaq.cpp:(.text+0xf65): undefined reference to `libzpaq::error(char const*)'
libzpaq.cpp:(.text+0xfe0): undefined reference to `libzpaq::error(char const*)'
zpaq/libzpaq.o: In function `libzpaq::Array<unsigned short>::resize(unsigned long, int) [clone .constprop.131]':
libzpaq.cpp:(.text+0x1050): undefined reference to `libzpaq::error(char const*)'
libzpaq.cpp:(.text+0x10d0): undefined reference to `libzpaq::error(char const*)'
zpaq/libzpaq.o: In function `libzpaq::Array<unsigned char>::resize(unsigned long, int) [clone .constprop.132]':
libzpaq.cpp:(.text+0x1188): undefined reference to `libzpaq::error(char const*)'
zpaq/libzpaq.o:libzpaq.cpp:(.text+0x11a0): more undefined references to `libzpaq::error(char const*)' follow
collect2: error: ld returned 1 exit status
make: *** [makefile:436: turbobench] Error 1
4k:-b4k
./turbobench ../dataset/book -efsehuf -b4k
851499254 79.3 681.85 1002.65 855.22 fsehuf
Oodle is on the plugin list, but when I try to use it error occurs:
$ ./turbobench.exe ./data/big_building.ppm -eoodle,89
codec 'oodle' not found
I have dlls in turbobench.exe directory:
oo2core_4_win64.dll
oo2core_6_win64.dll
oo2core_7_win64.dll
oo2core_8_win64.dll
Windows 10 x64, Turbobench latest release.
Thanks.
Latest update: 2019.08.09
Binary:mingw-w64_x86_64-6.3.0-posix-seh-rt_v5-rev2.tar
C Size ratio% C MB/s D MB/s Name (bold = pareto) MB=1.000.0000 42179292 9.1 0.91 189.42 lzturbo 49 42457543 9.1 2.07 186.33 lzma 9d29:fb273:mf=bt4 42988097 9.2 0.57 857.10 brotli 11d29 45871347 9.9 1.50 2206.52 lzturbo 39 46458069 10.0 2.66 1491.27 zstd 22d29 56248534 12.1 1.93 2966.68 lzturbo 29 58474096 12.6 16.25 27.38 bsc 0 58654705 12.6 125.48 2100.32 lzturbo 32 60760645 13.1 13.12 60.33 lzturbo 59 60765065 13.1 14.08 90.42 lzturbo 59t2 73060389 15.7 2.24 5717.94 lzturbo 19 87416420 18.8 11.75 1397.49 zstd 15 88006994 18.9 45.49 514.92 brotli 5 88252775 19.0 1.78 2232.29 lizard 49 94672486 20.3 1.91 3073.42 lizard 29 111853281 24.0 12.56 39.18 bzip2 126614955 27.2 9.25 340.34 zlib 9 133921751 28.8 2.97 2230.60 lizard 39 143616390 30.9 38.77 3233.08 lz4 9 144311136 31.0 3.34 3519.31 lizard 19 145898393 31.3 1.51 3828.81 lzsse8 17 174569457 37.5 871.54 4503.78 lzturbo 10 179181598 38.5 657.00 2866.63 lz4 1 465457156 100.0 13824.50 14242.87 memcpy
English Text:Encyclopaedia_Judaica_(in_22_volumes)_TXT.tar
C Size ratio% C MB/s D MB/s Name 21905524 20.3 9.45 15.92 bsc 0 22279381 20.7 9.46 49.89 lzturbo 59 22280917 20.7 10.04 71.61 lzturbo 59t2 27070097 25.1 0.91 82.32 lzturbo 49 27526894 25.5 1.23 86.14 lzma 9d29:fb273:mf=bt4 27600942 25.6 0.46 305.70 brotli 11d29 27945445 25.9 1.06 672.33 lzturbo 39 28008216 26.0 1.43 640.32 zstd 22d29 31570314 29.3 1.10 1124.56 lzturbo 29 31734097 29.4 12.91 29.08 bzip2 32696227 30.3 46.86 781.41 lzturbo 32 33249008 30.8 4.94 851.66 zstd 15 35077489 32.5 1.85 694.91 lizard 49 36277020 33.7 26.67 367.32 brotli 5 41420278 38.4 1.92 973.00 lizard 29 42104125 39.1 8.15 1145.53 lizard 39 42236588 39.2 12.32 243.79 zlib 9 43696364 40.5 9.52 3085.19 lzsse8 17 45341158 42.1 1.28 2958.75 lzturbo 19 47834366 44.4 8.36 1555.15 lizard 19 48214128 44.7 22.56 2825.87 lz4 9 67850583 63.0 438.16 3994.67 lzturbo 10 68340992 63.4 358.85 2790.46 lz4 1 107784196 100.0 14225.18 14694.50 memcpy
PDF: Learn-Hot-English_magazine_(18-issues).tar
C Size ratio% C MB/s D MB/s Name 244285827 54.0 0.47 212.50 brotli 11d30 245265022 54.2 2.17 33.91 lzturbo 49 246117453 54.4 3.22 3616.72 lzturbo 39 246415683 54.5 3.48 34.90 lzma 9d30:fb273 246786277 54.6 3.84 2182.46 zstd 22d30 250400776 55.4 81.47 3913.83 lzturbo 32 253265114 56.0 3.62 7728.73 lzturbo 19 253761470 56.1 3.53 6090.31 lzturbo 29 289049146 63.9 5.62 6516.19 lizard 29 289084982 63.9 5.30 5445.98 lizard 49 299881226 66.3 17.58 4348.64 zstd 15 330124786 73.0 47.57 6174.21 lz4 9 330179011 73.0 8.09 6278.61 lizard 39 330769147 73.1 28.04 351.13 zlib 9 330921620 73.2 9.18 6710.84 lizard 19 334380724 73.9 1940.57 7960.84 lzturbo 10 349951390 77.4 0.69 4303.37 lzsse8 17 368898163 81.6 0.69 2528.31 lzsse4 17 403005235 89.1 0.69 1329.14 lzsse2 17 452223492 100.0 13956.22 13771.35 memcpy
Your default levels for ZSTD seem a little wonky:
-5
| --fast=5
(the fastest level currently available). This should push it into LZ4 territory.You might want to comment the .ini file, as it's not clear what each preset means (like BWT). Also, the README mentions includes a FASTEST preset the turbobench.ini file only has TURBO.
Thanks a lot for the nifty TurboBench, after wandering here and there, finally thought it is a good idea to post my endless quest (in form of console logs and tables) for giving a rich picture of [de]compressors of today.
Hope, you don't mind me posting on a weekly basis inhere.
My wish is we to have on a public site (GitHub is fully okay) as many as possible interesting datasets tested.
The single PDF file unfinished (134 pages) overview of Textual Benchmarks (done on my other laptop with i5-7200u) is here:
https://drive.google.com/file/d/1DUVowDtp__WGiC3HIw1whE2dTKt7kvBD/view?usp=sharing
Wanted to be much more versatile and bigger, but it is what it is.
The testmachine will be (already acquired) the Toshiba laptop that I used to torture:
My wish was (has to wait) to buy a dedicated machine for heavy compression benchmarks with i3-8100 and 64GB DDR4: http://www.overclock.net/forum/158-laptops-netbooks/1619592-kaby-lake-intel-core-i5-7200u-how-fast.html#post26536756
The key thing inhere is the eventual running of any file (<1.5GB due to free RAM being ~14GB). It would be nice once and for all we to have one versatile roster. My preference towards textual data remains, so the files listed below will be run first.
E:\Textual_Madness_quickoverview>dir/og/on
02/13/2018 02:48 AM <DIR> .
02/13/2018 02:48 AM <DIR> ..
02/13/2018 03:40 AM <DIR> _Check_Integrity_Folder
02/13/2018 03:42 AM 193,367,552 Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar
02/13/2018 03:42 AM 681,979,392 book_serie_SPETSNAZ_803_novels_(Russian).tar
02/13/2018 03:42 AM 1,000,000,000 enwik9
02/13/2018 03:42 AM 648,260,096 gcc-6.3.0_(96398_Files_5502_Folders).tar
02/13/2018 03:42 AM 2,037,880,832 INTERNET_SACRED_TEXT_ARCHIVE_DVD-ROM_9_(English_140479_htm_files).tar
02/13/2018 03:42 AM 1,640,759,296 Machine-Learning_amazon_review_full_csv_(35_million_reviews).tar
02/13/2018 03:42 AM 4,680,140,800 Machine-Learning_British-National-Corpus_XML-edition.tar
02/13/2018 03:42 AM 405,610,647 Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
02/13/2018 03:42 AM 150,950,913 Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
02/13/2018 03:42 AM 1,917,822,288 Machine-Learning_Urban_Dictionary_Definitions_Corpus_(1999_-_May-2016).words.json
02/13/2018 03:42 AM 310,654,639 Machine-Learning_Wikipedia_Article_Titles_(September-20-2017).txt
02/13/2018 03:42 AM 133,901,432 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
02/13/2018 03:42 AM 630,339,584 Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
02/13/2018 03:42 AM 203,288,144 Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
02/13/2018 03:42 AM 385,286,656 Machine-Learning_www.kaggle.com_japaneseenglish-bilingual-corpus_(500000_pairs_of_manually-translated_sentences).tar
02/13/2018 03:42 AM 282,218,054 Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
02/13/2018 03:42 AM 560,714,120 Oxford_English_Dictionary_2nd_Edition_Version_4_(En-En).dsl
02/13/2018 03:42 AM 325,071,872 Star_Trek_-_737_Ebooks.tar
02/13/2018 03:42 AM 630,349,312 Stephan_Kaze_http_unbound.biola.edu_103-bibles.tar
02/13/2018 03:42 AM 1,028,290,560 Stephan_Kaze_windows_nt_4_source_code.tar
02/13/2018 03:42 AM 1,382,122,496 TEXTFILES.COM_(58096_files).tar
02/13/2018 03:42 AM 1,499,100,672 the-anarchist-library-2016-01-18-en.tar
02/13/2018 03:42 AM 1,036,155,727 Urban_Dictionary_2015_(Eng-Eng)_utf8.dsl
02/13/2018 03:42 AM 681,378,816 webdevdata.org_8000_home_pages_from_the_top_10000_most_popular_web_sites.tar
02/13/2018 03:42 AM 686,991,360 www.kernel.org_linux-4.8.4.tar
02/13/2018 03:42 AM 975,021,056 www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar
E:\Textual_Madness_quickoverview>type _Check_Integrity_Folder\_Check_Integrity.ORIGINAL
d32999b84d3a0c1395c4b5ed9200a248ce6f3d38 ..\Big_Soviet_Encyclopedia_in_30_volumes_(1239-HTMs).tar
ad49660290b680d759238ab0c5a8d15307080b68 ..\book_serie_SPETSNAZ_803_novels_(Russian).tar
2996e86fb978f93cca8f566cc56998923e7fe581 ..\enwik9
c103fbe221bfb384c2417e27fcb7c6420fd114f1 ..\gcc-6.3.0_(96398_Files_5502_Folders).tar
7c2e32a76716e184d302e5542b96c16e95047002 ..\INTERNET_SACRED_TEXT_ARCHIVE_DVD-ROM_9_(English_140479_htm_files).tar
05a382d4d82a2b81f954f17a3cbe8950e36c3a55 ..\Machine-Learning_amazon_review_full_csv_(35_million_reviews).tar
e199cd3f606db268cdd0be5eef1c4932f37acd13 ..\Machine-Learning_British-National-Corpus_XML-edition.tar
c3a9a9116551646c04317abb1c1b4d612cdeb3e9 ..\Machine-Learning_Douban_Movie_Short_Comments_(Chinese).csv
7db6ec6256ac5346e8f98ca52076d12193ae84d7 ..\Machine-Learning_Global_Terrorism_Database_(more_than_170000_terrorist_attacks_worldwide_1970-2016).csv
fd0567e4a5d800f1880c6efa459125d4256646f4 ..\Machine-Learning_Urban_Dictionary_Definitions_Corpus_(1999_-_May-2016).words.json
89d5905943237c81ce0c6ff0c08a001e4f0b6355 ..\Machine-Learning_Wikipedia_Article_Titles_(September-20-2017).txt
56ffb512c8e055a5b3bb959c0210521dca13d178 ..\Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
b5e3ce3ec31d7088709fde1a6465088beeaf4c91 ..\Machine-Learning_www.kaggle.com_every-song-you-have-heard-almost_(over_500000_song_lyrics).tar
ab83ab51bb94b6ddeac80639fc83cbfe7ed35743 ..\Machine-Learning_www.kaggle.com_examine-the-examiner_(headlines_of_3_million_articles).csv
6014d55046c734a5be1c006c136432203bab0c3a ..\Machine-Learning_www.kaggle.com_japaneseenglish-bilingual-corpus_(500000_pairs_of_manually-translated_sentences).tar
6a4dfc9b77af3c15ba58c1032f549448e4da2dcb ..\Machine-Learning_www.kaggle.com_opencorpora-russian_(A_Tagged_1.5_Million_Word_Corpus_of_Russian).txt
31ae7c9ecdbfbf79221bab5db963268acec3f77a ..\Oxford_English_Dictionary_2nd_Edition_Version_4_(En-En).dsl
b8e5c4b1932300dee41c1e9655951599d4f7a036 ..\Star_Trek_-_737_Ebooks.tar
199769ffd4c11c8d01fa0bac00b65edf830455bd ..\Stephan_Kaze_http_unbound.biola.edu_103-bibles.tar
4b42a93f4cb3952dc878f1c3ee600bf9162b4f8b ..\Stephan_Kaze_windows_nt_4_source_code.tar
f8cd4ad35c75f3f40df1658cd54bdc14eb8d952e ..\TEXTFILES.COM_(58096_files).tar
76287ba461260fb11bfd6d4f6acc44a1d5857bee ..\the-anarchist-library-2016-01-18-en.tar
d7827d7458bcd5457aa1499a8479fd05a574a591 ..\Urban_Dictionary_2015_(Eng-Eng)_utf8.dsl
ef845f6b1a57d69228999664e000e2e678fe501a ..\webdevdata.org_8000_home_pages_from_the_top_10000_most_popular_web_sites.tar
b569d3709706ff240ade45b276f5ed78b931a417 ..\www.kernel.org_linux-4.8.4.tar
25cc7d3b4f18e63f5452127f2b915adc834f1f67 ..\www.ncbi.nlm.nih.gov_Dragonfly_(Ladona_fulva)_whole_genome_shotgun.tar
E:\Textual_Madness_quickoverview>type Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
id;budget;genres;imdb_id;original_language;original_title;overview;popularity;production_companies;production_countries;release_date;revenue;runtime;spoken_languages;status;tagline;title;vote_average;vote_count;production_companies_number;production_countries_number;spoken_languages_number
2;0;Drama|Crime;tt0094675;fi;Ariel;Taisto Kasurinen is a Finnish coal miner whose father has just committed suicide and who is framed for a crime he did not commit. In jail, he starts to dream about leaving the country and starting a new life. He escapes from prison but things don't go as planned...;0.823904;Villealfa Filmproduction Oy;Finland;21/10/1988;0;69;suomi;Released;;Ariel;7.1;40;2;1;2
3;0;Drama|Comedy;tt0092149;fi;Varjoja paratiisissa;An episode in the life of Nikander, a garbage man, involving the death of a co-worker, an affair and much more.;0.47445;Villealfa Filmproduction Oy;Finland;16/10/1986;0;76;English;Released;;Shadows in Paradise;7.0;32;1;1;3
5;4000000;Crime|Comedy;tt0113101;en;Four Rooms;It's Ted the Bellhop's first night on the job...and the hotel's very unusual guests are about to place him in some outrageous predicaments. It seems that this evening's room service is serving up one unbelievable happening after another.;1.698;Miramax Films;United States of America;25/12/1995;4300000;98;English;Released;Twelve outrageous guests. Four scandalous requests. And one lone bellhop, in his first day on the job, who's in for the wildest New year's Eve of his life.;Four Rooms;6.5;485;2;1;1
6;0;Action|Thriller|Crime;tt0107286;en;Judgment Night;"While racing to a boxing match, Frank, Mike, John and Rey get more than they bargained for. A wrong turn lands them directly in the path of Fallon, a vicious, wise-cracking drug lord. After accidentally witnessing Fallon murder a disloyal henchman, the four become his unwilling prey in a savage game of cat & mouse as they are mercilessly stalked through the urban jungle in this taut suspense drama";1.32287;Universal Pictures;Japan;15/10/1993;12136938;110;English;Released;Don't move. Don't whisper. Don't even breathe.;Judgment Night;6.5;69;3;2;1
8;42000;Documentary;tt0825671;en;Life in Loops (A Megacities RMX);"Timo Novotny labels his new project an experimental music documentary film, in a remix of the celebrated film Megacities (1997), a visually refined essay on the hidden faces of several world ""megacities"" by leading Austrian documentarist Michael Glawogger. Novotny complements 30 % of material taken straight from the film (and re-edited) with 70 % as yet unseen footage in which he blends original shots unused by Glawogger with his own sequences (shot by Megacities cameraman Wolfgang Thaler) from Tokyo. Alongside the Japanese metropolis, Life in Loops takes us right into the atmosphere of Mexico City, New York, Moscow and Bombay. This electrifying combination of fascinating film images and an equally compelling soundtrack from Sofa Surfers sets us off on a stunning audiovisual adventure across the continents. The film also makes an original contribution to the discussion on new trends in documentary filmmaking. Written by KARLOVY VARY IFF 2006";0.054716;inLoops;Austria;01/01/2006;0;80;English;Released;A Megacities remix.;Life in Loops (A Megacities RMX);6.4;4;1;1;5
9;0;Drama;tt0425473;de;Sonntag im August;;0.001647;none;Germany;02/09/2004;0;15;Deutsch;Released;;Sunday in August;5.3;2;0;1;1
11;11000000;Adventure|Action|Science Fiction;tt0076759;en;Star Wars;Princess Leia is captured and held hostage by the evil Imperial forces in their effort to take over the galactic Empire. Venturesome Luke Skywalker and dashing captain Han Solo team together with the loveable robot duo R2-D2 and C-3PO to rescue the beautiful princess and restore peace and justice in the Empire.;10.492614;Lucasfilm;United States of America;25/05/1977;775398007;121;English;Released;A long time ago in a galaxy far, far away...;Star Wars;8.0;6168;2;1;1
12;94000000;Animation|Family;tt0266543;en;Finding Nemo;Nemo, an adventurous young clownfish, is unexpectedly taken from his Great Barrier Reef home to a dentist's office aquarium. It's up to his worrisome father Marlin and a friendly but forgetful fish Dory to bring Nemo home -- meeting vegetarian sharks, surfer dude turtles, hypnotic jellyfish, hungry seagulls, and more along the way.;9.915573;Pixar Animation Studios;United States of America;30/05/2003;940335536;100;English;Released;There are 3.7 trillion fish in the ocean, they're looking for one.;Finding Nemo;7.6;5531;1;1;1
13;55000000;Comedy|Drama|Romance;tt0109830;en;Forrest Gump;A man with a low IQ has accomplished great things in his life and been present during significant historic events - in each case, far exceeding what anyone imagined he could do. Yet, despite all the things he has attained, his one true love eludes him. 'Forrest Gump' is the story of a man who rose above his challenges, and who proved that determination, courage, and love are more important than ability.;10.351236;Paramount Pictures;United States of America;06/07/1994;677945399;142;English;Released;The world will never be the same, once you've seen it through the eyes of Forrest Gump.;Forrest Gump;8.2;7204;1;1;1
14;15000000;Drama;tt0169547;en;American Beauty;Lester Burnham, a depressed suburban father in a mid-life crisis, decides to turn his hectic life around after developing an infatuation with his daughter's attractive friend.;8.191009;DreamWorks SKG;United States of America;15/09/1999;356296601;122;English;Released;Look closer.;American Beauty;7.9;2994;2;1;1
15;839727;Mystery|Drama;tt0033467;en;Citizen Kane;Newspaper magnate, Charles Foster Kane is taken from his mother as a boy and made the ward of a rich industrialist. As a result, every well-meaning, tyrannical or self-destructive move he makes for the rest of his life appears in some way to be a reaction to that deeply wounding event.;3.82689;RKO Radio Pictures;United States of America;30/04/1941;23217674;119;English;Released;It's Terrific!;Citizen Kane;7.9;1110;2;1;1
16;12800000;Drama|Crime|Music;tt0168629;en;Dancer in the Dark;Selma, a Czech immigrant on the verge of blindness, struggles to make ends meet for herself and her son, who has inherited the same genetic disorder and will suffer the same fate without an expensive operation. When life gets too difficult, Selma learns to cope through her love of musicals, escaping life's troubles - even if just for a moment - by dreaming up little numbers to the rhythmic beats of her surroundings.;2.106217;Fine Line Features;Argentina;17/05/2000;40031879;140;English;Released;You don't need eyes to see.;Dancer in the Dark;7.6;348;26;12;1
17;0;Horror|Thriller|Mystery;tt0411267;en;The Dark;Adèle and her daughter Sarah are traveling on the Welsh coastline to see her husband James when Sarah disappears. A different but similar looking girl appears who says she died in a past time. Adèle tries to discover what happened to her daughter as she is tormented by Celtic mythology from the past.;1.253999;Constantin Film;Germany;26/01/2006;0;87;English;Released;One of the living for one of the dead.;The Dark;5.6;69;4;2;2
18;90000000;Adventure|Fantasy|Action|Thriller|Science Fiction;tt0119116;en;The Fifth Element;In 2257, a taxi driver is unintentionally given the task of saving a young girl who is part of the key that will ensure the survival of humanity.;9.233786;Columbia Pictures;France;07/05/1997;263920180;126;English;Released;There is no future without it.;The Fifth Element;7.2;3629;2;1;3
19;92620000;Drama|Science Fiction;tt0017136;de;Metropolis;In a futuristic city sharply divided between the working class and the city planners, the son of the city's mastermind falls in love with a working class prophet who predicts the coming of a savior to mediate their differences.;3.669986;Paramount Pictures;Germany;10/01/1927;650422;153;No Language;Released;There can be no understanding between the hands and the brain unless the heart acts as mediator.;Metropolis;8.0;614;2;1;1
20;0;Drama|Romance;tt0314412;en;My Life Without Me;A Pedro Almodovar production in which a fatally ill mother with only two months to live creates a list of things she wants to do before she dies with out telling her family of her illness.;0.911462;El Deseo;Canada;07/03/2003;9726954;106;English;Released;;My Life Without Me;7.2;75;2;2;1
21;0;Documentary;tt0060371;en;The Endless Summer;The Endless Summer, by Bruce Brown, is one of the first and most influential surf movies of all times. The film documents American surfers Mike Hynson and Robert August as they travel the world during California’s winter (which back in 1965 was off-season for surfing) in search of the perfect wave and an endless summer.;0.144179;Bruce Brown Films;United States of America;15/06/1966;0;95;English;Released;;The Endless Summer;7.8;20;1;1;1
22;140000000;Adventure|Fantasy|Action;tt0325980;en;Pirates of the Caribbean: The Curse of the Black Pearl;Jack Sparrow, a freewheeling 17th-century pirate who roams the Caribbean Sea, butts heads with a rival pirate bent on pillaging the village of Port Royal. When the governor's daughter is kidnapped, Sparrow decides to help the girl's love save her. But their seafaring mission is hardly simple.;28.769026;Walt Disney Pictures;United States of America;09/07/2003;655011224;143;English;Released;Prepare to be blown out of the water.;Pirates of the Caribbean: The Curse of the Black Pearl;7.4;6368;2;1;1
24;30000000;Action|Crime;tt0266697;en;Kill Bill: Vol. 1;An assassin is shot at the altar by her ruthless employer, Bill and other members of their assassination circle – but 'The Bride' lives to plot her vengeance. Setting out for some payback, she makes a death list and hunts down those who wronged her, saving Bill for last.;7.891837;Miramax Films;United States of America;10/10/2003;180949000;111;English;Released;Go for the kill.;Kill Bill: Vol. 1;7.7;4486;3;1;3
25;72000000;Drama|War;tt0418763;en;Jarhead;Jarhead is a film about a US Marine Anthony Swofford’s experience in the Gulf War. After putting up with an arduous boot camp, Swafford and his unit are sent to the Persian Gulf where they are earger to fight but are forced to stay back from the action. Meanwhile Swofford gets news of his girlfriend is cheating on him. Desperately he wants to kill someone and finally put his training to use.;2.41718;Universal Pictures;Germany;04/11/2005;96889998;125;English;Released;Welcome to the suck.;Jarhead;6.5;722;4;2;4
26;1400000;Drama;tt0352994;en;LaLehet Al HaMayim;Eyal, an Israeli Mossad agent, is given the mission to track down and kill the very old Alfred Himmelman, an ex-Nazi officer, who might still be alive. Pretending to be a tourist guide, he befriends his grandson Axel, in Israel to visit his sister Pia. The two men set out on a tour of the country during which, Axel challenges Eyal's values.;0.455665;Lama Films;Israel;05/02/2004;0;103;العربية;Released;He was trained to hate until he met the enemy.;Walk on Water;6.4;18;2;2;6
27;1000000;Drama|Music|Romance;tt0411705;en;9 Songs;Matt, a young glaciologist, soars across the vast, silent, icebound immensities of the South Pole as he recalls his love affair with Lisa. They meet at a mobbed rock concert in a vast music hall - London's Brixton Academy. They are in bed at night's end. Together, over a period of several months, they pursue a mutual sexual passion whose inevitable stages unfold in counterpoint to nine live-concert songs.;2.939728;Revolution Films;United Kingdom;16/07/2004;1574623;66;English;Released;2 lovers, one summer, and the 9 songs that defined them.;9 Songs;5.1;95;1;1;1
28;31500000;Drama|War;tt0078788;en;Apocalypse Now;"At the height of the Vietnam war, Captain Benjamin Willard is sent on a dangerous mission that, officially, ""does not exist, nor will it ever exist."" His goal is to locate - and eliminate - a mysterious Green Beret Colonel named Walter Kurtz, who has been leading his personal army on illegal guerrilla missions into enemy territory.";7.620077;United Artists;United States of America;15/08/1979;89460381;153;;Released;This is the end...;Apocalypse Now;8.0;1869;2;1;4
30;0;Animation|Science Fiction;tt1530535;ja;彼女の想いで;Koji Morimato’s animated science fiction short story about how the boarder between reality and illusion on a space station become blurry.;0.811221;Studio 4°C;Japan;23/12/1995;0;44;日本語;Released;;Magnetic Rose;7.7;10;1;1;1
31;0;Action|Animation|Comedy;tt6266826;ja;最臭兵器;Tensai Okamura’s animated action packed short story with lots of humorous elements in which a person transforms into a weapon of mass destruction without themselves being aware.;1.281042;Studio 4°C;Japan;23/12/1995;0;40;日本語;Released;;Stink Bomb;5.3;3;1;1;1
32;0;Animation|History;tt6264824;ja;大砲の街;Otomo Katsuhiro’s short anime story;0.838219;Studio 4°C;Japan;23/12/1995;0;21;日本語;Released;;Cannon Fodder;5.3;3;1;1;1
...
H:\smashshop_2018-Feb-12_Judaica>_BENCH_a_file.BAT Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
H:\smashshop_2018-Feb-12_Judaica>lzbench173 -c4 -i1,15 -o3 -ebrotli24,1,5,11/tornado,16/blosclz,9/brieflz/crush,2/csc,5/density,3/fastlz,2/gipfeli/zstd24,12,22/zstd24LDM,12,22/lzo1b,999/lzham,4/lzham24,4/libdeflate,1,12/lz4fast,1,99/lz4/lz4hc,10,12/lizard,19,29,39,49/lzf,1/lzfse/lzg,9/lzham,1/lzjb/lzlib,9/lzma,9/lzrw,5/lzsse2,17/lzsse4,17/lzsse8,17/lzvn/pithy,9/quicklz,3/snappy/slz_zlib,3/ucl_nrv2b,9/ucl_nrv2d,9/ucl_nrv2e,9/xpack,9/xz,9/yalz77,12/yappy,99/zlib,1,5,9/zling,4/shrinker/wflz/lzmat Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzbench 1.7.3 (64-bit Windows) Assembled by P.Skibinski
Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename
memcpy 6365 MB/s 6365 MB/s 133901432 133901432 100.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
...
The results sorted by column number 4:
Compressor name Compress. Decompress. Orig. size Compr. size Ratio Filename
csc 2016-10-13 -5 1.65 MB/s 45 MB/s 133901432 34749865 25.95 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzma 16.04 -9 0.78 MB/s 60 MB/s 133901432 34816931 26.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
xz 5.2.3 -9 0.82 MB/s 55 MB/s 133901432 34817747 26.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzham 1.0 -d26 -4 0.58 MB/s 122 MB/s 133901432 35117612 26.23 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzlib 1.8 -9 0.74 MB/s 40 MB/s 133901432 35346969 26.40 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
tornado 0.6a -16 0.87 MB/s 119 MB/s 133901432 36088796 26.95 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brotli24 2017-12-12 -11 0.33 MB/s 172 MB/s 133901432 36237727 27.06 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzham24 1.0 -4 0.71 MB/s 122 MB/s 133901432 36536959 27.29 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24 1.3.3 -22 0.89 MB/s 335 MB/s 133901432 36966062 27.61 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24LDM 1.3.3 -22 0.91 MB/s 337 MB/s 133901432 37129453 27.73 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zling 2016-01-10 -4 18 MB/s 103 MB/s 133901432 41708378 31.15 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzham 1.0 -d26 -1 1.43 MB/s 125 MB/s 133901432 42404312 31.67 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24 1.3.3 -12 5.02 MB/s 344 MB/s 133901432 44018384 32.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zstd24LDM 1.3.3 -12 4.88 MB/s 344 MB/s 133901432 44019143 32.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
crush 1.0 -2 0.20 MB/s 219 MB/s 133901432 45171471 33.73 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brotli24 2017-12-12 -5 13 MB/s 246 MB/s 133901432 45354869 33.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
xpack 2016-06-02 -9 8.36 MB/s 448 MB/s 133901432 49434209 36.92 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -49 0.83 MB/s 664 MB/s 133901432 52052053 38.87 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
libdeflate 0.7 -12 5.71 MB/s 380 MB/s 133901432 52203890 38.99 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -29 0.87 MB/s 776 MB/s 133901432 52980228 39.57 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzfse 2017-03-08 35 MB/s 367 MB/s 133901432 53217444 39.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zlib 1.2.11 -9 13 MB/s 192 MB/s 133901432 54103240 40.41 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzsse2 2016-05-14 -17 5.90 MB/s 1808 MB/s 133901432 54676447 40.83 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zlib 1.2.11 -5 21 MB/s 190 MB/s 133901432 54783631 40.91 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
ucl_nrv2e 1.03 -9 0.92 MB/s 188 MB/s 133901432 55218581 41.24 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzsse4 2016-05-14 -17 6.82 MB/s 1876 MB/s 133901432 55450256 41.41 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzsse8 2016-05-14 -17 6.41 MB/s 1846 MB/s 133901432 55598759 41.52 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
ucl_nrv2d 1.03 -9 0.94 MB/s 190 MB/s 133901432 55700314 41.60 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brotli24 2017-12-12 -1 91 MB/s 200 MB/s 133901432 56499193 42.19 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
ucl_nrv2b 1.03 -9 0.94 MB/s 188 MB/s 133901432 56995941 42.57 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
libdeflate 0.7 -1 88 MB/s 349 MB/s 133901432 58406821 43.62 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzo1b 2.09 -999 10 MB/s 393 MB/s 133901432 60063312 44.86 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4hc 1.8.0 -12 8.74 MB/s 1610 MB/s 133901432 60256681 45.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -19 5.38 MB/s 1781 MB/s 133901432 60301678 45.03 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4hc 1.8.0 -10 15 MB/s 1627 MB/s 133901432 60525615 45.20 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lizard 1.0 -39 5.43 MB/s 1626 MB/s 133901432 61587571 45.99 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzg 1.0.8 -9 0.62 MB/s 403 MB/s 133901432 61803020 46.16 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
zlib 1.2.11 -1 51 MB/s 190 MB/s 133901432 62234317 46.48 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzmat 1.01 22 MB/s 193 MB/s 133901432 62428884 46.62 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
yalz77 2015-09-19 -12 11 MB/s 174 MB/s 133901432 62982598 47.04 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
brieflz 1.1.0 70 MB/s 108 MB/s 133901432 63919589 47.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
density 0.12.5 beta -3 104 MB/s 196 MB/s 133901432 63987540 47.79 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
quicklz 1.5.0 -3 31 MB/s 532 MB/s 133901432 66622450 49.75 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzvn 2017-03-08 30 MB/s 546 MB/s 133901432 66786297 49.88 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
gipfeli 2016-07-13 102 MB/s 208 MB/s 133901432 68309192 51.01 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzrw 15-Jul-1991 -5 83 MB/s 279 MB/s 133901432 70927828 52.97 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
pithy 2011-12-24 -9 162 MB/s 815 MB/s 133901432 73138033 54.62 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
shrinker 0.1 193 MB/s 577 MB/s 133901432 75857895 56.65 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzf 3.6 -1 167 MB/s 346 MB/s 133901432 78638419 58.73 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
fastlz 0.1 -2 164 MB/s 302 MB/s 133901432 80024724 59.76 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
slz_zlib 1.0.0 -3 126 MB/s 167 MB/s 133901432 80345979 60.00 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
yappy 2014-03-22 -99 61 MB/s 1431 MB/s 133901432 81157014 60.61 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
blosclz 2015-11-10 -9 140 MB/s 470 MB/s 133901432 81334487 60.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
snappy 1.1.4 192 MB/s 652 MB/s 133901432 84587804 63.17 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4fast 1.8.0 -1 273 MB/s 1774 MB/s 133901432 86683715 64.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4 1.8.0 273 MB/s 1773 MB/s 133901432 86683715 64.74 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
wflz 2015-09-16 123 MB/s 508 MB/s 133901432 90616277 67.67 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lzjb 2010 155 MB/s 300 MB/s 133901432 101511464 75.81 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
lz4fast 1.8.0 -99 3442 MB/s 5355 MB/s 133901432 133073240 99.38 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
H:\smashshop_2018-Feb-12_Judaica>"turbobenchs_Official_v17.04_-_build_07_Apr_2017.exe" Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv -elibdeflate,12/oodle,19,49,89,112,114,116,118,129/lzsse2,17/lzturbo,19,12,29,22,39,32,49,59/zstd,12,22/lizard,19,29,39,49/brotli,11/lzma,9/bzip2/xpack,9/chameleon,2/density,3/lzham,4/trle/bsc,3,6/zpaq,2,5 -g -I1 -J31 -k1 -B2G
C Size ratio% C MB/s D MB/s Name File
27751984 20.7 6.28 13.79 lzturbo 59 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
28626045 21.4 0.29 0.29 zpaq 5 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
29365736 21.9 12.30 5.76 bsc 6 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
33800605 25.2 0.60 61.26 lzturbo 49 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
34816935 26.0 0.79 60.60 lzma 9 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
34897942 26.1 0.59 122.19 lzham 4 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
35231273 26.3 0.69 314.19 lzturbo 39 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
35447440 26.5 0.67 324.30 zstd 22 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
36219098 27.0 0.34 342.69 oodle 129 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
36219098 27.0 0.39 342.40 oodle 89 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
37551395 28.0 0.20 206.53 oodle 19 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
38197262 28.5 0.35 212.81 brotli 11 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
38824278 29.0 15.90 10.57 bsc 3 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
39475083 29.5 9.44 18.91 bzip2 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
39775117 29.7 0.73 516.82 lzturbo 29 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
41317955 30.9 16.96 351.28 lzturbo 32 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
42616730 31.8 0.40 224.70 xpack 9 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
44018388 32.9 4.99 381.61 zstd 12 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
44786082 33.4 2.71 51.32 zpaq 2 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
52052057 38.9 0.85 680.56 lizard 49 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
52203900 39.0 5.74 360.87 libdeflate 12 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
52980232 39.6 0.87 795.93 lizard 29 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
53378419 39.9 16.99 603.31 lzturbo 22 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
54676451 40.8 5.62 1800.68 lzsse2 17 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
56151529 41.9 0.34 1621.35 oodle 116 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
56763285 42.4 0.45 1607.96 oodle 118 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
60271701 45.0 0.84 2400.20 lzturbo 19 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
60290973 45.0 2.73 1716.35 oodle 49 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
60301682 45.0 5.30 1850.90 lizard 19 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
61519812 45.9 8.27 1765.82 oodle 114 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
61587575 46.0 5.38 1738.74 lizard 39 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
63905083 47.7 29.59 2402.07 lzturbo 12 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
63987386 47.8 114.56 205.91 density 3 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
67517585 50.4 29.13 2244.17 oodle 112 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
82491467 61.6 1043.42 1581.51 chameleon 2 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
133625257 99.8 104.70 1420.42 trle Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.tbb
H:\smashshop_2018-Feb-12_Judaica>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o6 -m256 -fMachine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O6.PPMd_varI Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Fast PPMII compressor for textual data, variant I, Apr 3 2016
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv:133901432
Kernel Time = 0.156 = 0%
User Time = 28.797 = 97%
Process Time = 28.953 = 97% Virtual Memory = 258 MB
Global Time = 29.555 = 100% Physical Memory = 203 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 PPMd_varI_rev2_Intel15_32bit.exe e -o16 -m256 -fMachine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O16.PPMd_varI Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Fast PPMII compressor for textual data, variant I, Apr 3 2016
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv:133901432
Kernel Time = 0.234 = 0%
User Time = 48.719 = 98%
Process Time = 48.953 = 99% Virtual Memory = 258 MB
Global Time = 49.416 = 100% Physical Memory = 259 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "7za_v16.04_x64.exe" a -tgzip -mx9 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9.zip Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04
Scanning the drive:
1 file, 133901432 bytes (128 MiB)
Creating archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9.zip
Items to compress: 1
Files read from disk: 1
Archive size: 52319186 bytes (50 MiB)
Everything is Ok
Kernel Time = 0.093 = 0%
User Time = 143.130 = 99%
Process Time = 143.224 = 99% Virtual Memory = 6 MB
Global Time = 143.710 = 100% Physical Memory = 7 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "7za_v16.04_x64.exe" a -t7z -mx9 -md=29 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9Dict512.7z Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
7-Zip (a) [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04
Scanning the drive:
1 file, 133901432 bytes (128 MiB)
Creating archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9Dict512.7z
Items to compress: 1
Files read from disk: 1
Archive size: 34610398 bytes (34 MiB)
Everything is Ok
Kernel Time = 1.372 = 1%
User Time = 180.368 = 136%
Process Time = 181.741 = 137% Virtual Memory = 1356 MB
Global Time = 132.258 = 100% Physical Memory = 1293 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "xz_v5.2.3_x64.exe" -z -k -f -9 -e -v -v --lzma2=dict=512MiB --threads=1 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
xz_v5.2.3_x64: Filter chain: --lzma2=dict=512MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz_v5.2.3_x64: 5,378 MiB of memory is required. The limiter is disabled.
xz_v5.2.3_x64: Decompression will need 513 MiB of memory.
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv (1/1)
100 % 33.0 MiB / 127.7 MiB = 0.258 750 KiB/s 2:54
Kernel Time = 0.811 = 0%
User Time = 171.975 = 98%
Process Time = 172.786 = 99% Virtual Memory = 5389 MB
Global Time = 174.423 = 100% Physical Memory = 1525 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "bsc_v3.1.0_x64.exe" e Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.ST6Block512.bsc -b512 -m6 -cp -Tt
This is bsc, Block Sorting Compressor. Version 3.1.0. 8 July 2012.
Copyright (c) 2009-2012 Ilya Grebnov <[email protected]>.
Machine-Learning_www.kaggle.com_350000-movies-from-them compressed 133901432 into 28865952 in 11.420 seconds.
Kernel Time = 0.374 = 3%
User Time = 10.623 = 87%
Process Time = 10.998 = 90% Virtual Memory = 715 MB
Global Time = 12.197 = 100% Physical Memory = 709 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "zpaq_v7.05_x64.exe" add Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method29.zpaq Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv -method 29 -threads 1
zpaq v7.05 journaling archiver, compiled Apr 17 2015
Adding 133.901432 MB in 1 files -method 29 -threads 1 at 2017-12-02 08:53:06.
100.00% 0:00:00 + Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv 133901432
100.00% 0:00:00 [1..1898] 133909032 -method 29,121,1
1 +added, 0 -removed.
0.000000 + (133.901432 -> 133.901432 -> 41.293302) = 41.293302 MB
62.728 seconds (all OK)
Kernel Time = 0.234 = 0%
User Time = 60.559 = 96%
Process Time = 60.793 = 96% Virtual Memory = 1190 MB
Global Time = 62.769 = 100% Physical Memory = 774 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "zpaq_v7.05_x64.exe" add Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method59.zpaq Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv -method 59 -threads 1
zpaq v7.05 journaling archiver, compiled Apr 17 2015
Adding 133.901432 MB in 1 files -method 59 -threads 1 at 2017-12-02 08:54:09.
100.00% 0:00:00 + Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv 133901432
100.00% 0:00:00 [1..1898] 133909032 -method 59,121,1
1 +added, 0 -removed.
0.000000 + (133.901432 -> 133.901432 -> 26.153739) = 26.153739 MB
547.049 seconds (all OK)
Kernel Time = 0.483 = 0%
User Time = 544.100 = 99%
Process Time = 544.583 = 99% Virtual Memory = 2064 MB
Global Time = 547.053 = 100% Physical Memory = 1656 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 lz5 -49 -B7 --no-frame-crc Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
using blocks of size 262144 KB
Compressed filename will be : Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.lz5
Compressed 133901432 bytes into 45174349 bytes ==> 33.74%
Kernel Time = 0.202 = 0%
User Time = 161.554 = 98%
Process Time = 161.757 = 98% Virtual Memory = 676 MB
Global Time = 163.938 = 100% Physical Memory = 463 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "nanozip-0.09a.win64.exe" a -t1 -cc Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.cc.nz Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
NanoZip 0.09 alpha/Win64 (C) 2008-2011 Sami Runsas www.nanozip.net
Intel(R) Core(TM) i5-2430M CPU @ 2.40GHz|35487 MHz|#2+HT|14832/16332 MB
Archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.cc.nz
Threads: 1, memory: 512 MB
Compressor #0: nz_cm [524 MB]
Compressed 133 901 432 into 26 762 899 in 3m 16.03s, 667 KB/s
IO-in: 0.08s, 1502 MB/s. IO-out: 0.01s, 1702 MB/s
Kernel Time = 0.390 = 0%
User Time = 200.008 = 99%
Process Time = 200.398 = 99% Virtual Memory = 539 MB
Global Time = 200.953 = 100% Physical Memory = 474 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 "nanozip-0.09a.win64.exe" a -t1 -co Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.co.nz Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
NanoZip 0.09 alpha/Win64 (C) 2008-2011 Sami Runsas www.nanozip.net
Intel(R) Core(TM) i5-2430M CPU @ 2.40GHz|49756 MHz|#2+HT|14843/16332 MB
Archive: Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.co.nz
Threads: 1, memory: 512 MB
Compressor #0: nz_optimum1 [540 MB]
Compressed 133 901 432 into 27 087 613 in 16.23s, 8055 KB/s
IO-in: 0.10s, 1228 MB/s. IO-out: 0.13s, 187 MB/s
Kernel Time = 0.858 = 4%
User Time = 15.927 = 89%
Process Time = 16.785 = 94% Virtual Memory = 559 MB
Global Time = 17.731 = 100% Physical Memory = 407 MB
CABARC, Microsoft (R) Cabinet Tool - Version 5.1.2600.0, Copyright (c) Microsoft Corporation.
H:\smashshop_2018-Feb-12_Judaica>timer64 cabarc.exe -m LZX:21 N Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.LZX21.cab Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Microsoft (R) Cabinet Tool - Version 5.1.2600.0
Copyright (c) Microsoft Corporation. All rights reserved..
Creating new cabinet 'Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.LZX21.cab' with compression 'LZX:21':
-- adding Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Completed successfully
Kernel Time = 0.436 = 0%
User Time = 135.206 = 98%
Process Time = 135.642 = 98% Virtual Memory = 20 MB
Global Time = 137.652 = 100% Physical Memory = 21 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 cabarc.exe -m MSZIP N Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MSZIP.cab Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Microsoft (R) Cabinet Tool - Version 5.1.2600.0
Copyright (c) Microsoft Corporation. All rights reserved..
Creating new cabinet 'Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MSZIP.cab' with compression 'MSZIP':
-- adding Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Completed successfully
Kernel Time = 0.280 = 1%
User Time = 15.007 = 86%
Process Time = 15.288 = 88% Virtual Memory = 1 MB
Global Time = 17.293 = 100% Physical Memory = 3 MB
Compress, version: (N)compress 4.2.4.4, compiled: Fri, Aug 23, 2013 11:56:09. Authors: Peter Jannesen, Dave Mack, Spencer W. Thomas, Jim McKie, Steve Davies, Ken Turkowski, James A. Woods, Joe Orost.
H:\smashshop_2018-Feb-12_Judaica>timer64 compress.exe -c Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv 1>Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.Z
H:\smashshop_2018-Feb-12_Judaica>timer64 zstd-v1.3.3-win64.exe --ultra -22 --zstd=wlog=29,clog=30,hlog=30,slog=26 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv : 26.40% (133901432 => 35354048 bytes, Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.zst)
Kernel Time = 1.965 = 0%
User Time = 1353.917 = 99%
Process Time = 1355.882 = 99% Virtual Memory = 8339 MB
Global Time = 1359.043 = 100% Physical Memory = 8340 MB
H:\smashshop_2018-Feb-12_Judaica>timer64 rz_1.01.exe a -d 512M Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.512M.rz Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
*** RAZOR Archiver 1.01 (2017-09-14) - DEMO/TEST version ***
*** (c) Christian Martelock ([email protected]) ***
Scanning h:\smashshop_2018-feb-12_judaica\machine-learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
Found 0 dirs, 1 files, 133901432 bytes.
Creating archive Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.512M.rz
Window : 130764K (512M..128G)
Header : 98
Size : 29922927
Archive ok. Added 0 dirs, 1 files, 133901432 bytes.
CPU time = 639.230s / wall time = 477.908s
Kernel Time = 2.620 = 0%
User Time = 639.229 = 133%
Process Time = 641.850 = 134% Virtual Memory = 1600 MB
Global Time = 478.017 = 100% Physical Memory = 1494 MB
12/02/2017 01:03 AM 26,153,739 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method59.zpaq
12/02/2017 01:09 AM 26,762,899 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.cc.nz
12/02/2017 01:09 AM 27,087,613 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.co.nz
12/02/2017 12:53 AM 28,865,952 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.ST6Block512.bsc
12/02/2017 12:44 AM 29,087,569 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O6.PPMd_varI
12/02/2017 01:42 AM 29,922,927 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.512M.rz
12/02/2017 12:45 AM 30,120,412 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.O16.PPMd_varI
12/02/2017 12:49 AM 34,610,398 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9Dict512.7z
11/13/2017 12:08 AM 34,610,800 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.L9Dict512.xz
11/13/2017 12:08 AM 35,354,048 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.zst
12/02/2017 01:11 AM 40,245,342 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.LZX21.cab
12/02/2017 12:54 AM 41,293,302 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.method29.zpaq
11/13/2017 12:08 AM 45,174,349 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.M49Block256.lz5
12/02/2017 12:47 AM 52,319,186 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MX9.zip
12/02/2017 01:12 AM 54,300,130 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.MSZIP.cab
12/02/2017 01:12 AM 58,780,239 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv.Z
11/13/2017 12:08 AM 133,901,432 Machine-Learning_www.kaggle.com_350000-movies-from-themoviedb.org.csv
...
Above dump is only a candidate, I am open for suggestions as to what additional compressors/options to be added.
The package used is this:
smashshop_2018-Feb-13_Judaica.zip 226 MB (237,723,608 bytes)
I saw in turbobench.ini
the following line LZMA9Z lzma,9mt2:d29:a1:fb273:mf=bt4:mc999:lc8:lp0:pb2
What is the meaning of the colon :
and mf=
? Also where can i read about what things like pb
mean and what numbers after that are accepted?
In file included from turbobench.c:66:
time_.h:34:26: error: conflicting types for ‘uint64_t’; have ‘long long unsigned int’
34 | typedef unsigned __int64 uint64_t;
| ^~~~~~~~
In file included from /usr/include/sys/types.h:46,
from /usr/include/stdio.h:61,
from turbobench.c:31:
/usr/include/sys/_stdint.h:60:20: note: previous declaration of ‘uint64_t’ with type ‘uint64_t’ {aka ‘long unsigned int’}
60 | typedef __uint64_t uint64_t ;
| ^~~~~~~~
make: *** [makefile:726: turbobench.o] Error 1
I don't think Turbobench and GCC10 are friends
gcc version 10.2.1 20200723 releases/gcc-10.2.0-3-g677b80db41 (Clear Linux OS for Intel Architecture)
/usr/bin/ld: glza/GLZAformat.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here
/usr/bin/ld: glza/GLZAcompress.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here
/usr/bin/ld: glza/GLZAencode.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:45: multiple definition of `bin_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:37: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:39: multiple definition of `sum_nbob'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:38: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `max_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:43: multiple definition of `queue_offset'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:44: multiple definition of `queue'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:41: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size_az'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size_other'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:38: multiple definition of `queue_size_space'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:39: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:41: multiple definition of `prior_is_cap'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:41: multiple definition of `prior_end'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:46: multiple definition of `queue_miss_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:37: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:41: multiple definition of `use_mtf'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `UTF8_compliant'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `max_regular_code_length'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:37: multiple definition of `num_base_symbols'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:40: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:45: multiple definition of `symbol_lengths'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:37: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:42: multiple definition of `cap_symbol_defined'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:42: multiple definition of `cap_lock_symbol_defined'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:36: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZAdecode.c:40: multiple definition of `cap_encoded'; glza/GLZAencode.o:/root/TurboBench/glza/GLZAencode.c:35: first defined here
/usr/bin/ld: glza/GLZAdecode.o:/root/TurboBench/glza/GLZA.h:6: multiple definition of `params'; glza/GLZAcomp.o:/root/TurboBench/glza/GLZA.h:6: first defined here
file html8 : 100MB random html pages from a 1m Alexa Top sites corpus.
Number of pages = 1178
Average length = 84886 bytes
The pages (length + content) are concatenated into a single html8 file,
but compressed/decompressed separately using the multiblock mode in TurboBench.
compress: page1,page2,...pageN
decompress : Page1,page2,...pageN
In the speedup (see below) plots you can see the best compressors for content providers:
Remarks:
Unlike other benchmarks on the net, this is pure memory benchmark without any additional (http) server overhead.
Page Statistics:
Page Length: Minimum = 16kb, Maximum = 128kb
bits histogram:
15:######################## 24%
16:####################################### 39%
17:##################################### 37%
Lenovo Ideapad Pro 5 / CPU 7840HS 3.8-5.1GHz, DDR5 6400MHz
C Size | R% | C MB/s | D MB/s | Name | C Mem | D Mem | C Stack | D Stack |
---|---|---|---|---|---|---|---|---|
16457986 | 16.5 | 1.41 | 612.34 | brotli 11 | 10,629,680 | 247,672 | ||
18579017 | 18.6 | 60.35 | 703.56 | brotli 5 | 10,711,032 | 199,384 | ||
18996682 | 19.0 | 0.53 | 1787.41 | zstd 22 | 815,429,856 | 191,952 | ||
19615056 | 19.6 | 0.34 | 1003.78 | zopfli | 33,644,120 | 14,352 | ||
19766557 | 19.8 | 7.10 | 1661.60 | libdeflate 12 | 18,027,032 | 23,152 | ||
19971457 | 20.0 | 135.89 | 716.37 | brotli 4 | 10,153,192 | 198,904 | ||
19974358 | 20.0 | 5.64 | 2092.36 | zstd 15 | 72,091,872 | 191,952 | ||
20282366 | 20.3 | 96.65 | 1700.68 | libdeflate 9 | 1,336,592 | 23,152 | ||
20363869 | 20.4 | 55.00 | 687.49 | zlib 9 | 274,096 | 14,320 | ||
20451000 | 20.5 | 84.95 | 1130.76 | zlib_ng 9 | 36 GB | 778,098,680 | ||
20485533 | 20.5 | 92.93 | 682.52 | zlib 6 | 274,096 | 14,320 | ||
20502399 | 20.5 | 201.65 | 1702.04 | libdeflate 6 | 1,340,024 | 23,152 | ||
20568950 | 20.6 | 78.85 | 2004.33 | zstd 5 | 5,503,608 | 191,952 | ||
20592048 | 20.6 | 173.56 | 1141.23 | zlib_ng 6 | 36 GB! | 778,098,680 | ||
21624240 | 21.6 | 242.98 | 702.77 | brotli 2 | 9,235,608 | 231,792 | ||
22159165 | 22.2 | 434.23 | 1733.07 | libdeflate 1 | 407,544 | 23,152 | ||
22484852 | 22.5 | 370.72 | 1483.97 | igzip 3 | 696,336 | 0 | 7,340,032 | 7,340,032 |
22867303 | 22.9 | 374.92 | 2064.67 | zstd 1 | 1,374,840 | 191,952 | ||
23098360 | 23.1 | 766.27 | 1446.97 | igzip 2 | 663,568 | 0 | 7,340,032 | 7,340,032 |
23153034 | 23.2 | 518.39 | 678.09 | brotli 1 | 1,200,512 | 165,704 | ||
23287503 | 23.3 | 850.51 | 1444.42 | igzip 1 | 569,336 | 0 | 7,340,032 | 7,340,032 |
23723266 | 23.7 | 238.93 | 629.48 | zlib 1 | 274,096 | 14,320 | ||
25355327 | 25.4 | 982.05 | 1543.00 | igzip 0 | 1,032 | 0 | 7,340,032 | 7,340,032 |
28214601 | 28.2 | 573.26 | 692.20 | slz 6 | 0 | 14,320 | 65,536 | |
28214601 | 28.2 | 576.88 | 684.35 | slz 9 | 0 | 14,320 | 65,536 | |
29476316 | 29.5 | 587.78 | 679.40 | slz 1 | 1,032 | 14,320 | 65,536 | |
30040886 | 30.0 | 569.15 | 1078.03 | zlib_ng 1 | 25 GB | 778,098,680 |
R: Compression ratio
Mem: Heap Memory usage in bytes
Stack: Stack Memory usage in bytes
Zstd is added only as indication. There is actually no zstd content-encoding for the web
Running ./turbobench -eBWT ascii.txt
We can see we can do the first two items in the test:
1576 8.9 42.84 261.86 bzip2 ascii.txt
1347 7.6 40.68 0.03 bzip3 ascii.txt
But after that, we get a segfault:
Segmentation fault (core dumped)
Running Arch and the latest release.
It would be interesting to add codecs
libzlf
lzfx
lz88
aPlib
delta encoding
runlength encoding
zstandard
all but the last one are potentially useful in use on low power microcontrollers
After compiling TurboBench, the executable is not generated. How do I fix this?
There is in the readme a list of compressors https://github.com/powturbo/TurboBench#compressor-lz77rolzbwtzpaq i assume these can be tested with turbobench and also that they can be specified in turbobench.ini.
I also assume that a compressor like Intel(R) Intelligent Storage Acceleration Library
must be specified with a different name in the turbobench.ini
Could there be added a option to list all possible compressors? At first i thought -l1
or -l2
would print this list. But they print less entries than the 57 which can be found in the readme.
Test data: 0x0000
during encoding, the cum prob table is: 0, 2731
However, when decoding, the cum prob table is: 0, 4096 (because of scaled_cum_prob[a] = (1<<SSE_BITS))
Therefore, we should save the whole cum prob table and read it back when decoding, i.e.,
plugin.cc:891, useless now
plugin.cc:906, i < a + 1
plugin.cc:921, i < a + 1
plugin.cc:922, delete.
Hope this helps.
Dear @powturbo,
Could you be so kind to generate static .exe
for the rest of us who have no various lib*.dll
libraries?
version 2023-03 fails on linux, with compiler gcc (GCC) 12.2.0
$ gdb --args turbobench test-files.tar
(gdb) r
15.66 3_01
Program received signal SIGSEGV, Segmentation fault.
stackpeak (_sp=_sp@entry=0x0) at turbobench.c:249
249 for(sp = _sp - STACK_SIZE; *sp == STACK_MAGIC; sp++);
(gdb) bt
#0 stackpeak (_sp=_sp@entry=0x0) at turbobench.c:249
#1 0x0000000000576cff in plugfile (plug=plug@entry=0x6e8940 <plug>, finame=finame@entry=0x7fffffff9989 "test-files.tar", filenmax=filenmax@entry=1073741824, bsize=bsize@entry=1073741824, plugr=plugr@entry=0x6db980 <plugr>, tid=<optimized out>, krep=0) at turbobench.c:1249
#2 0x0000000000403365 in main (argc=2, argv=<optimized out>) at turbobench.c:1510
Lines 238 to 251 in 51e2961
replace -O3
with -O1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.