Hello~. It seems the paq8qx project performs so excellent in compression ratio. But I want to implement it into
compress big data(.e.g., 1GB ) text compress. And the process speed is kB/s, which is too slow to practice in real.

Could you so nicely to give me some advises in how to optimize the speed in text compress?

here's a profile from https://software.intel.com/content/www/us/en/develop/tools/vtune-profiler.html

measured on an intel platform (avx2 build, gcc 9.2.0)

hopefully, it might be useful for someone to investigate where the bottlenecks are

paq8px.exe -8 data.dat // default CM
overview

paq8px.exe -8l data.dat // default CM
overview (ltsm)

paq8px.exe -8 data.dat // default CM
tree

paq8px.exe -8 data.dat // default CM
tree (ltsm)

Text block is not detected if data is like TEXT-DEFAULT-(...)

Hi again,
In order to optimize compression of small texts (and may be some email headers with no random data or keys, just words, separators and ascii numbers) for English, Spanish and Portuguese, what would be the best approach? Can I insert some dictionaries to paq8px?

g++ -v

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu116.04.5' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu116.04.5)

root@641d9216354d:/paq8px# g++ paq8px.cpp -DUNIX -static -o paq8px
paq8px.cpp:617:0: warning: "UNIX" redefined
#define UNIX //to compile for Unix, Linux, Solairs, MacOS / Darwin, etc)
^
:0:0: note: this is the location of the previous definition
paq8px.cpp:955:25: error: macro "putc" requires 2 arguments, but only 1 given
virtual void putc(U8 c) = 0;
^
paq8px.cpp:956:67: error: macro "putc" requires 2 arguments, but only 1 given
void append(const char* s) { for (int i = 0; s[i]; i++)putc(s[i]); }
^
paq8px.cpp:960:41: error: macro "putc" requires 2 arguments, but only 1 given
void put32(U32 x){putc((x >> 24) & 255); putc((x >> 16) & 255); putc((x >> 8) & 255); putc(x & 255);}
^
paq8px.cpp:960:64: error: macro "putc" requires 2 arguments, but only 1 given
void put32(U32 x){putc((x >> 24) & 255); putc((x >> 16) & 255); putc((x >> 8) & 255); putc(x & 255);}
^
paq8px.cpp:960:86: error: macro "putc" requires 2 arguments, but only 1 given
void put32(U32 x){putc((x >> 24) & 255); putc((x >> 16) & 255); putc((x >> 8) & 255); putc(x & 255);}
^
paq8px.cpp:960:101: error: macro "putc" requires 2 arguments, but only 1 given
void put32(U32 x){putc((x >> 24) & 255); putc((x >> 16) & 255); putc((x >> 8) & 255); putc(x & 255);}
^
paq8px.cpp:979:17: error: macro "putc" requires 2 arguments, but only 1 given
void putc(U8 c) { fputc(c, file); }
^
paq8px.cpp:1051:17: error: macro "putc" requires 2 arguments, but only 1 given
void putc(U8 c) {
^
paq8px.cpp:1061:27: error: macro "putc" requires 2 arguments, but only 1 given
(file_on_disk).putc(c);
^
paq8px.cpp:7017:47: error: macro "putc" requires 2 arguments, but only 1 given
if (mode==COMPRESS) archive->putc(x2>>24);
^
paq8px.cpp:7036:22: error: macro "putc" requires 2 arguments, but only 1 given
archive->putc(c);
^
paq8px.cpp:7088:25: error: macro "putc" requires 2 arguments, but only 1 given
archive->putc(x1>>24); // Flush first unequal byte of range
^
paq8px.cpp:7876:35: error: macro "putc" requires 2 arguments, but only 1 given
out->putc((blockresidual>>8)&255);
^
paq8px.cpp:7877:31: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(blockresidual &255);
^
paq8px.cpp:7940:18: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(g);
^
paq8px.cpp:7941:30: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?r:g-r);
^
paq8px.cpp:7942:30: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?b:g-b);
^
paq8px.cpp:7944:55: error: macro "putc" requires 2 arguments, but only 1 given
for (int j=0; j<width%3; j++) out->putc(in->getc());
^
paq8px.cpp:7957:32: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?r:b-r);
^
paq8px.cpp:7958:20: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(b);
^
paq8px.cpp:7959:32: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?g:b-g);
^
paq8px.cpp:7971:34: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(en.decompress());
^
paq8px.cpp:7990:18: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(g);
^
paq8px.cpp:7991:30: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?r:g-r);
^
paq8px.cpp:7992:30: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?b:g-b);
^
paq8px.cpp:7993:18: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(a);
^
paq8px.cpp:7995:55: error: macro "putc" requires 2 arguments, but only 1 given
for (int j=0; j<width%4; j++) out->putc(in->getc());
^
paq8px.cpp:8008:32: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8008:46: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8008:72: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8008:86: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8021:34: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(en.decompress());
^
paq8px.cpp:8092:42: error: macro "putc" requires 2 arguments, but only 1 given
if (mode==FDECOMPRESS) out->putc(c[5]);
^
paq8px.cpp:8241:29: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(diffCount[index]);
^
paq8px.cpp:8242:19: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(window);
^
paq8px.cpp:8243:18: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(index);
^
paq8px.cpp:8249:77: error: macro "putc" requires 2 arguments, but only 1 given
for (int i=0; i<diffCount[index]; i++) out->putc(diffByte[indexLIMIT+i+1]);
^
paq8px.cpp:8316:57: error: macro "putc" requires 2 arguments, but only 1 given
if (mode==FDECOMPRESS) out->putc(diffByte[diffIndex]);
^
paq8px.cpp:8468:23: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(hdrsize>>8);
^
paq8px.cpp:8469:24: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(hdrsize&255);
^
paq8px.cpp:8470:18: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(bsize);
^
paq8px.cpp:8471:24: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(clearpos>>8);
^
paq8px.cpp:8472:25: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(clearpos&255);
^
paq8px.cpp:8473:21: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(codesize);
^
paq8px.cpp:8509:61: error: macro "putc" requires 2 arguments, but only 1 given
if (code==maxcode+1) { if (phase==1) out->putc(j); else diffpos++; }
^
paq8px.cpp:8536:23: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(hdrsize>>8);
^
paq8px.cpp:8537:24: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(hdrsize&255);
^
paq8px.cpp:8538:22: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(255-bsize);
^
paq8px.cpp:8539:30: error: macro "putc" requires 2 arguments, but only 1 given
out->putc((clearpos>>8)&255);
^
paq8px.cpp:8540:25: error: macro "putc" requires 2 arguments, but only 1 given
out->putc(clearpos&255);
^
paq8px.cpp:8575:44: error: macro "putc" requires 2 arguments, but only 1 given
if (mode==FDECOMPRESS) out->putc(codesize);
^
paq8px.cpp:8602:37: error: macro "putc" requires 2 arguments, but only 1 given
if (mode==FDECOMPRESS) out->putc(0);
^
paq8px.cpp:8794:57: error: macro "putc" requires 2 arguments, but only 1 given
if (mode==FDECOMPRESS) out->putc(en.decompress());
^
paq8px.cpp:9068:22: error: macro "putc" requires 2 arguments, but only 1 given
archive->putc(0);
^
paq8px.cpp:9069:97: error: macro "putc" requires 2 arguments, but only 1 given
archive->putc(level | (trainEXE << 4) | (trainTXT << 5) | (adaptive << 6) | (skipRGB << 7));
^
paq8px.cpp: In member function 'void ProgramChecker::print() const':
paq8px.cpp:736:131: warning: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'long long unsigned int' [-Wformat=]
printf("Time %1.2f sec, used %I64u MB (%I64u bytes) of memory\n",double(clock()-start_time)/CLOCKS_PER_SEC,maxmem>>20,maxmem);
^
paq8px.cpp:736:131: warning: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'U64 {aka long long unsigned int}' [-Wformat=]
paq8px.cpp: In member function 'void Array<T, Align>::create(U32)':
paq8px.cpp:812:15: error: 'uintptr_t' was not declared in this scope
data=(T*)(((uintptr_t)ptr+pad) & ~(uintptr_t)pad);
^
paq8px.cpp:812:25: error: expected ')' before 'ptr'
data=(T*)(((uintptr_t)ptr+pad) & ~(uintptr_t)pad);
^
paq8px.cpp:812:52: error: expected ')' before ';' token
data=(T*)(((uintptr_t)ptr+pad) & ~(uintptr_t)pad);
^
paq8px.cpp: At global scope:
paq8px.cpp:950:21: warning: defaulted and deleted functions only available with -std=c++11 or -std=gnu++11
virtual ~File() = default;
^
paq8px.cpp:955:29: error: variable or field 'putc' declared void
virtual void putc(U8 c) = 0;
^
paq8px.cpp:979:37: error: variable or field 'putc' declared void
void putc(U8 c) { fputc(c, file); }
^
paq8px.cpp:979:37: error: expected ';' at end of member declaration
paq8px.cpp:1018:51: warning: non-static data member initializers only available with -std=c++11 or -std=gnu++11
const U32 MAX_RAM_FOR_TMP_CONTENT = 64 * 1024 * 1024; //64 MB (per file)
^
paq8px.cpp:1062:3: error: variable or field 'putc' declared void
}
^
paq8px.cpp:1062:3: error: expected ';' at end of member declaration
paq8px.cpp:1031:3: error: looser throw specifier for 'virtual FileTmp::~FileTmp()'
~FileTmp() {close();}
^
paq8px.cpp:950:11: error: overriding 'virtual File::~File() throw ()'
virtual ~File() = default;
^
paq8px.cpp: In copy constructor 'StationaryMap::StationaryMap(const StationaryMap&)':
paq8px.cpp:797:3: error: 'Array<T, Align>::Array(const Array<T, Align>&) [with T = unsigned int; int Align = 16]' is private
Array(const Array&); // no copy or assignment
^
paq8px.cpp:2082:7: error: within this context
class StationaryMap {
^
paq8px.cpp: In function 'void im24bitModel(Mixer&, int, int, int)':
paq8px.cpp:3684:146: note: synthesized method 'StationaryMap::StationaryMap(const StationaryMap&)' first required here
static StationaryMap Map[nMaps] = {12, 12, 12, 12, 10, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0};
^
paq8px.cpp:2087:3: note: after user-defined conversion: StationaryMap::StationaryMap(int, int, int)
StationaryMap(int BitsOfContext, int BitsPerContext = 8, int Rate = 0): Data(1<<(BitsOfContext+BitsPerContext)), Context(0), Mask(1<<BitsPerContext), bCount(1), B(1) {
^
paq8px.cpp: In function 'void wavModel(Mixer&, int, ModelStats*)':
paq8px.cpp:5125:23: warning: extended initializer lists only available with -std=c++11 or -std=gnu++11
static Array pr{32}; // [3][2]
^
paq8px.cpp:5125:27: error: in C++98 'pr' must be initialized by constructor, not by '{...}'
static Array pr{32}; // [3][2]
^
paq8px.cpp:5126:22: warning: extended initializer lists only available with -std=c++11 or -std=gnu++11
static Array n{2};
^
paq8px.cpp:5126:24: error: in C++98 'n' must be initialized by constructor, not by '{...}'
static Array n{2};
^
paq8px.cpp:5127:28: warning: extended initializer lists only available with -std=c++11 or -std=gnu++11
static Array counter{2};
^
paq8px.cpp:5127:30: error: in C++98 'counter' must be initialized by constructor, not by '{...}'
static Array counter{2};
^
paq8px.cpp:5128:25: warning: extended initializer lists only available with -std=c++11 or -std=gnu++11
static Array F{49492}; //[49][49][2]
^
paq8px.cpp:5128:33: error: in C++98 'F' must be initialized by constructor, not by '{...}'
static Array F{49492}; //[49][49][2]
^
paq8px.cpp:5129:25: warning: extended initializer lists only available with -std=c++11 or -std=gnu++11
static Array L{4949}; //[49][49]
^
paq8px.cpp:5129:31: error: in C++98 'L' must be initialized by constructor, not by '{...}'
static Array L{4949}; //[49][49]
^
paq8px.cpp: In member function 'void exeModel::Train()':
paq8px.cpp:5993:17: error: 'MAX_PATH' was not declared in this scope
char filename[MAX_PATH+1];
^
paq8px.cpp:5994:34: error: 'filename' was not declared in this scope
if ((i=GetModuleFileName(NULL, filename,MAX_PATH)) && i<=MAX_PATH){
^
paq8px.cpp:5994:51: error: 'GetModuleFileName' was not declared in this scope
if ((i=GetModuleFileName(NULL, filename,MAX_PATH)) && i<=MAX_PATH){
^
paq8px.cpp: In member function 'void ContextModel::Train()':
paq8px.cpp:6794:17: error: 'MAX_PATH' was not declared in this scope
char filename[MAX_PATH+12];
^
paq8px.cpp:6795:34: error: 'filename' was not declared in this scope
if ((i=GetModuleFileName(NULL, filename,MAX_PATH)) && i<=MAX_PATH){
^
paq8px.cpp:6795:51: error: 'GetModuleFileName' was not declared in this scope
if ((i=GetModuleFileName(NULL, filename,MAX_PATH)) && i<=MAX_PATH){
^
paq8px.cpp: In member function 'int Encoder::code(int)':
paq8px.cpp:7017:36: error: 'class File' has no member named 'putc'
if (mode==COMPRESS) archive->putc(x2>>24);
^
paq8px.cpp: In member function 'void Encoder::compress(int)':
paq8px.cpp:7036:16: error: 'class File' has no member named 'putc'
archive->putc(c);
^
paq8px.cpp: In member function 'void Encoder::flush()':
paq8px.cpp:7088:14: error: 'class File' has no member named 'putc'
archive->putc(x1>>24); // Flush first unequal byte of range
^
paq8px.cpp: In function 'void encode_cd(File*, File*, U64, int)':
paq8px.cpp:7876:8: error: 'class File' has no member named 'putc'
out->putc((blockresidual>>8)&255);
^
paq8px.cpp:7877:8: error: 'class File' has no member named 'putc'
out->putc(blockresidual &255);
^
paq8px.cpp: In function 'void encode_bmp(File*, File*, U64, int)':
paq8px.cpp:7940:12: error: 'class File' has no member named 'putc'
out->putc(g);
^
paq8px.cpp:7941:12: error: 'class File' has no member named 'putc'
out->putc(skipRGB?r:g-r);
^
paq8px.cpp:7942:12: error: 'class File' has no member named 'putc'
out->putc(skipRGB?b:g-b);
^
paq8px.cpp:7944:40: error: 'class File' has no member named 'putc'
for (int j=0; j<width%3; j++) out->putc(in->getc());
^
paq8px.cpp: In function 'U64 decode_bmp(Encoder&, U64, int, File*, FMode, U64&)':
paq8px.cpp:7957:14: error: 'class File' has no member named 'putc'
out->putc(skipRGB?r:b-r);
^
paq8px.cpp:7958:14: error: 'class File' has no member named 'putc'
out->putc(b);
^
paq8px.cpp:7959:14: error: 'class File' has no member named 'putc'
out->putc(skipRGB?g:b-g);
^
paq8px.cpp:7971:14: error: 'class File' has no member named 'putc'
out->putc(en.decompress());
^
paq8px.cpp: In function 'void encode_im32(File*, File*, U64, int)':
paq8px.cpp:7990:12: error: 'class File' has no member named 'putc'
out->putc(g);
^
paq8px.cpp:7991:12: error: 'class File' has no member named 'putc'
out->putc(skipRGB?r:g-r);
^
paq8px.cpp:7992:12: error: 'class File' has no member named 'putc'
out->putc(skipRGB?b:g-b);
^
paq8px.cpp:7993:12: error: 'class File' has no member named 'putc'
out->putc(a);
^
paq8px.cpp:7995:40: error: 'class File' has no member named 'putc'
for (int j=0; j<width%4; j++) out->putc(in->getc());
^
paq8px.cpp: In function 'U64 decode_im32(Encoder&, U64, int, File*, FMode, U64&)':
paq8px.cpp:8008:14: error: 'class File' has no member named 'putc'
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8008:40: error: 'class File' has no member named 'putc'
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8008:54: error: 'class File' has no member named 'putc'
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8008:80: error: 'class File' has no member named 'putc'
out->putc(skipRGB?r:b-r); out->putc(b); out->putc(skipRGB?g:b-g); out->putc(a);
^
paq8px.cpp:8021:14: error: 'class File' has no member named 'putc'
out->putc(en.decompress());
^
paq8px.cpp: In function 'U64 decode_exe(Encoder&, U64, File*, FMode, U64&)':
paq8px.cpp:8092:33: error: 'class File' has no member named 'putc'
if (mode==FDECOMPRESS) out->putc(c[5]);
^
paq8px.cpp: In function 'int encode_zlib(File*, File*, U64, int&)':
paq8px.cpp:8241:8: error: 'class File' has no member named 'putc'
out->putc(diffCount[index]);
^
paq8px.cpp:8242:8: error: 'class File' has no member named 'putc'
out->putc(window);
^
paq8px.cpp:8243:8: error: 'class File' has no member named 'putc'
out->putc(index);
^
paq8px.cpp:8249:47: error: 'class File' has no member named 'putc'
for (int i=0; i<diffCount[index]; i++) out->putc(diffByte[indexLIMIT+i+1]);
^
paq8px.cpp: In function 'int decode_zlib(File, U64, File*, FMode, U64&)':
paq8px.cpp:8316:33: error: 'class File' has no member named 'putc'
if (mode==FDECOMPRESS) out->putc(diffByte[diffIndex]);
^
paq8px.cpp: In function 'int encode_gif(File*, File*, U64, int&)':
paq8px.cpp:8468:8: error: 'class File' has no member named 'putc'
out->putc(hdrsize>>8);
^
paq8px.cpp:8469:8: error: 'class File' has no member named 'putc'
out->putc(hdrsize&255);
^
paq8px.cpp:8470:8: error: 'class File' has no member named 'putc'
out->putc(bsize);
^
paq8px.cpp:8471:8: error: 'class File' has no member named 'putc'
out->putc(clearpos>>8);
^
paq8px.cpp:8472:8: error: 'class File' has no member named 'putc'
out->putc(clearpos&255);
^
paq8px.cpp:8473:8: error: 'class File' has no member named 'putc'
out->putc(codesize);
^
paq8px.cpp:8509:55: error: 'class File' has no member named 'putc'
if (code==maxcode+1) { if (phase==1) out->putc(j); else diffpos++; }
^
paq8px.cpp:8536:8: error: 'class File' has no member named 'putc'
out->putc(hdrsize>>8);
^
paq8px.cpp:8537:8: error: 'class File' has no member named 'putc'
out->putc(hdrsize&255);
^
paq8px.cpp:8538:8: error: 'class File' has no member named 'putc'
out->putc(255-bsize);
^
paq8px.cpp:8539:8: error: 'class File' has no member named 'putc'
out->putc((clearpos>>8)&255);
^
paq8px.cpp:8540:8: error: 'class File' has no member named 'putc'
out->putc(clearpos&255);
^
paq8px.cpp: In function 'int decode_gif(File*, U64, File*, FMode, U64&)':
paq8px.cpp:8575:31: error: 'class File' has no member named 'putc'
if (mode==FDECOMPRESS) out->putc(codesize);
^
paq8px.cpp:8602:31: error: 'class File' has no member named 'putc'
if (mode==FDECOMPRESS) out->putc(0);
^
paq8px.cpp: In function 'void transform_encode_block(Filetype, File*, U64, Encoder&, int, char*, int, float, float, U64)':
paq8px.cpp:8671:68: warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'U64 {aka long long unsigned int}' [-Wformat=]
printf("Transform fails at %I64u, skipping...\n", diffFound-1);
^
paq8px.cpp: In function 'void compressRecursive(File*, U64, Encoder&, char*, int, float, float)':
paq8px.cpp:8731:150: warning: unknown conversion type character 'I' in format [-Wformat=]
printf(" %-11s | %-16s |%10I64u bytes [%I64u - %I64u]",blstr,typenames[(type==ZLIB && (info>>24)>=PNG8)?info>>24:type],len,begin,detected_end-1);
^
paq8px.cpp:8731:150: warning: format '%u' expects argument of type 'unsigned int', but argument 4 has type 'U64 {aka long long unsigned int}' [-Wformat=]
paq8px.cpp:8731:150: warning: format '%u' expects argument of type 'unsigned int', but argument 5 has type 'U64 {aka long long unsigned int}' [-Wformat=]
paq8px.cpp:8731:150: warning: too many arguments for format [-Wformat-extra-args]
paq8px.cpp: In function 'void compress(const char*, U64, Encoder&)':
paq8px.cpp:8762:76: warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'U64 {aka long long unsigned int}' [-Wformat=]
printf("Compressed from %I64u to %I64u bytes.\n",filesize,en.size()-start);
^
paq8px.cpp:8762:76: warning: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'U64 {aka long long unsigned int}' [-Wformat=]
paq8px.cpp: In function 'U64 decompressRecursive(File*, U64, Encoder&, FMode, int)':
paq8px.cpp:8794:37: error: 'class File' has no member named 'putc'
if (mode==FDECOMPRESS) out->putc(en.decompress());
^
paq8px.cpp: In function 'void decompress(const char*, U64, Encoder&)':
paq8px.cpp:8827:45: warning: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'U64 {aka long long unsigned int}' [-Wformat=]
printf(" %s %I64u -> ", filename, filesize);
^
paq8px.cpp:8832:63: warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'U64 {aka long long unsigned int}' [-Wformat=]
else if (mode==FCOMPARE && r) printf("differ at %I64u\n",r-1);
^
paq8px.cpp: In function 'int putsize(String&, String&, const char*, int)':
paq8px.cpp:8861:34: warning: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'U64 {aka long long unsigned int}' [-Wformat=]
sprintf(blk, "%I64u\t", len);
^
paq8px.cpp: In function 'int main(int, char**)':
paq8px.cpp:9068:16: error: 'class File' has no member named 'putc'
archive->putc(0);
^
paq8px.cpp:9069:16: error: 'class File' has no member named 'putc'
archive->putc(level | (trainEXE << 4) | (trainTXT << 5) | (adaptive << 6) | (skipRGB << 7));
^
paq8px.cpp:9102:48: warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'U64 {aka long long unsigned int}' [-Wformat=]
printf("\nFile list (%I64u bytes)\n", len);
^
paq8px.cpp:9113:75: warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'U64 {aka long long unsigned int}' [-Wformat=]
printf("Compressed from %I64u to %I64u bytes.\n",len,en.size()-start);
^
paq8px.cpp:9113:75: warning: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'U64 {aka long long unsigned int}' [-Wformat=]
paq8px.cpp:9156:87: warning: format '%u' expects argument of type 'unsigned int', but argument 5 has type 'long long unsigned int' [-Wformat=]
printf("\n%d/%d Filename: %s (%I64u bytes)\n", i+1, files, fname[i], fsize[i]);
^
paq8px.cpp:9160:87: warning: format '%u' expects argument of type 'unsigned int', but argument 2 has type 'U64 {aka long long unsigned int}' [-Wformat=]
printf("\nTotal %I64u bytes compressed to %I64u bytes.\n", total_size, en.size());
^
paq8px.cpp:9160:87: warning: format '%u' expects argument of type 'unsigned int', but argument 3 has type 'U64 {aka long long unsigned int}' [-Wformat=]

If you recommend the use of this project in production, is there a Python 3.7 interface that can be had for it? Thanks.

What is the maximum file size supported? Afaik it's mentioned nowhere.

It is possible to add the function to compress a folder like the version 124. Thank's.

Ideally 32 and 64 bit variants

I'm currently getting two errors. First is with cmake at build-linux.sh:

CMake Error: Parse error in command line argument: NDEBUG
 Should be: VAR:type=value

Then if just using "cmake -G "Unix Makefiles" ..", I get (Debian12 aarch64):

cc1plus: error: unknown value ‘nocona’ for ‘-march’
cc1plus: note: valid arguments are: armv8-a armv8.1-a armv8.2-a armv8.3-a armv8.4-a armv8.5-a armv8.6-a armv8.7-a armv8.8-a armv8-r armv9-a native

Properly adjusting the compiler flags paq8px works as expect in a Raspberry Pi.

I have this issue on 2 very different devices (one amd64 one i686), both running debian testing.

build$ cmake ..
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found ZLIB: /usr/lib/i386-linux-gnu/libz.so (found version "1.2.13") 
-- IPO / LTO enabled
-- Configuring done
-- Generating done
-- Build files have been written to: build
build$ make
[  1%] Building CXX object CMakeFiles/paq8px.dir/ProgramChecker.cpp.o
...
[ 98%] Building CXX object CMakeFiles/paq8px.dir/StateMapPair.cpp.o
[100%] Linking CXX executable paq8px
during GIMPLE pass: graphite
during GIMPLE pass: graphite
SmallStationaryContextMap.cpp: In member function ‘__ct_base .constprop’:
text/EnglishStemmer.cpp: In member function ‘isVowel’:
SmallStationaryContextMap.cpp:3:1: internal compiler error: Ungültiger Maschinenbefehl
    3 | SmallStationaryContextMap::SmallStationaryContextMap(const Shared* const sh, const int bitsOfContext, const int inputBits, const int rate, const int scale) :
      | ^
text/EnglishStemmer.cpp:526:6: internal compiler error: Ungültiger Maschinenbefehl
  526 | bool EnglishStemmer::isVowel(const char c) {
      |      ^
0x94d3bec internal_error(char const*, ...)
	???:0
0xb7cc0042 isl_ctx_alloc_with_options
	???:0
0xb7cc0272 isl_ctx_alloc
	???:0
0x9362f86 graphite_transform_loops()
	???:0
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <file:///usr/share/doc/gcc-12/README.Bugs> for instructions.
make[3]: *** [/tmp/ccKDW8Wc.mk:35: /tmp/ccA5LC32.ltrans11.ltrans.o] Fehler 1
make[3]: *** Es wird auf noch nicht beendete Prozesse gewartet....
0x94d3bec internal_error(char const*, ...)
	???:0
0xb7cc0042 isl_ctx_alloc_with_options
	???:0
0xb7cc0272 isl_ctx_alloc
	???:0
0x9362f86 graphite_transform_loops()
	???:0
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <file:///usr/share/doc/gcc-12/README.Bugs> for instructions.
make[3]: *** [/tmp/ccKDW8Wc.mk:26: /tmp/ccA5LC32.ltrans8.ltrans.o] Fehler 1
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/paq8px.dir/build.make:1586: paq8px] Fehler 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/paq8px.dir/all] Fehler 2
make: *** [Makefile:91: all] Fehler 2

On Ubuntu, it builds fine. So I suspect a bug in the debian variant of the link-time optimizer. But I have 0 experience with all of this so I hope you know what to do next.

Is any of these warnings worrying?

/home/rafael2k/files/rhizomatica/hermes/paq8px/ContextMap.cpp:22:27: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
   22 |     p[1 + ((c >> 5) & 1)] = 1 + ((c >> 4) & 1);
      |                           ^
/home/rafael2k/files/rhizomatica/hermes/paq8px/HashElementForContextMap.hpp:7:11: note: at offset 0 to object ‘bitState’ with size 1 declared here
    7 |   uint8_t bitState;  // state of bit0 (1st slot) / state of bit2 (2nd slot)  /  state of bit5 (in slot 2)
      |           ^
/home/rafael2k/files/rhizomatica/hermes/paq8px/ContextMap.cpp:23:27: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
   23 |     p[3 + ((c >> 4) & 3)] = 1 + ((c >> 3) & 1);
      |                           ^
/home/rafael2k/files/rhizomatica/hermes/paq8px/HashElementForContextMap.hpp:7:11: note: at offset 0 to object ‘bitState’ with size 1 declared here
    7 |   uint8_t bitState;  // state of bit0 (1st slot) / state of bit2 (2nd slot)  /  state of bit5 (in slot 2)
      |           ^
/home/rafael2k/files/rhizomatica/hermes/paq8px/ContextMap.cpp:26:27: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
   26 |     p[1 + ((c >> 2) & 1)] = 1 + ((c >> 1) & 1);
      |                           ^
/home/rafael2k/files/rhizomatica/hermes/paq8px/HashElementForContextMap.hpp:7:11: note: at offset 0 to object ‘bitState’ with size 1 declared here
    7 |   uint8_t bitState;  // state of bit0 (1st slot) / state of bit2 (2nd slot)  /  state of bit5 (in slot 2)
      |           ^
/home/rafael2k/files/rhizomatica/hermes/paq8px/ContextMap.cpp:27:27: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
   27 |     p[3 + ((c >> 1) & 3)] = 1 + (c & 1);
      |                           ^
/home/rafael2k/files/rhizomatica/hermes/paq8px/HashElementForContextMap.hpp:7:11: note: at offset 0 to object ‘bitState’ with size 1 declared here
    7 |   uint8_t bitState;  // state of bit0 (1st slot) / state of bit2 (2nd slot)  /  state of bit5 (in slot 2)
      |           ^

Hi! I'm not very experienced in building software myself and the thread in the README is outdated. When I try to compile using VS19 I get the following error:

1>LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library
1>ContextModel.obj : error LNK2001: unresolved external symbol "public: static unsigned char const * const DECAlpha::rel_op" (?rel_op@DECAlpha@@2QBEB)
1>ContextModel.obj : error LNK2001: unresolved external symbol "public: static enum DECAlpha::InstructionFormat const * const DECAlpha::formats" (?formats@DECAlpha@@2QBW4InstructionFormat@1@B)
1>paq8px.obj : error LNK2001: unresolved external symbol "public: static void __cdecl DECAlpha::Unshuffle(unsigned int &)" (?Unshuffle@DECAlpha@@SAXAEAI@Z)
1>paq8px.obj : error LNK2001: unresolved external symbol "public: static void __cdecl DECAlpha::Shuffle(unsigned int &)" (?Shuffle@DECAlpha@@SAXAEAI@Z)
1>paq8px.obj : error LNK2001: unresolved external symbol "public: static bool __cdecl DECAlpha::IsValidInstruction(unsigned int)" (?IsValidInstruction@DECAlpha@@SA_NI@Z)
1>C:\Users\snack\Downloads\paq8px-master\paq8px-master\x64\Release\paq8px.exe : fatal error LNK1120: 5 unresolved externals
1>Done building project "paq8px.vcxproj" -- FAILED.

I'm gonna assume that this is because of missing dependencies. Please add either more detailed build instructions or link a new build. Thank you!

Hi! Do you think you could post an example of using this compression directly on a char array? Something along the lines of: compress(char* buffer, int size, int compression_mode, int compression_type);

Thank you so much!

cmake ..
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
CMake Error at CMakeLists.txt:40 (add_executable):
Syntax error in cmake code at

/home/slava/Загрузки/paq8px-master/CMakeLists.txt:40

when parsing string

model\BlockModel.cpp

Invalid character escape '\B'.

CMake Error at CMakeLists.txt:40 (add_executable):
Syntax error in cmake code at

/home/slava/Загрузки/paq8px-master/CMakeLists.txt:40

when parsing string

model\ContextModelBlock.cpp

Invalid character escape '\C'.

-- Configuring incomplete, errors occurred!
See also "/home/slava/Загрузки/paq8px-master/build/CMakeFiles/CMakeOutput.log".

Hi,

Readme says that for windows I should just download the .exe file and run it in the folder where my files are. However I cannot find any .exe, neither here nor in kaitz's. Is there another way to install/ simple build on windows, perhaps with python?

Thanks a lot!

Hi all,
I'd like to use paq8px for email compression before over-the-air transmission on a production system. Do you think it might be safe? Today I use xz, but paq8px gives me significant gains, sometimes 20% more compression.