mateidavid / zstr Goto Github PK
View Code? Open in Web Editor NEWA C++ header-only ZLib wrapper
License: MIT License
A C++ header-only ZLib wrapper
License: MIT License
Lines 36 to 43 in 6811097
In the GNU-specific strerror_r() case, line 42 here is buggy. Because the string has been swapped with "tmp", it will not contain a null character, and then it will attempt to invalidly resize to be of size std::string::npos, which is of course undesirable. (It will of course still be null terminated, but std::string::find will not find a null character within the body of the string, which is the problem).
Line 24 in 6811097
Also, for what it's worth, 80 byte buffer here might be a bit on the small side, and may cause unnecessary truncation of errors on some OSes in some locales, at least this is my reading of searching online such as https://stackoverflow.com/questions/423248/what-size-should-i-allow-for-strerror-r#comment5466341_423301
Going to 256 or 512 might be preferable - it doesn't seem like there should be a need to be so extremely stingy with memory on modern systems.
(edit: fixed some wording, added some clarifications)
I am using msys2 gcc
and I getting crashes from this
and I use Dr. memory to get some info about that error and it showed me this before crashing
Error #20: UNADDRESSABLE ACCESS beyond top of stack: reading 0x00000097d2df3700-0x00000097d2df3708 8 byte(s)
# 0 .text [../../../gcc-10.3.0/libgcc/config/i386/cygwin.S:132]
# 1 d_demangle_callback.constprop.0 [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6314]
# 2 __cxa_demangle [C:\_\M\mingw-w64-gcc\src\build-x86_64-w64-mingw32\x86_64-w64-mingw32\libstdc++-v3\libsupc++/cp-demangle.c:6380]
# 3 __gnu_cxx::__verbose_terminate_handler [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/vterminate.cc:61]
# 4 __cxxabiv1::__terminate [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:48]
# 5 std::terminate [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:58]
# 6 __cxa_throw [../../../../gcc-10.3.0/libstdc++-v3/libsupc++/eh_throw.cc:95]
# 7 strict_fstream::detail::static_method_holder::check_open [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:153]
# 8 zstr::ofstream::ofstream [C:/enough_words_c_tcod/include/zstr/strict_fstream.hpp:212]
# 9 deload_tile [C:/enough_words_c_tcod/src/systems/saving/dynimic_tile_loader.hpp:111]
#10 fill_roads [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:133]
#11 fill_inered_roads [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:186]
#12 puploate_huge_tiles [C:/enough_words_c_tcod/src/systems/map/huge_map_fill.hpp:566]
#13 Create_huge_map::init [C:/enough_words_c_tcod/src/huge_map.cpp:69]
#14 Engine::handle_events [C:/enough_words_c_tcod/src/engine.cpp:14]
#15 SDL_main [C:/enough_words_c_tcod/src/main.cpp:45]
#16 main_getcmdline [../src/main/windows/SDL_windows_main.c:126]
#17 __tmainCRTStartup [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:321]
#18 .l_start [C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:202]
#19 KERNEL32.dll!BaseThreadInitThunk
Note: @0:11:20.399 in thread 26304
Note: 0x00000097d2df3700 refers to 200 byte(s) beyond the top of the stack 0x00000097d2df37c8
Note: instruction: or $0x0000000000000000 (%rcx) -> (%rcx)
my code works correctly in the debug mode and without any optimization but when I try it with the O3 that happen
I know that isn't that helpful and I willing to share any more info if that needed, also any idea of what the cause of it is helpful
Am I missing something?
zstr.zip/tgz doesn't seem to be cpp since some if not most the logical operators have been translated to their lexical counterparts, thus rendering the lib useless for use in e.g. vcpkg:
Example from near the beginning of strict_fstream.hpp
.....
line 63 res += (not res.empty()? "|" : "");
line 72 if ((mode & std::ios_base::trunc) and not (mode & std::ios_base::out))
...
and lot's more
so I am trying to build for android and android ndk use clang
and I am getting this error
C:\enough_words_c_tcod\include\zstr/strict_fstream.hpp:77:12: error: cannot initialize a variable of type 'char *' with
an rvalue of type 'int'
char * p = strerror_r(errno, &buff[0], buff.size());
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Via an extra parameter.
For analogy to zstr::ifstream that defaults to std::ifstream if input is not compressed.
Is there a way to set / control compression parameters when opening a stream?
[Nice library! At first I tried boost, as per your suggestion. But boost's filters are not completely general; they can only write to a "sink", not to a general ostream. So that meant I couldn't write to a boost::interprocess::ovectorstream, which is what I really wanted to use as my sink. Using zstr to write to a ovectorstream works like a charm!]
It appears that zstr does not open input streams in binary mode (std::ios_base::binary
):
explicit ifstream(const std::string& filename, std::ios_base::openmode mode = std::ios_base::in)
: detail::strict_fstream_holder< strict_fstream::ifstream >(filename, mode),
std::istream(new istreambuf(_fs.rdbuf()))
Although it does for the output stream:
explicit ofstream(const std::string& filename, std::ios_base::openmode mode = std::ios_base::out)
: detail::strict_fstream_holder< strict_fstream::ofstream >(filename, mode | std::ios_base::binary),
std::ostream(new ostreambuf(_fs.rdbuf()))
This might be a bug? When running on Linux, I haven't found any issues, but when compiling and running on Windows, non-binary mode appears to corrupt the bytes read in, presumably because in text mode it attempts to do some Windows-specific end-of-line character conversions. Adding std::ios_base::binary
to the ifstream mode fixes this.
tellg(), gcount() and seekg() functions might be helpful for estimating file size.
Hi,
I recently started playing with zstr in one of my projects and I noticed the library does not compile with GCC 4.8.2. I know that GCC 4.8.2 is reaaaaally old now, I think almost 10 years old, but unfortunately my users also use very old compilers...
Anyway, here is the issue (one issue at 2 places):
zstr.hpp: In member function ‘void zstr::ifstream::open(std::string, std::ios_base::openmode)’:
zstr.hpp:429:31: error: use of deleted function ‘std::basic_istream<char>& std::basic_istream<char>::operator=(const std::basic_istream<char>&)’
std::istream::operator=(std::istream(new istreambuf(_fs.rdbuf())));
zstr.hpp: In member function ‘void zstr::ofstream::open(std::string, std::ios_base::openmode, int)’:
zstr.hpp:467:31: error: use of deleted function ‘std::basic_ostream<char>& std::basic_ostream<char>::operator=(const std::basic_ostream<char>&)’
std::ostream::operator=(std::ostream(new ostreambuf(_fs.rdbuf(), default_buff_size, level)));
If I compile with GCC 5.4.0 everything is fine. Would there be any way to work around that?
Thank you,
Guillaume
I'm trying to use zstr::ifstream
with a CSV parser but I'm getting:
//from the IDE (clion)
In template: call to implicitly-deleted copy constructor of 'zstr::ifstream' error occurred here
in instantiation of member function 'csv::internals::StreamParser<zstr::ifstream>::StreamParser' requested here
in instantiation of function template specialization 'csv::CSVReader::CSVReader<zstr::ifstream, 0>' requested here
copy constructor of 'ifstream' is implicitly deleted because base
class 'detail::strict_fstream_holder<strict_fstream::ifstream>' has a deleted copy constructor
copy constructor of 'strict_fstream_holder<strict_fstream::ifstream>' is implicitly deleted because
field '_fs' has a deleted copy constructor copy constructor of 'ifstream' is implicitly deleted because
base class 'std::ifstream' (aka 'basic_ifstream<char>') has a deleted copy constructor 'basic_ifstream' has
been explicitly marked deleted here
//and from the compiler
lib/zstr/src/zstr.hpp:411:7: error: use of deleted function ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’
lib/zstr/src/zstr.hpp:400:8: note: ‘zstr::detail::strict_fstream_holder<strict_fstream::ifstream>::strict_fstream_holder(const zstr::detail::strict_fstream_holder<strict_fstream::ifstream>&)’ is implicitly deleted because the default definition would be ill-formed:
400 | struct strict_fstream_holder
| ^~~~~~~~~~~~~~~~~~~~~
lib/zstr/src/zstr.hpp:400:8: error: use of deleted function ‘strict_fstream::ifstream::ifstream(const strict_fstream::ifstream&)’
lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ifstream<_CharT, _Traits>::basic_ifstream(const std::basic_ifstream<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
lib/zstr/src/strict_fstream.hpp:177:7: error: use of deleted function ‘std::basic_ios<_CharT, _Traits>::basic_ios(const std::basic_ios<_CharT, _Traits>&) [with _CharT = char; _Traits = std::char_traits<char>]’
177 | class ifstream
| ^~~~~~~~
//this works
std::ifstream file(fp, std::ios::binary);
csv::CSVReader reader(file, csv_format);
//doesn't compile - getting the error above
zstr::ifstream file(fp, std::ios::binary);
csv::CSVReader reader(file, csv_format);
Any help appreciated.
Hello everyone,
I've been looking through the zstr.hpp
header and I can't seem to find out how to set the compression level in zstr::ofstream
?
I see zlib internally is tracking the compression level within the zstr::ostreambuf
's private zstrm_p
parameter. The zstr::ostreambuf
object is accessible from the rdbuf()
of the ofstream
, but the zstrm_p
of course is not.
This statement looks only for three specific possible zlib headers. There are many more valid zlib headers that would be missed. 61 to be precise. Look at RFC 1950 for the definition of a zlib header.
The various ifdef
s at the beginning of strict_fstream.hpp
that determine how to interact with strerror_r
break for me on a Mac. Specifically, it drops into the third, GNU-specific case, but it should be using the second one. I quickly fixed my local copy by adding defined(__APPLE__)
to the second case, but I'm not sure if it's the correct universal fix.
Their is no default constructor for zstr::fstream
, zstr::ofstream
and zstr::ifstream
. The standard file streams all have default constructors, would it be possible to add this behaviour?
Hi,
when i compile this library with a recent gcc (6.2.1) i get the following warning:
zstr.hpp: In destructor ‘virtual zstr::ostreambuf::~ostreambuf( ’:
zstr.hpp:271:79: warning: throw will always call terminate() [-Wterminate]
if (sync() != 0) throw Exception("~ostreambuf(): error in sink stream");
Cheers,
Georg
Hi,
A friend has been trying to compile a project of mine including zstr
on his Macbook (not the ARM ones but an older x86 version). Apparently, the compiler issues an error for the following line in strict_fstream.hpp
#elif ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600 || defined(__APPLE__)) && ! _GNU_SOURCE) || defined(__MUSL__)
The error is error: expected value in expression
and points to _GNU_SOURCE
. My friend solved the issue (temporarily) by removing the ! _GNU_SOURCE
. My short understanding of macros and preprocessor is that an undefined macro value initializes automatically to 0. So not sure what is the problem. Unfortunately, I do not have access to a Macbook so I cannot really test for it.
Thanks,
Guillaume
zstr.hpp:99 and zstr.hpp:103 use 15+32
and 12+16
as default window size. The semantics of this behavior are documented in https://zlib.net/manual.html, but I find this interface not very intuitive. It's not clear that gzip is used by default, which lead to subtle bugs in some project code.
I would appreciate if there were constants or separate parameters to make the used window size and header explicit. It would be more clear that the default value 15 + 16
refers to a 64KiB window with gzip header. Also, changing the compression header would not involve setting arbitrary numbers to the windowBits parameter.
I would propose adding an enum class type for the compression header:
enum class header {auto_detect, zlib, gzip, raw_deflate};
The streambuf classes get additional constructors which take a compression header argument and calculate the windowBits value based on the compression header and the desired window size (+32, +16 or negate window size).
The LBANN project that uses zstr fails to compile:
In file included from /usr/local/include/zstr.hpp:15,
from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/include/lbann/data_readers/sample_list_impl.hpp:57,
from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/include/lbann/data_readers/data_reader_sample_list_impl.hpp:32,
from /disk-samsung/freebsd-ports/misc/lbann/work/lbann-0.102-254-g2e3012a9a/src/data_readers/data_reader_smiles.cpp:29:
/usr/local/include/strict_fstream.hpp: In function 'std::string strict_fstream::strerror()':
/usr/local/include/strict_fstream.hpp:77:26: error: invalid conversion from 'int' to 'char*' [-fpermissive]
77 | char * p = strerror_r(errno, &buff[0], buff.size());
| ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| |
| int
gcc-11
FreeBSD 13.1
Hey guys,
I was wondering if it is possible to add GZIP compression functionality? The original zlib.h library contains code for GZIP compression, yet the wrapper does not.
Thanks!
When zstr.hpp
(and therefore strict_fstream.hpp
) is included in multiple files, there are multiple definitions of strerror
. This can easily be fixed by declaring it as static
, although I do not know if link-time optimization will de-duplicate it.
could you share an exemple of how to use miniz with zstr on windows?
The istreambuf
and ostreambuf
classes are given defaulted move constructors and assignment operators. However, they both inherit from std::streambuf
, which is not move constructible (see https://cplusplus.github.io/LWG/issue911 for some discussion), so the default declarations don't actually accomplish anything. Clang 8 flags this via the -Wdefaulted-function-deleted
warning (which is turned into an error by -Werror
).
One fix would be to remove the default declarations, since they don't accomplish anything (and I can put up a PR for this if it's deemed acceptable). Another would be to add proper move constructors and assignment operators, but that's beyond the scope of my current familiarity with the library.
A buffer is needed to hold the whole file after decompressing, how could I get the final file size?
It seems "seekg" and "tellg" could only get the original compressed size.
add useful changes from https://github.com/JGI-Bioinformatics/zstr/commits/master
How can i use zstr decompress char* or std::string data?
Could you please give me an example ?
Thanks!
https://github.com/tmaklin/bxzstr
See what things they share and if they can be merged
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.