google / riegeli Goto Github PK

Riegeli/records is a file format for storing a sequence of string records, typically serialized protocol buffers.

License: Apache License 2.0

Python 1.94% C++ 94.45% Shell 0.05% Smarty 0.04% Starlark 3.49% C 0.03%

riegeli's Introduction

Riegeli

Riegeli/records is a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding.

See documentation.

Status

Riegeli file format will only change in a backward compatible way (i.e. future readers will understand current files, but current readers might not understand files using future features).

Riegeli C++ API might change in incompatible ways.

riegeli's People

Contributors

Stargazers

Watchers

riegeli's Issues

build error

Getting the following build error:

With clang-10

clang version 10.0.0
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

I'm getting the following error.

0x' -MD -MF bazel-out/k8-fastbuild/bin/python/riegeli/records/_objs/record_position_cc/record_position.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/python/riegeli/records/_objs/record_position_cc/record_position.pic.o' -fPIC -D__CLANG_SUPPORT_DYN_ANNOTATION__ -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/com_google_absl -iquote bazel-out/k8-fastbuild/bin/external/com_google_absl -iquote external/local_config_python -iquote bazel-out/k8-fastbuild/bin/external/local_config_python -iquote external/highwayhash -iquote bazel-out/k8-fastbuild/bin/external/highwayhash -isystem external/local_config_python/python_include -isystem bazel-out/k8-fastbuild/bin/external/local_config_python/python_include '-std=c++14' '-D_GLIBCXX_USE_CXX11_ABI=0' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c python/riegeli/records/record_position.cc -o bazel-out/k8-fastbuild/bin/python/riegeli/records/_objs/record_position_cc/record_position.pic.o)
ERROR: /home/ubuntu/repos/riegeli/riegeli/base/BUILD:75:1: C++ compilation of rule '//riegeli/base:buffer' failed (Exit 1) clang failed: error executing command /usr/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 23 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from riegeli/base/buffer.cc:15:
In file included from ./riegeli/base/buffer.h:26:
./riegeli/base/memory.h:149:5: error: no matching function for call to 'operator delete'
    operator delete(allocated, sizeof(void*) + num_bytes + alignment -
    ^~~~~~~~~~~~~~~
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/new:144:6: note: candidate function not viable: no known conversion from 'unsigned long' to 'const std::nothrow_t' for 2nd argument
void operator delete(void*, const std::nothrow_t&) _GLIBCXX_USE_NOEXCEPT
     ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/new:180:13: note: candidate function not viable: no known conversion from 'unsigned long' to 'void *' for 2nd argument
inline void operator delete  (void*, void*) _GLIBCXX_USE_NOEXCEPT { }
            ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/new:130:6: note: candidate function not viable: requires 1 argument, but 2 were provided
void operator delete(void*) _GLIBCXX_USE_NOEXCEPT
     ^
In file included from riegeli/base/buffer.cc:15:
./riegeli/base/buffer.h:100:25: error: no matching function for call to 'operator delete'
  if (data_ != nullptr) operator delete(data_, size_);
                        ^~~~~~~~~~~~~~~
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/new:180:13: note: candidate function not viable: no known conversion from 'size_t' (aka 'unsigned long') to 'void *' for 2nd argument; take the address of the argument with &
inline void operator delete  (void*, void*) _GLIBCXX_USE_NOEXCEPT { }
            ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/new:144:6: note: candidate function not viable: no known conversion from 'size_t' (aka 'unsigned long') to 'const std::nothrow_t' for 2nd argument
void operator delete(void*, const std::nothrow_t&) _GLIBCXX_USE_NOEXCEPT
     ^
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/new:130:6: note: candidate function not viable: requires 1 argument, but 2 were provided
void operator delete(void*) _GLIBCXX_USE_NOEXCEPT
     ^
2 errors generated.
Target //python:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.056s, Critical Path: 0.90s
INFO: 14 processes: 14 linux-sandbox.
FAILED: Build did NOT complete successfully

Probably because you specify the --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0

Riegeli depends on deprecated target @com_google_protobuf//:cc_wkt_protos

I'm getting annoying warnings from the Riegeli build depending on protobuf targets that were deprecated, like this:

WARNING: /home/blais/.cache/bazel/_bazel_blais/7d265136e440aae0298519930a834a34/external/com_google_riegeli/riegeli/records/BUILD:45:11: in cc_library rule @com_google_riegeli//riegeli/records:record_reader: target '@com_google_riegeli//riegeli/records:record_reader' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /home/blais/.cache/bazel/_bazel_blais/7d265136e440aae0298519930a834a34/external/com_google_riegeli/riegeli/records/BUILD:7:11: in cc_library rule @com_google_riegeli//riegeli/records:record_writer: target '@com_google_riegeli//riegeli/records:record_writer' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.

Is it possible to update the targets to link against non-deprecated code?

Build failure with C++17 w/ Clang

Use --sandbox_debug to see verbose messages from the sandbox
In file included from external/com_google_riegeli/riegeli/chunk_encoding/chunk.cc:15:
In file included from external/com_google_riegeli/riegeli/chunk_encoding/chunk.h:23:
external/com_google_riegeli/riegeli/base/base.h:607:1: error: no template named 'type_identity_t' in namespace 'riegeli'; did you mean 'internal::type_identity_t'?
RIEGELI_INTERNAL_INLINE_CONSTEXPR(size_t, kDefaultBufferSize, size_t{64} << 10);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:287:20: note: expanded from macro 'RIEGELI_INTERNAL_INLINE_CONSTEXPR'
  inline constexpr ::riegeli::type_identity_t<type> name = init
                   ^~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:276:1: note: 'internal::type_identity_t' declared here
using type_identity_t = typename type_identity<T>::type;
^
external/com_google_riegeli/riegeli/base/base.h:610:1: error: no template named 'type_identity_t' in namespace 'riegeli'; did you mean 'internal::type_identity_t'?
RIEGELI_INTERNAL_INLINE_CONSTEXPR(size_t, kMinBufferSize, 256);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:287:20: note: expanded from macro 'RIEGELI_INTERNAL_INLINE_CONSTEXPR'
  inline constexpr ::riegeli::type_identity_t<type> name = init
                   ^~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:276:1: note: 'internal::type_identity_t' declared here
using type_identity_t = typename type_identity<T>::type;
^
external/com_google_riegeli/riegeli/base/base.h:611:1: error: no template named 'type_identity_t' in namespace 'riegeli'; did you mean 'internal::type_identity_t'?
RIEGELI_INTERNAL_INLINE_CONSTEXPR(size_t, kMaxBufferSize, size_t{64} << 10);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:287:20: note: expanded from macro 'RIEGELI_INTERNAL_INLINE_CONSTEXPR'
  inline constexpr ::riegeli::type_identity_t<type> name = init
                   ^~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:276:1: note: 'internal::type_identity_t' declared here
using type_identity_t = typename type_identity<T>::type;
^
external/com_google_riegeli/riegeli/base/base.h:619:1: error: no template named 'type_identity_t' in namespace 'riegeli'; did you mean 'internal::type_identity_t'?
RIEGELI_INTERNAL_INLINE_CONSTEXPR(size_t, kMaxBytesToCopy, 255);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:287:20: note: expanded from macro 'RIEGELI_INTERNAL_INLINE_CONSTEXPR'
  inline constexpr ::riegeli::type_identity_t<type> name = init
                   ^~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:276:1: note: 'internal::type_identity_t' declared here
using type_identity_t = typename type_identity<T>::type;
^
In file included from external/com_google_riegeli/riegeli/chunk_encoding/chunk.cc:15:
In file included from external/com_google_riegeli/riegeli/chunk_encoding/chunk.h:29:
external/com_google_riegeli/riegeli/chunk_encoding/constants.h:43:1: error: no template named 'type_identity_t' in namespace 'riegeli'; did you mean 'internal::type_identity_t'?
RIEGELI_INTERNAL_INLINE_CONSTEXPR(uint64_t, kMaxNumRecords,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:287:20: note: expanded from macro 'RIEGELI_INTERNAL_INLINE_CONSTEXPR'
  inline constexpr ::riegeli::type_identity_t<type> name = init
                   ^~~~~~~~~~~
external/com_google_riegeli/riegeli/base/base.h:276:1: note: 'internal::type_identity_t' declared here
using type_identity_t = typename type_identity<T>::type;
^
5 errors generated.
Target //experiments/v3/protos:export_as_protos failed to build
Use --verbose_failures to see the command lines of failed build steps.

I haven't looked into the details yet.
C++14 w/ Clang also fails to build (differently), as well as C++17 w/ gcc.

build error

I run command: bazel build //python:build_pip_package

and get the following errors:

ERROR: /home/libin/Downloads/riegeli-master/riegeli/bytes/BUILD:438:1: C++ compilation of rule '//riegeli/bytes:cord_writer' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 23 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
riegeli/bytes/cord_writer.cc:37:31: error: conflicting declaration 'constexpr const int riegeli::CordWriterBase::kShortBufferSize'
constexpr int CordWriterBase::kShortBufferSize;
^
In file included from riegeli/bytes/cord_writer.cc:15:0:
./riegeli/bytes/cord_writer.h:123:27: note: previous declaration as 'constexpr const size_t riegeli::CordWriterBase::kShortBufferSize'
static constexpr size_t kShortBufferSize = 64;
^
riegeli/bytes/cord_writer.cc:37:31: error: declaration of 'constexpr const size_t riegeli::CordWriterBase::kShortBufferSize' outside of class is not definition [-fpermissive]
constexpr int CordWriterBase::kShortBufferSize;
^
Target //python:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 91.754s, Critical Path: 12.92s
INFO: 479 processes: 479 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

Riegeli is incompatible with --incompatible_load_proto_rules_from_bzl

This should be pretty straightfoward to fix, quoting from kythe/kythe#4264 -- the bug Bazel filed against us (Kythe):

The Bazel team is in progress of migrating the native Protobuf rules to Starlark. As a first step towards this goal, starting with Bazel 3.0, all Protobuf rules will require explicit load statements.

You can use the following buildifier command to apply most of the required migrations for you:

buildifier --lint=fix --warnings=native-proto $(find . -name "BUILD" -o -name "BUILD.bazel" -o -name "*.bzl")
For further information, see bazelbuild/bazel#8922 or ping me.

bazel build python:build_pip_package fails

The C++ build is now working, but I can't build the python package. It's failing with the following error:

$ bazel build python:build_pip_package
INFO: Repository local_config_tf instantiated at:
  /mnt/c/Users/miles/Development/riegeli/WORKSPACE:124:13: in <toplevel>
Repository rule tf_configure defined at:
  /mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl:206:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_tf':
   Traceback (most recent call last):
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 177, column 46, in _tf_pip_impl
                tf_header_rule = _symlink_genrule_for_dir(
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 147, column 29, in _symlink_genrule_for_dir
                sorted(_read_dir(repository_ctx, src_dir).splitlines()),
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 83, column 31, in _read_dir
                find_result = _execute(
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 50, column 14, in _execute
                _fail("\n".join([
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 20, column 9, in _fail
                fail("{}Python Configuration Error:{} {}\n".format(red, no_color, msg))
Error in fail: Python Configuration Error: Repository command failed
find: ‘’: No such file or directory

ERROR: /mnt/c/Users/miles/Development/riegeli/WORKSPACE:124:13: fetching tf_configure rule //external:local_config_tf: Traceback (most recent call last):
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 177, column 46, in _tf_pip_impl
                tf_header_rule = _symlink_genrule_for_dir(
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 147, column 29, in _symlink_genrule_for_dir
                sorted(_read_dir(repository_ctx, src_dir).splitlines()),
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 83, column 31, in _read_dir
                find_result = _execute(
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 50, column 14, in _execute
                _fail("\n".join([
        File "/mnt/c/Users/miles/Development/riegeli/tf_dependency/tf_configure.bzl", line 20, column 9, in _fail
                fail("{}Python Configuration Error:{} {}\n".format(red, no_color, msg))
Error in fail: Python Configuration Error: Repository command failed
find: ‘’: No such file or directory

ERROR: /mnt/c/Users/miles/Development/riegeli/python/riegeli/tensorflow/BUILD:11:10: //python/riegeli/tensorflow:ops/_riegeli_dataset_ops.so depends on @local_config_tf//:libtensorflow_framework in repository @local_config_tf which failed to fetch. no such package '@local_config_tf//': Python Configuration Error: Repository command failed
find: ‘’: No such file or directory

ERROR: /mnt/c/Users/miles/Development/riegeli/python/riegeli/tensorflow/BUILD:11:10: //python/riegeli/tensorflow:ops/_riegeli_dataset_ops.so depends on @local_config_tf//:tf_header_lib in repository @local_config_tf which failed to fetch. no such package '@local_config_tf//': Python Configuration Error: Repository command failed
find: ‘’: No such file or directory

ERROR: Analysis of target '//python:build_pip_package' failed; build aborted:
INFO: Elapsed time: 0.625s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (12 packages loaded, 269 targets configured)

Build error: clang w/ c++17

Hi,
I'm having a build error on C++17 w/ Clang:

cd ~/p/beancount/ && bazel build //experiments/v3/serialization/...
Loading:
Loading: 0 packages loaded
Analyzing: target //experiments/v3/serialization:riegeli_print (0 packages loaded, 0 targets configured)
INFO: Analyzed target //experiments/v3/serialization:riegeli_print (10 packages loaded, 267 targets configured).
INFO: Found 1 target...
bazel: Entering directory `/home/blais/.cache/bazel/_bazel_blais/7d265136e440aae0298519930a834a34/execroot/beancount/'
[22 / 514] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[397 / 601] Compiling experiments/v3/serialization/riegeli_print.cc; 1s linux-sandbox ... (4 actions running)
[406 / 605] Compiling experiments/v3/serialization/riegeli_print.cc; 3s linux-sandbox ... (4 actions running)
[413 / 607] Compiling external/com_google_riegeli/riegeli/bytes/brotli_reader.cc; 2s linux-sandbox ... (4 actions, 3 running)
ERROR: /home/blais/.cache/bazel/_bazel_blais/7d265136e440aae0298519930a834a34/external/com_google_riegeli/riegeli/base/BUILD:76:11: C++ compilation of rule '@com_google_riegeli//riegeli/base:buffer' failed (Exit 1) clang failed: error executing command /usr/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 21 \
argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from external/com_google_riegeli/riegeli/base/buffer.cc:15:
external/com_google_riegeli/riegeli/base/buffer.h:99:25: error: no matching function for call to 'operator delete'
  if (data_ != nullptr) operator delete(data_, size_);
                        ^~~~~~~~~~~~~~~
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/new:174:13: note: candidate function not viable: no known conversion from 'size_t' (aka 'unsigned long') to 'void *' for 2nd argument; take the address of the argument with &
inline void operator delete  (void*, void*) _GLIBCXX_USE_NOEXCEPT { }
            ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/new:138:6: note: candidate function not viable: no known conversion from 'size_t' (aka 'unsigned long') to 'const std::nothrow_t' for 2nd argument
void operator delete(void*, const std::nothrow_t&) _GLIBCXX_USE_NOEXCEPT
     ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/new:147:6: note: candidate function not viable: no known conversion from 'size_t' (aka 'unsigned long') to 'std::align_val_t' for 2nd argument
void operator delete(void*, std::align_val_t)
     ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/new:124:6: note: candidate function not viable: requires 1 argument, but 2 were provided
void operator delete(void*) _GLIBCXX_USE_NOEXCEPT
     ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/new:149:6: note: candidate function not viable: requires 3 arguments, but 2 were provided
void operator delete(void*, std::align_val_t, const std::nothrow_t&)
     ^
1 error generated.
bazel: Leaving directory `/home/blais/.cache/bazel/_bazel_blais/7d265136e440aae0298519930a834a34/execroot/beancount/'
Target //experiments/v3/serialization:riegeli_print failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.382s, Critical Path: 4.36s
INFO: 6 processes: 6 linux-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

To reproduce:

git checkout https://github.com/beancount/beancount
cd beancount
bazel build //experiments/v3/serialization/...

Note: compiler and std is set in beancount/.bazelrc.

Thank you for open sourcing Riegeli!

Add release tags, or release to pypi?

It would be very useful for riegeli to expose release tags of some form, even if only a calver-ish tag such as v2022.11.21.
We could then provide python library builds via conda-forge.

Non-tfrecord build

It would be good to limit the dependencies on tfrecords since a lot people might only want to use this independently. Maybe have a config_setting for it.

It would also be good to include examples for building a .so for non-bazel users. (Maybe a makefile?)

C++ version failing to build

I'm synced to commit faac845.

I'm trying to build riegeli to try it out, but it's failing to build with the following error:

$ bazel build riegeli/records:all
INFO: Analyzed 9 targets (0 packages loaded, 0 targets configured).
INFO: Found 9 targets...
ERROR: /mnt/c/Users/miles/Development/riegeli/riegeli/chunk_encoding/BUILD:219:11: Compiling riegeli/chunk_encoding/transpose_decoder.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 35 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from ./riegeli/chunk_encoding/decompressor.h:28,
                 from riegeli/chunk_encoding/transpose_decoder.cc:44:
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::ChainReader<> >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::ChainReader<>]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::ChainReader<>; ManagerArgs = {riegeli::Chain*&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:187:5:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain*&&>; Src = riegeli::ChainReader<>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain*&&}; Src = riegeli::ChainReader<>]'
riegeli/chunk_encoding/transpose_decoder.cc:501:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::BrotliReader<riegeli::ChainReader<> > >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::BrotliReader<riegeli::ChainReader<> >]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::BrotliReader<riegeli::ChainReader<> >; ManagerArgs = {riegeli::ChainReader<const riegeli::Chain*>&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:202:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain*&&>; Src = riegeli::ChainReader<>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain*&&}; Src = riegeli::ChainReader<>]'
riegeli/chunk_encoding/transpose_decoder.cc:501:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::ZstdReader<riegeli::ChainReader<> > >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::ZstdReader<riegeli::ChainReader<> >]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::ZstdReader<riegeli::ChainReader<> >; ManagerArgs = {riegeli::ChainReader<const riegeli::Chain*>&&, riegeli::ZstdReaderBase::Options&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:206:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain*&&>; Src = riegeli::ChainReader<>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain*&&}; Src = riegeli::ChainReader<>]'
riegeli/chunk_encoding/transpose_decoder.cc:501:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::SnappyReader<riegeli::ChainReader<> > >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::SnappyReader<riegeli::ChainReader<> >]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::SnappyReader<riegeli::ChainReader<> >; ManagerArgs = {riegeli::ChainReader<const riegeli::Chain*>&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:211:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain*&&>; Src = riegeli::ChainReader<>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain*&&}; Src = riegeli::ChainReader<>]'
riegeli/chunk_encoding/transpose_decoder.cc:501:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::Reader*>':
./riegeli/base/any_dependency.h:454:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::Reader*]'
./riegeli/base/any_dependency.h:454:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(Manager&&) [with Manager = riegeli::Reader*; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:187:5:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = riegeli::Reader*; Src = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:171:3:   required from 'void riegeli::internal::Decompressor<Src>::Reset(Src&&, riegeli::CompressionType) [with Src = riegeli::Reader*]'
riegeli/chunk_encoding/transpose_decoder.cc:746:59:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::BrotliReader<riegeli::Reader*> >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::BrotliReader<riegeli::Reader*>]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::BrotliReader<riegeli::Reader*>; ManagerArgs = {riegeli::Reader*&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:202:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = riegeli::Reader*; Src = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:171:3:   required from 'void riegeli::internal::Decompressor<Src>::Reset(Src&&, riegeli::CompressionType) [with Src = riegeli::Reader*]'
riegeli/chunk_encoding/transpose_decoder.cc:746:59:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::ZstdReader<riegeli::Reader*> >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::ZstdReader<riegeli::Reader*>]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::ZstdReader<riegeli::Reader*>; ManagerArgs = {riegeli::Reader*&&, riegeli::ZstdReaderBase::Options&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:206:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = riegeli::Reader*; Src = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:171:3:   required from 'void riegeli::internal::Decompressor<Src>::Reset(Src&&, riegeli::CompressionType) [with Src = riegeli::Reader*]'
riegeli/chunk_encoding/transpose_decoder.cc:746:59:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::SnappyReader<riegeli::Reader*> >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::SnappyReader<riegeli::Reader*>]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::SnappyReader<riegeli::Reader*>; ManagerArgs = {riegeli::Reader*&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:211:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = riegeli::Reader*; Src = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:171:3:   required from 'void riegeli::internal::Decompressor<Src>::Reset(Src&&, riegeli::CompressionType) [with Src = riegeli::Reader*]'
riegeli/chunk_encoding/transpose_decoder.cc:746:59:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::ChainReader<riegeli::Chain> >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::ChainReader<riegeli::Chain>]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::ChainReader<riegeli::Chain>; ManagerArgs = {riegeli::Chain&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:187:5:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain&&>; Src = riegeli::ChainReader<riegeli::Chain>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain&&}; Src = riegeli::ChainReader<riegeli::Chain>]'
/usr/include/c++/9/ext/new_allocator.h:147:4:   required from 'void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >]'
/usr/include/c++/9/bits/alloc_traits.h:484:4:   required from 'static void std::allocator_traits<std::allocator<_Tp1> >::construct(std::allocator_traits<std::allocator<_Tp1> >::allocator_type&, _Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; std::allocator_traits<std::allocator<_Tp1> >::allocator_type = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
/usr/include/c++/9/bits/vector.tcc:115:30:   required from 'void std::vector<_Tp, _Alloc>::emplace_back(_Args&& ...) [with _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Alloc = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
riegeli/chunk_encoding/transpose_decoder.cc:796:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::BrotliReader<riegeli::ChainReader<riegeli::Chain> > >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::BrotliReader<riegeli::ChainReader<riegeli::Chain> >]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::BrotliReader<riegeli::ChainReader<riegeli::Chain> >; ManagerArgs = {riegeli::ChainReader<riegeli::Chain>&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:202:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain&&>; Src = riegeli::ChainReader<riegeli::Chain>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain&&}; Src = riegeli::ChainReader<riegeli::Chain>]'
/usr/include/c++/9/ext/new_allocator.h:147:4:   required from 'void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >]'
/usr/include/c++/9/bits/alloc_traits.h:484:4:   required from 'static void std::allocator_traits<std::allocator<_Tp1> >::construct(std::allocator_traits<std::allocator<_Tp1> >::allocator_type&, _Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; std::allocator_traits<std::allocator<_Tp1> >::allocator_type = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
/usr/include/c++/9/bits/vector.tcc:115:30:   required from 'void std::vector<_Tp, _Alloc>::emplace_back(_Args&& ...) [with _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Alloc = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
riegeli/chunk_encoding/transpose_decoder.cc:796:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::ZstdReader<riegeli::ChainReader<riegeli::Chain> > >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::ZstdReader<riegeli::ChainReader<riegeli::Chain> >]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::ZstdReader<riegeli::ChainReader<riegeli::Chain> >; ManagerArgs = {riegeli::ChainReader<riegeli::Chain>&&, riegeli::ZstdReaderBase::Options&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:206:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain&&>; Src = riegeli::ChainReader<riegeli::Chain>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain&&}; Src = riegeli::ChainReader<riegeli::Chain>]'
/usr/include/c++/9/ext/new_allocator.h:147:4:   required from 'void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >]'
/usr/include/c++/9/bits/alloc_traits.h:484:4:   required from 'static void std::allocator_traits<std::allocator<_Tp1> >::construct(std::allocator_traits<std::allocator<_Tp1> >::allocator_type&, _Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; std::allocator_traits<std::allocator<_Tp1> >::allocator_type = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
/usr/include/c++/9/bits/vector.tcc:115:30:   required from 'void std::vector<_Tp, _Alloc>::emplace_back(_Args&& ...) [with _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Alloc = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
riegeli/chunk_encoding/transpose_decoder.cc:796:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
./riegeli/base/any_dependency.h: In instantiation of 'struct riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, riegeli::SnappyReader<riegeli::ChainReader<riegeli::Chain> > >':
./riegeli/base/any_dependency.h:463:14:   recursively required by substitution of 'template<class Manager> struct riegeli::AnyDependency<riegeli::Reader*>::MethodsFor<Manager, typename std::enable_if<riegeli::internal::AnyDependencyIsInline<riegeli::Reader*, Manager>::value, void>::type> [with Manager = riegeli::SnappyReader<riegeli::ChainReader<riegeli::Chain> >]'
./riegeli/base/any_dependency.h:463:14:   required from 'void riegeli::AnyDependency<Ptr>::Reset(std::tuple<_Args2 ...>) [with Manager = riegeli::SnappyReader<riegeli::ChainReader<riegeli::Chain> >; ManagerArgs = {riegeli::ChainReader<riegeli::Chain>&&}; Ptr = riegeli::Reader*]'
./riegeli/chunk_encoding/decompressor.h:211:7:   required from 'void riegeli::internal::Decompressor<Src>::Initialize(SrcInit&&, riegeli::CompressionType) [with SrcInit = std::tuple<riegeli::Chain&&>; Src = riegeli::ChainReader<riegeli::Chain>]'
./riegeli/chunk_encoding/decompressor.h:134:3:   required from 'riegeli::internal::Decompressor<Src>::Decompressor(std::tuple<_Args1 ...>, riegeli::CompressionType) [with SrcArgs = {riegeli::Chain&&}; Src = riegeli::ChainReader<riegeli::Chain>]'
/usr/include/c++/9/ext/new_allocator.h:147:4:   required from 'void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >]'
/usr/include/c++/9/bits/alloc_traits.h:484:4:   required from 'static void std::allocator_traits<std::allocator<_Tp1> >::construct(std::allocator_traits<std::allocator<_Tp1> >::allocator_type&, _Up*, _Args&& ...) [with _Up = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; std::allocator_traits<std::allocator<_Tp1> >::allocator_type = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
/usr/include/c++/9/bits/vector.tcc:115:30:   required from 'void std::vector<_Tp, _Alloc>::emplace_back(_Args&& ...) [with _Args = {std::tuple<riegeli::Chain&&>, riegeli::CompressionType&}; _Tp = riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> >; _Alloc = std::allocator<riegeli::internal::Decompressor<riegeli::ChainReader<riegeli::Chain> > >]'
riegeli/chunk_encoding/transpose_decoder.cc:796:63:   required from here
./riegeli/base/any_dependency.h:253:8: error: 'template<class Manager, class Enable> struct riegeli::AnyDependency<riegeli::Reader*>::IsInline' is private within this context
  253 | struct AnyDependencyIsInline : AnyDependency<Ptr>::template IsInline<Manager> {
      |        ^~~~~~~~~~~~~~~~~~~~~
./riegeli/base/any_dependency.h:167:10: note: declared private here
  167 |   struct IsInline;
      |          ^~~~~~~~
In file included from ./riegeli/chunk_encoding/transpose_decoder.h:27,
                 from riegeli/chunk_encoding/transpose_decoder.cc:15:
./riegeli/chunk_encoding/transpose_internal.h: In function 'bool riegeli::internal::HasSubtype(uint32_t)':
./riegeli/chunk_encoding/transpose_internal.h:119:1: warning: control reaches end of non-void function [-Wreturn-type]
  119 | }
      | ^
./riegeli/chunk_encoding/transpose_internal.h: In function 'bool riegeli::internal::HasDataBuffer(uint32_t, riegeli::internal::Subtype)':
./riegeli/chunk_encoding/transpose_internal.h:97:1: warning: control reaches end of non-void function [-Wreturn-type]
   97 | }
      | ^
riegeli/chunk_encoding/transpose_decoder.cc: In function 'riegeli::internal::CallbackType riegeli::internal::GetCallbackType(riegeli::{anonymous}::FieldIncluded, uint32_t, riegeli::internal::Subtype, size_t, bool)':
riegeli/chunk_encoding/transpose_decoder.cc:359:1: warning: control reaches end of non-void function [-Wreturn-type]
  359 | }
      | ^
INFO: Elapsed time: 21.153s, Critical Path: 12.16s
INFO: 65 processes: 14 internal, 51 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

Implementing a custom Writer / Reader

Hi,

I wanted to implement a streaming writer/reader that can directly interface with Amazon S3. Looking through the code, it seemed like I should just be able to extend BufferedWriter and implement some of its methods.

I created an implementation and overwrote the pure virtual Flush and WriteInternal methods, but it seems that other than the constructor, these methods never get called. When calling Done on the RecordWriter, I get a segfault:

*** Segmentation fault (@0xc0) received by PID 22734 (TID 0x7f3eeee419c0) from PID 192; stack trace: ***
    @     0x7f3efbf76e11 common_internal::signalHandler()
    @     0x7f3ef3c47390 (unknown)
    @     0x7f3ef53b12e7 riegeli::RecordWriterBase::Done()
    @     0x5595de55274f riegeli::RecordWriter<>::Done()
    @     0x5595de551899 run()
    @     0x5595de55077b main
    @     0x7f3ef29c9830 __libc_start_main
    @     0x5595de550909 _start

I tried to understand how the FdWriter does the implementation, but couldn't figure it out. Can you provide some pointers what I need to implement in my custom Writer to make it work?

Thanks!
Arno

Supporting CMake build

Hi dear maintainer,

First I would like to express my gratitude for your open sourcing this library. As an ex-googler I am getting pretty used to recordio and this is exactly what I need in my use cases.

In one of my recent project I need to add this library as a cmake dependency. Tried to search in the issues but seems no one is mentioning having CMake support. I personally think CMake is still popular among the open source world and it might be good if the library support it so that more projects can enjoy the benefit of this library.

Thanks!

Make Riegeli available as an external Bazel dependency

I'd prefer to not manually vendor (clone) Riegeli into my source tree, but rather use it as an external Bazel dependency (either by modifying our WORKSPACE or MODULE.bazel). Might you consider providing that functionality and document it?

(Apologies if this already exists and I missed it. I didn't find any documentation like, to pick a random example, https://github.com/hedronvision/bazel-compile-commands-extractor#first-do-the-usual-workspace-setup. Nor did I find any macros for pulling in Riegeli's deps as is typical in any non-bzlmod repo that's used as an external dependency.)

David

Any chance of a Java port?

This looks like a really neat as a way to serialize a large number of protobuffers. I see there's C++ and Python but at least my reader process is Spark based, so Java. Any chance there's a Java port on the way? Or at least JNI/Swig bindings and README on how to make it work from Java?

base/buffer.h: Type qualifiers ignored on function return type

Hi,

Compiling Riegeli in our project setup creates the following GCC warning/error:
riegeli/base/buffer.h:52:3: type qualifiers ignored on function return type [-Werror=ignored-qualifiers]

The reason is that the first const in the line has no effect, since it returns by value:
const size_t size() const { return size_; }

It would be great if you could just merge it - I've created #3, but Googlebot doesn't like me. :)

Thanks,
Arno

The size of float is no diff between double

I used double in protobuf , now I want to reduce the size of data, so I change it to float, but the size is no change
Is riegeli compression double data well than float?

import error

Simply import riegeli gives me this error:

Traceback (most recent call last):
File "test.py", line 22, in
import riegeli
File "build/bdist.linux-x86_64/egg/riegeli/init.py", line 21, in
ImportError: cannot import name record_position

This is very mundane -- how to install (and run bazel..).

For the life of me I can't find any installation instructions nor by thrashing around can I figure out the right bazel invocation.

I'm guessing the instructions would be something like:

install bazel
git clone https://github.com/google/riegeli.git
cd riegeli
./configure
bazel build ???

I was able to get bazel build //riegeli/base:status to work, but bazel build //riegeli/... is unhappy:
`ERROR: /Users/dave/Projects/riegeli/riegeli/tensorflow/io/BUILD:27:11: no such package 'third_party/tensorflow/core/platform':
BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.

/Users/dave/Projects/riegeli/third_party/tensorflow/core/platform and referenced by '//riegeli/tensorflow/io:file_writer'`

I'm on a Macbook Air, the M1, and its tensorflow is a bit funky, which may be another issue
as I had to fix the output of ./configure since when you import tensorflow you get some
debug output:

 python -c "import tensorflow"
Init Plugin
Init Graph Optimizer
Init Kernel

My goal is to get a c++ and python library built.

Basic info:
uname -a: Darwin daves-air.lan 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:27 PDT 2021; root:xnu-7195.141.2~5/RELEASE_ARM64_T81
01 arm64

Python 3.9.4

TF: 2.5.0

thx,
Dave

Provide examples

Could you give an example of writing records to a file and reading the data back? I'm reading the code but the API seems fairly complex..

warning: class 'Chunk' was previously declared as a struct

Clang gives the following warning:

external/riegeli/riegeli/records/record_reader.h:42:1: warning: class 'Chunk' was previously declared as a struct [-Wmismatched-tags]
class Chunk;
^
external/riegeli/riegeli/chunk_encoding/chunk.h:83:8: note: previous use is here
struct Chunk {
       ^
external/riegeli/riegeli/records/record_reader.h:42:1: note: did you mean struct here?
class Chunk;
^~~~~
struct
1 warning generated.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.