smartfile / python-libarchive Goto Github PK

View Code? Open in Web Editor NEW

This project forked from travcunn/python-libarchive

22.0 22.0 10.0 290 KB

Libarchive wrapper for Python.

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.17% Python 15.63% C 79.13% SWIG 4.88% Shell 0.19%

python-libarchive's People

Contributors

Stargazers

Watchers

Forkers

abarnert pombreda ltd-beget linearregression skyleach aaron-sierra kser93 richnetdesign jmgurney 6769

python-libarchive's Issues

pip --global-options ignored

For Mac users there is a catch-22 with the default libarchive shipped with the OS. (Tested on Sierra and El Capitain).

If one updates the core library, tons of things break in OSX.

If one doesn't update the core library, libarchive doesn't work.

missing symbol:

Traceback (most recent call last):
  File "/Users/skyleach/.virtualenvs/imgunbundler/bin/imgunbundle", line 11, in <module>
    load_entry_point('imgunbundler', 'console_scripts', 'imgunbundle')()
  File "/Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2627, in load_entry_point
    return ep.load()
  File "/Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2287, in load
    return self.resolve()
  File "/Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2293, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/skyleach/src/imgunbundle/imgunbundler/imgunbundle.py", line 57, in <module>
    from unbundlers import unbundle_cpio
  File "/Users/skyleach/src/imgunbundle/imgunbundler/unbundlers/unbundle_cpio.py", line 2, in <module>
    import libarchive.public
  File "/Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/libarchive/public.py", line 1, in <module>
    from libarchive.adapters.archive_read import \
  File "/Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/libarchive/adapters/archive_read.py", line7, in <module>
    import libarchive.calls.archive_read
  File "/Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/libarchive/calls/archive_read.py", line 17, in <module>
    c_archive_read_support_filter_all = libarchive.archive_read_support_filter_all
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 375, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 380, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fa97a61aca0, archive_read_support_filter_all): symbol not found

Generally the solution for mac users would be to install an updated libarchive using homebrew into /usr/local/opt/libarchive.

Folowing this we would instruct pip to pass the location to the module compiler using pip install --global-option="-L/usr/local/opt/libarchive/lib" --global-option="-I/usr/local/opt/libarchive/include" libarchive

However this doesn't work with libarchive:

        /Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages/pip/commands/install.py:194: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
          cmdoptions.check_install_build_global(options)
        Collecting libarchive
          Using cached libarchive-0.4.3.tar.gz
        Requirement already satisfied: nose in /Users/skyleach/.virtualenvs/imgunbundler/lib/python2.7/site-packages (from libarchive)
        Skipping bdist_wheel for libarchive, due to binaries being disabled for it.
        Installing collected packages: libarchive
          Running setup.py install for libarchive: started
            Running setup.py install for libarchive: finished with status 'error'
            Complete output from command /Users/skyleach/.virtualenvs/imgunbundler/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/kt/yxsj572j6z18h6gq073_zvdr0000gn/T/pip-build-1krdrd/libarchive/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" -L/usr/local/opt/libarchive/lib -I/usr/local/opt/libarchive/include install --record /var/folders/kt/yxsj572j6z18h6gq073_zvdr0000gn/T/pip-Lpnx1J-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/skyleach/.virtualenvs/imgunbundler/bin/../include/site/python2.7/libarchive:
            usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
               or: -c --help [cmd1 cmd2 ...]
               or: -c --help-commands
               or: -c cmd --help
            
            error: option -L not recognized
            
            ----------------------------------------
        Command "/Users/skyleach/.virtualenvs/imgunbundler/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/kt/yxsj572j6z18h6gq073_zvdr0000gn/T/pip-build-1krdrd/libarchive/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" -L/usr/local/opt/libarchive/lib -I/usr/local/opt/libarchive/include install --record /var/folders/kt/yxsj572j6z18h6gq073_zvdr0000gn/T/pip-Lpnx1J-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/skyleach/.virtualenvs/imgunbundler/bin/../include/site/python2.7/libarchive" failed with error code 1 in /private/var/folders/kt/yxsj572j6z18h6gq073_zvdr0000gn/T/pip-build-1krdrd/libarchive/

I grepped through the python-libarchive code for LDFLAGS and CPPFLAGS and didn't seem them being brought in. I also tried setting them before calling pip install as environment variables: export LDFLAGS=' -L/usr/local/opt/libarchive/lib';export CPPFLAGS='-I/usr/local/opt/libarchive/include';export PKG_CONFIG_PATH='/usr/local/opt/libarchive/lib/pkgconfig'; pip install libarchive but that didn't work either.

Seems to be no way to specify which version of libarchive to link against?

crash on improper use

I agree with EAFP,[*] but crashing breaks Python semantics, so this improper use should be guarded. I'm not clear if this bug is in libarchive.so or in python-libarchive. Unless libarchive.so (possibly implicitly) states arguments validity as a precondition, the bug is theirs. However, I only report it here, for now.

Python 2.7.9 (default, Jun 29 2016, 13:08:31) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libarchive.public
>>> with libarchive.public.memory_reader(open('v2.zip').read()) as e:
...  pass
... 
>>> next(e)

Program received signal SIGSEGV, Segmentation fault.
0x0000000000b0eca0 in ?? ()
(gdb) bt
#0  0x0000000000b0eca0 in ?? ()
#1  0x00007ffff6403dc0 in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#2  0x00007ffff6403828 in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#3  0x00007ffff665f055 in _call_function_pointer (argcount=2, resmem=0x7fffffffd110, restype=<optimized out>, atypes=<optimized out>, avalues=0x7fffffffd0f0, pProc=0x7ffff580b590 <archive_read_next_header>, flags=4353) at /home/jmm/scratch/spu/python2.7-2.7.9/Modules/_ctypes/callproc.c:836
#4  _ctypes_callproc (pProc=pProc@entry=0x7ffff580b590 <archive_read_next_header>, argtuple=argtuple@entry=0x7ffff63ba3b0, flags=4353, argtypes=argtypes@entry=0x7ffff6398518, restype=restype@entry=0xa67480, checker=checker@entry=0x0) at /home/jmm/scratch/spu/python2.7-2.7.9/Modules/_ctypes/callproc.c:1179
#5  0x00007ffff66639b2 in PyCFuncPtr_call.lto_priv.95 (self=<optimized out>, inargs=<optimized out>, kwds=<optimized out>) at /home/jmm/scratch/spu/python2.7-2.7.9/Modules/_ctypes/_ctypes.c:3965

[*] "It is Easier to Ask for Forgiveness than Permission"
https://en.wikipedia.org/wiki/Python_syntax_and_semantics#Exceptions

SeekableArchive raises an exception when querying the files a second time

After opening a SeekableArchive, the contained files can be queried by iterating over it:

echo foo > bar
tar -cjf single-file.tar.bz2 bar

import libarchive  # 4.1.0
a = libarchive.SeekableArchive('single-file.tar.bz2')
print(list(a))  # [<libarchive.Entry at 0x7fed38f9c850>]

But, when trying this a second time, it throws an exception, which isn't what I would expect and which complicates interactive usage by a lot:

print(list(a))
print(list(a))

Exception:

<ipython-input-8-ab50db00fb1d> in <module>
----> 1 print(list(a))

~/.local/lib/python3.10/site-packages/libarchive/__init__.py in __iter__(self)
    677         if not self.eof:
    678             try:
--> 679                 for entry in super(SeekableArchive, self).__iter__():
    680                     self.entries.append(entry)
    681                     yield entry

~/.local/lib/python3.10/site-packages/libarchive/__init__.py in __iter__(self)
    486         while True:
    487             try:
--> 488                 yield self.entry_class.from_archive(self, encoding=self.encoding)
    489             except EOF:
    490                 break

~/.local/lib/python3.10/site-packages/libarchive/__init__.py in from_archive(cls, archive, encoding)
    326         e = _libarchive.archive_entry_new()
    327         try:
--> 328             call_and_check(_libarchive.archive_read_next_header2, archive._a, archive._a, e)
    329             mode = _libarchive.archive_entry_filetype(e)
    330             mode |= _libarchive.archive_entry_perm(e)

~/.local/lib/python3.10/site-packages/libarchive/__init__.py in call_and_check(func, archive, *args)
    111         raise EOF()
    112     else:
--> 113         raise Exception('Problem executing function, message is: %s.' % get_error(archive))
    114 
    115

I think, it should be possible to iterate over the archive how many often the user desires.

Fix Travis CI version of libarchive-dev

It appears that Travis CI is using a version of libarchive-dev that is incompatible with our library. However, on Fedora 21, I am able to install the correct version of libarchive-devel with yum.

Failed to download on mac-x64 with python 3.11.5

(base) ➜  zip git:(main) ✗ pip install python-libarchive
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting python-libarchive
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/47/2f/4dc8be4fa6f835786c359b918d131ab748727446336e9030ad5b41603740/python-libarchive-4.2.1.tar.gz (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.5/58.5 kB 24.3 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: python-libarchive
  Building wheel for python-libarchive (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-x86_64-cpython-311
      creating build/lib.macosx-10.9-x86_64-cpython-311/libarchive
      copying libarchive/tar.py -> build/lib.macosx-10.9-x86_64-cpython-311/libarchive
      copying libarchive/__init__.py -> build/lib.macosx-10.9-x86_64-cpython-311/libarchive
      copying libarchive/_libarchive.py -> build/lib.macosx-10.9-x86_64-cpython-311/libarchive
      copying libarchive/zip.py -> build/lib.macosx-10.9-x86_64-cpython-311/libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      creating build/temp.macosx-10.9-x86_64-cpython-311
      creating build/temp.macosx-10.9-x86_64-cpython-311/libarchive
      clang -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/chris/soft/miniconda3/include -fPIC -O2 -isystem /Users/chris/soft/miniconda3/include -I/usr/local/include -Ilibarchive -I/Users/chris/soft/miniconda3/include/python3.11 -c libarchive/_libarchive_wrap.c -o build/temp.macosx-10.9-x86_64-cpython-311/libarchive/_libarchive_wrap.o
      libarchive/_libarchive_wrap.c:2712:10: fatal error: 'archive.h' file not found
      #include <archive.h>
               ^~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-libarchive
  Running setup.py clean for python-libarchive
Failed to build python-libarchive
ERROR: Could not build wheels for python-libarchive, which is required to install pyproject.toml-based projects

Package information out of date

The download URL still points to python-libarchive.googlecode.de
The mail now points to vadim1024. I tried the previous one and it was disfunctional, so, I guess it's a bit better but this was probably changed thinking a fork might be necessary.
A __version__ member might be helpful to me especially as the package name is different from the module name but I might not be mainstream with my dislike of importlib.metadata.version.

Steps to reproduce:

import importlib.metadata
print(dict(importlib.metadata.distribution("python-libarchive").metadata))

Building link is broken

Building link in the README file is broken: http://code.google.com/p/python-libarchive/wiki/Building

The matrix setting is missing

python-libarchive/.github/workflows/deploy.yml

Line 23 in ab93b59

PYVER: ${{ matrix.version }}

Opening the same file for the second time via SeekableArchive raises a RuntimeError

What I'm trying to do is open a file object to a file inside an archive and read from it. And when seeking back, I'm trying to reopen it from the start. My first guess was to simply class close on the file object and open it anew, but the second open throws.

The problem seems to be that the close on the file object for one of the many files inside the archive will actually close the whole archive. This is unexpected and the missing Archive.close method only adds to the confusion.
The close method actually only calls a deferred close according to the source code. I'm not sure how well behaved that is, especially when calling readstream multiple times and having multiple of those file objects open at the same time ...

Workflow:

import libarchive
a = libarchive.SeekableArchive('single-file.tar.bz2')
files = list(a)
f = a.readstream(files[0].pathname)
print(len(f.read()))  # 4
f.close()
f = a.readstream(files[0].pathname)
f.read()  # RuntimeError

RuntimeError                              Traceback (most recent call last)
<ipython-input-27-571e9fb02258> in <module>
----> 1 f.read()

~/.local/lib/python3.10/site-packages/libarchive/__init__.py in read(self, bytes)
    230             bytes = self.size - self.bytes
    231         # Read requested bytes
--> 232         data = _libarchive.archive_read_data_into_str(self.archive._a, bytes)
    233         self.bytes += len(data)
    234         return data

~/.local/lib/python3.10/site-packages/libarchive/_libarchive.py in archive_read_data_into_str(archive, len)
    583 
    584 def archive_read_data_into_str(archive, len):
--> 585     return __libarchive.archive_read_data_into_str(archive, len)
    586 
    587 def archive_write_data_from_str(archive, str):

RuntimeError: could not read requested data.

Looking at the code, the context manager for EntryReadStream should behave correctly, i.e., not call close but I can't use it because the lifetime is longer than a simple with-code-block and Python has no C++ like RAII unfortunately.

Building against libarchive on Windows

Windows 10, VS2022 build tools are installed, Python 3.10.10

> pip install python-libarchive
Collecting python-libarchive
  Using cached python-libarchive-4.2.1.tar.gz (58 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: python-libarchive
  Building wheel for python-libarchive (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win32-cpython-310
      creating build\lib.win32-cpython-310\libarchive
      copying libarchive\tar.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\zip.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\_libarchive.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\__init__.py -> build\lib.win32-cpython-310\libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-libarchive
  Running setup.py clean for python-libarchive
Failed to build python-libarchive
Installing collected packages: python-libarchive
  Running setup.py install for python-libarchive ... error
  error: subprocess-exited-with-error

  × Running setup.py install for python-libarchive did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running install
      C:\git\archive-test\venv\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build\lib.win32-cpython-310
      creating build\lib.win32-cpython-310\libarchive
      copying libarchive\tar.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\zip.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\_libarchive.py -> build\lib.win32-cpython-310\libarchive
      copying libarchive\__init__.py -> build\lib.win32-cpython-310\libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> python-libarchive

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

error: command 'swig' failed: No such file or directory

After trying to upgrade from 4.0.1.post1 to 4.1.0-1, I'm getting an error during the installation, which wasn't the case before. Also, I want to suggest adhering somewhat to SemVer.

I'm on Ubuntu 22.04 with Python 3.10.4 and pip 22.0.2

pip3 install --user --upgrade python-libarchive

pip3 install --user --upgrade python-libarchive
Requirement already satisfied: python-libarchive in /home/user/.local/lib/python3.10/site-packages (4.0.1.post1)
Collecting python-libarchive
  Downloading python-libarchive-4.1.0.post1.tar.gz (22 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: python-libarchive
  Building wheel for python-libarchive (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      /home/user/.local/lib/python3.10/site-packages/setuptools/dist.py:530: UserWarning: Normalizing '4.1.0-1' to '4.1.0.post1'
        warnings.warn(tmpl.format(**locals()))
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/__init__.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/_libarchive.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/zip.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/tar.py -> build/lib.linux-x86_64-cpython-310/libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      swigging libarchive/_libarchive.i to libarchive/_libarchive_wrap.c
      swig -python -o libarchive/_libarchive_wrap.c libarchive/_libarchive.i
      error: command 'swig' failed: No such file or directory
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-libarchive
  Running setup.py clean for python-libarchive
Failed to build python-libarchive
pInstalling collected packages: python-libarchive
  Attempting uninstall: python-libarchive
    Found existing installation: python-libarchive 4.0.1.post1
    Uninstalling python-libarchive-4.0.1.post1:
      Successfully uninstalled python-libarchive-4.0.1.post1
  Running setup.py install for python-libarchive ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for python-libarchive did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      /home/user/.local/lib/python3.10/site-packages/setuptools/dist.py:530: UserWarning: Normalizing '4.1.0-1' to '4.1.0.post1'
        warnings.warn(tmpl.format(**locals()))
      running install
      /home/user/.local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/__init__.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/_libarchive.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/zip.py -> build/lib.linux-x86_64-cpython-310/libarchive
      copying libarchive/tar.py -> build/lib.linux-x86_64-cpython-310/libarchive
      running build_ext
      building 'libarchive.__libarchive' extension
      swigging libarchive/_libarchive.i to libarchive/_libarchive_wrap.c
      swig -python -o libarchive/_libarchive_wrap.c libarchive/_libarchive.i
      error: command 'swig' failed: No such file or directory
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  WARNING: No metadata found in /home/user/.local/lib/python3.10/site-packages
  Rolling back uninstall of python-libarchive
  Moving to /home/user/.local/lib/python3.10/site-packages/libarchive/
   from /home/user/.local/lib/python3.10/site-packages/~ibarchive
  Moving to /home/user/.local/lib/python3.10/site-packages/python_libarchive-4.0.1.post1.dist-info/
   from /home/user/.local/lib/python3.10/site-packages/~ython_libarchive-4.0.1.post1.dist-info
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> python-libarchive

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Install fails on MacOS due to incorrect rpath flag with clang's ld

libarchive installed via Brew, and install path set via LIBARCHIVE_PREFIX environment variable. Installing via pip install python-libarchive results in this error:

  clang -bundle -undefined dynamic_lookup -arch arm64e -arch x86_64 -Wl,-headerpad,0x1000 -L/usr/local/opt/libarchive/lib build/temp.macosx-11.0-x86_64-cpython-38/libarchive/_libarchive_wrap.o -larchive -o build/lib.macosx-11.0-x86_64-cpython-38/libarchive/__libarchive.cpython-38-darwin.so -Wl,-rpath=/usr/local/opt/libarchive/lib
  ld: unknown option: -rpath=/usr/local/opt/libarchive/lib
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command '/usr/bin/clang' failed with exit code 1
  error: subprocess-exited-with-error

This is due to -rpath=/path being GNU ld syntax. Updating the setup.py to use -rpath,/path on macOS fixes this issue and successfully installs.

Cannot build on OSX 10.11 with XCode 7 toolchain

Collecting python-libarchive
  Downloading python-libarchive-3.1.2-1.tar.gz (73kB)
    100% |████████████████████████████████| 81kB 612kB/s 
Building wheels for collected packages: python-libarchive
  Running setup.py bdist_wheel for python-libarchive ... error
  Complete output from command /Users/bsergeant/sandbox/venv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/qz/cb1zd5756hnd2tykv7z5sn_j8408d8/T/pip-build-AP10r8/python-libarchive/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/qz/cb1zd5756hnd2tykv7z5sn_j8408d8/T/tmpuip1HQpip-wheel- --python-tag cp27:
  /Users/bsergeant/sandbox/venv/lib/python2.7/site-packages/setuptools/dist.py:331: UserWarning: Normalizing '3.1.2-1' to '3.1.2.post1'
    normalized_version,
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.11-intel-2.7
  creating build/lib.macosx-10.11-intel-2.7/libarchive
  copying libarchive/__init__.py -> build/lib.macosx-10.11-intel-2.7/libarchive
  copying libarchive/_libarchive.py -> build/lib.macosx-10.11-intel-2.7/libarchive
  copying libarchive/tar.py -> build/lib.macosx-10.11-intel-2.7/libarchive
  copying libarchive/zip.py -> build/lib.macosx-10.11-intel-2.7/libarchive
  running build_ext
  building 'libarchive.__libarchive' extension
  creating build/temp.macosx-10.11-intel-2.7
  creating build/temp.macosx-10.11-intel-2.7/libarchive
  cc -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -Ilibarchive -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c libarchive/_libarchive_wrap.c -o build/temp.macosx-10.11-intel-2.7/libarchive/_libarchive_wrap.o
  libarchive/_libarchive_wrap.c:2351:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                    res = SWIG_AddCast(res);
                    ~~~ ^              ~~~
  libarchive/_libarchive_wrap.c:2354:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                    res = SWIG_AddCast(res);
                    ~~~ ^              ~~~
  libarchive/_libarchive_wrap.c:2876:9: warning: variable 'res' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
      if (PyType_Ready(tp) < 0)
          ^~~~~~~~~~~~~~~~~~~~
  libarchive/_libarchive_wrap.c:2900:10: note: uninitialized use occurs here
    return res;
           ^~~
  libarchive/_libarchive_wrap.c:2876:5: note: remove the 'if' if its condition is always false
      if (PyType_Ready(tp) < 0)
      ^~~~~~~~~~~~~~~~~~~~~~~~~
  libarchive/_libarchive_wrap.c:2857:10: note: initialize the variable 'res' to silence this warning
    int res;
           ^
            = 0
  libarchive/_libarchive_wrap.c:5747:14: warning: explicitly assigning value of variable of type 'void *' to itself [-Wself-assign]
    clientdata = clientdata;
    ~~~~~~~~~~ ^ ~~~~~~~~~~
  4 warnings generated.
  libarchive/_libarchive_wrap.c:2351:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                    res = SWIG_AddCast(res);
                    ~~~ ^              ~~~
  libarchive/_libarchive_wrap.c:2354:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                    res = SWIG_AddCast(res);
                    ~~~ ^              ~~~
  libarchive/_libarchive_wrap.c:2876:9: warning: variable 'res' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
      if (PyType_Ready(tp) < 0)
          ^~~~~~~~~~~~~~~~~~~~
  libarchive/_libarchive_wrap.c:2900:10: note: uninitialized use occurs here
    return res;
           ^~~
  libarchive/_libarchive_wrap.c:2876:5: note: remove the 'if' if its condition is always false
      if (PyType_Ready(tp) < 0)
      ^~~~~~~~~~~~~~~~~~~~~~~~~
  libarchive/_libarchive_wrap.c:2857:10: note: initialize the variable 'res' to silence this warning
    int res;
           ^
            = 0
  libarchive/_libarchive_wrap.c:3357:15: warning: implicit conversion loses integer precision: 'Py_ssize_t' (aka 'long') to 'int' [-Wshorten-64-to-32]
      int len = PyString_Size(str);
          ~~~   ^~~~~~~~~~~~~~~~~~
  libarchive/_libarchive_wrap.c:5747:14: warning: explicitly assigning value of variable of type 'void *' to itself [-Wself-assign]
    clientdata = clientdata;
    ~~~~~~~~~~ ^ ~~~~~~~~~~
  5 warnings generated.
  cc -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -Wl,-F. build/temp.macosx-10.11-intel-2.7/libarchive/_libarchive_wrap.o -larchive -o build/lib.macosx-10.11-intel-2.7/libarchive/__libarchive.so -l:libarchive.so.13.1.2
  ld: library not found for -l:libarchive.so.13.1.2
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command 'cc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for python-libarchive
  Running setup.py clean for python-libarchive
Failed to build python-libarchive
Installing collected packages: python-libarchive
  Running setup.py install for python-libarchive ... error
    Complete output from command /Users/bsergeant/sandbox/venv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/qz/cb1zd5756hnd2tykv7z5sn_j8408d8/T/pip-build-AP10r8/python-libarchive/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/qz/cb1zd5756hnd2tykv7z5sn_j8408d8/T/pip-xJ6Gy7-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/bsergeant/sandbox/venv/include/site/python2.7/python-libarchive:
    /Users/bsergeant/sandbox/venv/lib/python2.7/site-packages/setuptools/dist.py:331: UserWarning: Normalizing '3.1.2-1' to '3.1.2.post1'
      normalized_version,
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.11-intel-2.7
    creating build/lib.macosx-10.11-intel-2.7/libarchive
    copying libarchive/__init__.py -> build/lib.macosx-10.11-intel-2.7/libarchive
    copying libarchive/_libarchive.py -> build/lib.macosx-10.11-intel-2.7/libarchive
    copying libarchive/tar.py -> build/lib.macosx-10.11-intel-2.7/libarchive
    copying libarchive/zip.py -> build/lib.macosx-10.11-intel-2.7/libarchive
    running build_ext
    building 'libarchive.__libarchive' extension
    creating build/temp.macosx-10.11-intel-2.7
    creating build/temp.macosx-10.11-intel-2.7/libarchive
    cc -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -Ilibarchive -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c libarchive/_libarchive_wrap.c -o build/temp.macosx-10.11-intel-2.7/libarchive/_libarchive_wrap.o
    libarchive/_libarchive_wrap.c:2351:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                      res = SWIG_AddCast(res);
                      ~~~ ^              ~~~
    libarchive/_libarchive_wrap.c:2354:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                      res = SWIG_AddCast(res);
                      ~~~ ^              ~~~
    libarchive/_libarchive_wrap.c:2876:9: warning: variable 'res' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
        if (PyType_Ready(tp) < 0)
            ^~~~~~~~~~~~~~~~~~~~
    libarchive/_libarchive_wrap.c:2900:10: note: uninitialized use occurs here
      return res;
             ^~~
    libarchive/_libarchive_wrap.c:2876:5: note: remove the 'if' if its condition is always false
        if (PyType_Ready(tp) < 0)
        ^~~~~~~~~~~~~~~~~~~~~~~~~
    libarchive/_libarchive_wrap.c:2857:10: note: initialize the variable 'res' to silence this warning
      int res;
             ^
              = 0
    libarchive/_libarchive_wrap.c:5747:14: warning: explicitly assigning value of variable of type 'void *' to itself [-Wself-assign]
      clientdata = clientdata;
      ~~~~~~~~~~ ^ ~~~~~~~~~~
    4 warnings generated.
    libarchive/_libarchive_wrap.c:2351:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                      res = SWIG_AddCast(res);
                      ~~~ ^              ~~~
    libarchive/_libarchive_wrap.c:2354:23: warning: explicitly assigning value of variable of type 'int' to itself [-Wself-assign]
                      res = SWIG_AddCast(res);
                      ~~~ ^              ~~~
    libarchive/_libarchive_wrap.c:2876:9: warning: variable 'res' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
        if (PyType_Ready(tp) < 0)
            ^~~~~~~~~~~~~~~~~~~~
    libarchive/_libarchive_wrap.c:2900:10: note: uninitialized use occurs here
      return res;
             ^~~
    libarchive/_libarchive_wrap.c:2876:5: note: remove the 'if' if its condition is always false
        if (PyType_Ready(tp) < 0)
        ^~~~~~~~~~~~~~~~~~~~~~~~~
    libarchive/_libarchive_wrap.c:2857:10: note: initialize the variable 'res' to silence this warning
      int res;
             ^
              = 0
    libarchive/_libarchive_wrap.c:3357:15: warning: implicit conversion loses integer precision: 'Py_ssize_t' (aka 'long') to 'int' [-Wshorten-64-to-32]
        int len = PyString_Size(str);
            ~~~   ^~~~~~~~~~~~~~~~~~
    libarchive/_libarchive_wrap.c:5747:14: warning: explicitly assigning value of variable of type 'void *' to itself [-Wself-assign]
      clientdata = clientdata;
      ~~~~~~~~~~ ^ ~~~~~~~~~~
    5 warnings generated.
    cc -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -Wl,-F. build/temp.macosx-10.11-intel-2.7/libarchive/_libarchive_wrap.o -larchive -o build/lib.macosx-10.11-intel-2.7/libarchive/__libarchive.so -l:libarchive.so.13.1.2
    ld: library not found for -l:libarchive.so.13.1.2
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command 'cc' failed with exit status 1

    ----------------------------------------
Command "/Users/bsergeant/sandbox/venv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/qz/cb1zd5756hnd2tykv7z5sn_j8408d8/T/pip-build-AP10r8/python-libarchive/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/qz/cb1zd5756hnd2tykv7z5sn_j8408d8/T/pip-xJ6Gy7-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/bsergeant/sandbox/venv/include/site/python2.7/python-libarchive" failed with error code 1 in /private/var/folders/qz/cb1zd5756hnd2tykv7z5sn_j8408d8/T/pip-build-AP10r8/python-libarchive/

Returning b'' instead of None when reaching EOF

As discussed with the maintainer of ijson over here it seems that the archive stream reader should be returning b'' instead of None when finished reading.

your file-object from libarchive is "weird" in the sense that when it gets to the endof the file it implicitly returns None instead of an empty string (here and here). File-like objects are expected to return an empty string ('' or b'', depending on the mode with which it operates) when the end of the file is reached. This is the behavior that ijson expects as well

Would that be something that can be fixed for future versions?

File permission on zip member reported incorrectly

https://code.google.com/p/python-libarchive/issues/detail?id=6

Reported by [email protected], Jan 10, 2015

What steps will reproduce the problem?

  $ cd /tmp
  $ cp /etc/resolv.conf test1
  $ cp /etc/resolv.conf test2
  $ chmod 444 test1
  $ chmod 666 test2
  $ zip test.zip test1 test2
  $ /tmp/t1.py

What is the expected output? What do you see instead?
Expected output is:

  test2
  33206
  test1
  33060

Output displayed is:

  test2
  33206
  test1
  33206

What version of the product are you using? On what operating system?
libarchive-3.1.2, python-libarchive-3.1.2-1
RHEL6

Please provide any additional information below.
The problem is most likely with libarchive/init.py. I wrote another python program, /tmp/t2.py, using the raw libarchive commands and it gave the expected output.

/tmp/t1.py

#!/opt/fsw/pkgutils16/lib/aux/python/bin/python

import os
import sys

import libarchive.zip

zipfile = '/tmp/test.zip'

z = libarchive.zip.ZipFile (zipfile, 'r')

for entry in z:
  print entry.pathname
  print entry.mode

z.close ()

/tmp/t2.py

#!/opt/fsw/pkgutils16/lib/aux/python/bin/python

import os
import sys

from libarchive import _libarchive

zipfile = '/tmp/test.zip'

a = _libarchive.archive_read_new()
_libarchive.archive_read_support_format_zip(a)

_libarchive.archive_read_open_filename(a, zipfile, 10240)

e = _libarchive.archive_entry_new()
while True:
  r = _libarchive.archive_read_next_header2(a, e)
  if r == _libarchive.ARCHIVE_EOF:
    break
  path = _libarchive.archive_entry_pathname(e).decode('utf-8')
  mode = _libarchive.archive_entry_filetype(e)
  perm = _libarchive.archive_entry_perm(e)

  print path, mode, perm, mode | perm

_libarchive.archive_entry_free(e)
_libarchive.archive_read_close(a)
_libarchive.archive_read_free(a)

make license clear, add LICENSE.txt file

It's not clear what the license for this project is.

It looks like it's a BSD 3-clause, but it'd be nice if there was a single file that could be referenced instead of having to check all the files for the project.

Missing support for password protected archives

It is pretty trivial to add, whould you accept a patch?

Allow directories to be written to zip files

Publish wheels to PyPI

Currently, installing this package with pip might fail with this error (excerpt from CI):

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include -Ilibarchive -I/opt/hostedtoolcache/Python/3.6.15/x64/include/python3.6m -c libarchive/_libarchive_wrap.c -o build/temp.linux-x86_64-3.6/libarchive/_libarchive_wrap.o
libarchive/_libarchive_wrap.c:2712:10: fatal error: archive.h: No such file or directory
 2712 | #include <archive.h>
      |          ^~~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1

Installing libarchive-dev on Ubuntu fixes this but it is an additional step the user has to be made aware of.

To ease installation issues like this one, providing wheels would be very much appreciated. Even better if it bundles libarchive, which thanks to its permissive license should be possible to do. It seems like @Vadiml1024 already tried to set up a CI for this. Personally, I'm using a pyproject.toml and the build package to build the tarball and cibuildwheel to build the wheels for my C++ indexed_bzip2/pragzip.

"pip install python-libarchive" ?

You readme states pip install python-libarchive.

This leads to some ancient package: https://pypi.org/project/python-libarchive/ (Jun 6, 2013).

Do you not publish your version on PyPI?