guzba / zippy Goto Github PK

Tried updating my nitter instance, which updated zippy since then. Getting instruction aborts in inflateBlock. Rolling back to 0.9.11 works fine.

localhost:~/zippy# nimble --version
nimble v compiled at 2022-08-05 03:01:31
git hash: 43804b9c6ecec455a6851242a45a19b31b9b0776

localhost:~/zippy# nim --version
Nim Compiler Version 1.6.8 [Linux: amd64]
Compiled at 2022-11-15
Copyright (c) 2006-2021 by Andreas Rumpf

active boot switches: -d:release

localhost:~/zippy# gcc --version
gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924

Steps to reproduce:

localhost:~/zippy# nimble test -d:Release
  Verifying dependencies for [email protected]
  Compiling /root/zippy/tests/test (from package zippy) using c backend
randtest1.gz compressed: 4119 gold: 4096
randtest2.gz compressed: 8215 gold: 8192
randtest3.gz compressed: 12311 gold: 12288
rfctest1.gz compressed: 9370 gold: 34127
SIGILL: Illegal operation.
Error: execution of an external program failed: '/root/zippy/tests/test '
       Tip: 1 messages have been suppressed, use --verbose to show them.
     Error: Execution failed with exit code 1
        ... Command: /usr/bin/nim c --noNimblePath -d:NimblePkgVersion=0.10.7 --hints:off -d:Release -r --path:. /root/zippy/tests/test

Works fine in debug, which suggest that there is some UB at play.
GDB points to this bit

    literalsHuffman = initHuffman(fixedLitLenCodeLengths)
    distancesHuffman = initHuffman(fixedDistanceCodeLengths)
  else:
    let
>      hlit = b.readBits(5).int + 257
      hdist = b.readBits(5).int + 1
      hclen = b.readBits(4).int + 4

cpuinfo.txt
test.zip

I can understand not supporting extraction of paths that are relative, but I think this is an edge case that isn't relative and also isn't supported:

Extracting paths containing `...` is not supported (nim-1.0.4/c_code/7_1/@m..@slib@spackages@[email protected])

This is from from nim-1.0.4-linux_x64.tar.xz.

I have similar checks in my library which it seems I was also not sure about (https://github.com/dom96/untar/blob/master/src/untar.nim#L149), but I think checking for just ".." is too broad.

It seems that mummy is using this library to automatically perform on-the-fly gzip compression for HTTP responses which is great for performance, but in certain scenarios can be a serious security vulnerability due to BREACH

The BREACH link mentions HTB (Heal-the-BREACH) as an inexpensive, but effective mitigation. It would be nice if this is implemented, so that gzip compression can be used safely without people having to worry about the specifics of the application, threat model, etc.

thanks

/home/hortinstein/.cache/nim/cross_compile_d/@m..@[email protected]@[email protected]@szippy@sadler32_simd.nim.c:9:10:
fatal error: arm_neon.h: No such file or directory
    9 | #include "arm_neon.h"

I am getting the following issue when I try to compile, here is a reproduction:
https://github.com/hortinstein/cross_compile
nimble buildmobile

By reading the source, I've discovered I can disable simd with -d:zippyNoSimd so this isn't a blocker for me anymore, but in case it's fixable, I've filed this anyway. Feel free to close if it's not worth the time.

# samp.nim
import zippy
let gz = compress("something")
doAssert uncompress(gz) == "something"

Compiling samp.nim with this command:

nim c --cpu:arm64 --threads:on --parallelBuild:1 --tlsEmulation:off "--passC:-arch arm64" "--passC:-isysroot" "--passC:/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.2.sdk"   /tmp/samp.nim

Results in the error Cannot select: intrinsic %llvm.aarch64.crc32b:

Hint: clang -c  -w -ferror-limit=3 -pthread -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.2.sdk   -I/Users/matt/.choosenim/toolchains/nim-1.6.6/lib -I/private/tmp -o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c.o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c [Exec]
fatal error: error in backend: Cannot select: intrinsic %llvm.aarch64.crc32b
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: aarch64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
clang: note: diagnostic msg: PLEASE submit a bug report to http://developer.apple.com/bugreporter/ and include the crash backtrace, preprocessed source, and associated run script.
clang: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /var/folders/mt/84ydn6s16y793cksy0ykq7zw0000gn/T/@m-2a6dff.c
clang: note: diagnostic msg: /var/folders/mt/84ydn6s16y793cksy0ykq7zw0000gn/T/@m-2a6dff.sh
clang: note: diagnostic msg: Crash backtrace is located in
clang: note: diagnostic msg: /Users/matt/Library/Logs/DiagnosticReports/clang_<YYYY-MM-DD-HHMMSS>_<hostname>.crash
clang: note: diagnostic msg: (choose the .crash file that corresponds to your crash)
clang: note: diagnostic msg: 

********************
Error: execution of an external program failed: 'clang -c  -w -ferror-limit=3 -pthread -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.2.sdk   -I/Users/matt/.choosenim/toolchains/nim-1.6.6/lib -I/private/tmp -o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c.o /Users/matt/.cache/nim/samp_d/@m..@s..@sUsers@[email protected]@[email protected]@szippy@scrc32_simd.nim.c'

when we use nimble install zippy, the message says Downloading https://github.com/guzba/zippy using git. the directory tests takes up 35.1M, so it takes a long time to clone; ok to be honest, I have a bad internet supplier, for most of the time I can't finish git clone. I have to download the zipped source, which will be success sometimes, then install by hand. As a result, it is pain for me to install nim's library.

So, I propose to seperate test and examples from zippy lib, as a result we can get a rapid installation of the lib.

thanks

Currently tarballs cannot have more than 100 characters. Resulting in an error.

I've found that usiing Zippy to uncompress archive & tarball doesn't preserve symlink (which breaks some libraries package as archive)

zedeus/nitter@d407051

As a pure Nim implementation, are there any plans to make this available for the JS backend?

I tried to compress the folder in which there were files with the vin code, everything worked well. Then I was wondering how to go inside folders inside archives and I created the following hierarchy:

code:

createTarball("../utils/testFolder", "examples.tar.gz")
var tarbol = Tarball()
tarbol.open("examples.tar.gz")
echo tarbol.contents

And I got this:

The wall of invalid data, but there are some correct things, For example I see key: "testFolder/sas"
examples.tar.gz

OS: Arch, nim: 1.4.8

I would like to be able to add symlinks to my tarball

control using : unzip
UnZip 6.00 of 20 April 2009, by Info-ZIP. Maintained by C. Spieler. Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

zippy: 0.5.7

experienced with this zip file: https://impbox.net/nico/android.zip (3MB)

It appears the extra data may be coming from the file names, running zippy over the same file multiple times gives different results in the extra data.

using extractAll("android.zip", "zippy/android")

diff -ur unzip/android/ zippy/android/

diff -ur unzip/android/ zippy/android/
diff -ur unzip/android/app/jni/SDL/src/hidapi/android/jni/Application.mk zippy/android/app/jni/SDL/src/hidapi/android/jni/Application.mk
--- unzip/android/app/jni/SDL/src/hidapi/android/jni/Application.mk     2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/android/jni/Application.mk     2021-05-08 09:06:37.330885000 +1000
@@ -1,2 +1,3 @@
 APP_STL := gnustl_static
 APP_ABI := armeabi-v7a
+▒riptenaudio.h
\ No newline at end of file
diff -ur unzip/android/app/jni/SDL/src/hidapi/bootstrap zippy/android/app/jni/SDL/src/hidapi/bootstrap
--- unzip/android/app/jni/SDL/src/hidapi/bootstrap      2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/bootstrap      2021-05-08 09:06:37.333887400 +1000
@@ -1,2 +1,4 @@
 #!/bin/sh -x
 autoreconf --install --verbose --force
+e, 1);
+    }
\ No newline at end of file
diff -ur unzip/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp zippy/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp
--- unzip/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp      2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/libusb/hidusb.cpp      2021-05-08 09:06:37.357888200 +1000
@@ -1,3 +1,4 @@

 #define NAMESPACE HIDUSB
 #include "hid.c"
+c
\ No newline at end of file
diff -ur unzip/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp zippy/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp
--- unzip/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp       2020-07-17 15:10:36.000000000 +1000
+++ zippy/android/app/jni/SDL/src/hidapi/linux/hidraw.cpp       2021-05-08 09:06:37.371886400 +1000
@@ -1,3 +1,4 @@

 #define NAMESPACE HIDRAW
 #include "hid.c"
+e.h
\ No newline at end of file
Only in unzip/android/app/src/main: AndroidManifest.xml
Binary files unzip/android/app/src/main/res/mipmap-hdpi/ic_launcher.png and zippy/android/app/src/main/res/mipmap-hdpi/ic_launcher.png differ
Binary files unzip/android/app/src/main/res/mipmap-xhdpi/ic_launcher.png and zippy/android/app/src/main/res/mipmap-xhdpi/ic_launcher.png differ
Binary files unzip/android/app/src/main/res/mipmap-xxhdpi/ic_launcher.png and zippy/android/app/src/main/res/mipmap-xxhdpi/ic_launcher.png differ
Only in unzip/android/app/src/main/res/values: strings.xml
diff -ur unzip/android/settings.gradle zippy/android/settings.gradle
--- unzip/android/settings.gradle       2020-07-17 15:10:40.000000000 +1000
+++ zippy/android/settings.gradle       2021-05-08 09:06:38.497886400 +1000
@@ -1 +1,2 @@
 include ':app'
+
\ No newline at end of file

thanks for this excellent library.

I'm a Windows user. When I clone your repo without any special treatment an then run the test using
nim c -r tests/test, it will fail when comparing decompression result of rfctest2.gold.

IIRC this can be solved using .gitattributes file to prevent specific files undergo automatic EOL conversion.
In also see in your CI matrix, there is no windows build.
That's why you are not aware of this subtle difference related to git.

I want to extract a Tarball located at tarPath to tmpDir (not existing sub-directory). However, nothing gets extracted.

When I extract this Tarball with PeaZip, everything gets extracted perfectly and works fine. However, when I do it with this procedure

proc extractAll(tarPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
 ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}

the files never get extracted.

There is no exception, no message, nothing. No idea what is going on there.

The Tarball is named properly, suffixed with a .tar and works perfectly fine.

Update 1

After debugging tarball.nim I found out that inside the extractAll procedure, the loop after

Update 2

As expected, the provided Tarball() does not have contents, so nothing is extracted, as nothing is provided by open():

ref 0000000004913fb0 --> [contents = [data = [],
counter = 0,
first = -1,
last = -1]]

Getting the above when running echo tarball.repr here:

Update 3

Through further debugging open() my suspicion was confirmed: open() never fills the tarball.contents; they always remain empty.

Update 4

The above line ensures that tarball.contents are only filled, when the typeFlag either equals 0 or 5.
The problem is: typeFlag is always empty!

Update 5

The above line should ensure, that a valid fileNamePrefix is given, if the header is specifying a ustar archive.
The issue is, that fileNamePrefix always remains empty. because neither header[345 ..< 500].trim() nor header[345 ..< 500] returns any output. Not sure if this is required or optional. It however does not seem to be directly related to the bigger problem at hand, but it's worth noting.

Update 6

I found the mistake!

When using writeTarball(), the headers for each file are built manually:

Now, when we look at the tar standard format, we see that the header byte 156 needs to contain char typeflag;. This typeflag is meant to be set here:

Now in this case, the number (from ord()) does not get converted to a char; it returns nothing. That's why the typeFlag is missing

It would be nice to have a procedure that just returns the binary of a zipfile instead of writing to disk.

On Windows, unzipping to a partition other than C:/ fails.

When unzipping, you creating a temporary folder & are writing archive contents to that temporary folder, then moving contents to the final folder. However, Nim currently has a bug where moveDir operations across partitions fail.

I would suggest either:

• waiting until the issue is fixed
• replacing moveDir(tempDir, dest) with a copyDir and removeDir operation
• write directly to the destination folder instead of using a temporary dir (is there a reason for that?)

Add support for Comment string in ZIP files (like zip -z "comment" in Linux zip command).

Would be good

In the in-progress reworking for zip archives, one can no longer read from an arbitrary stream.

Also, while the move to memory mapped files is a great option, there can be cases where a process isn't going to have any write-access to the file system (which would include the file used for the memory map).

It would be preferable to be flexible here, but have good defaults. For instance, I'd love to have you reading from a stream I hand you, and only deflate one file at a time as I ask for them... and I would even like the option to hand you an output stream. That'd give me options to save on memory consumption as well.

Hello!

Is it in the plans to add support for tar.xz files?

Thanks a lot and keep up the good work :)

Heya,

First off, thanks a ton for this library.

I'm not exactly sure what I'm doing wrong here, I have the following code:

import zippy
import zippy/ziparchives
import strformat
import os

const pyzip = slurp"../rsrc/python-3.9.1-embed-amd64.zip"
const tmpdir: string = getEnv("LOCALAPPDATA") / "Temp"

proc extractPython(): bool =
    var z = uncompress(pyzip)
    extractAll(z, tmpdir)
    return true

proc main(): void =
    echo fmt"[*] tmpdir: {tmpdir}"
    if extractPython():
        echo fmt"[*] Python extracted"

when isMainModule:
    main()

I'm cross compiling to Windows from MacOS using Mingw. The above code produces the following error when run:

Does zippy not support decompressing zip files this way?

Thanks

Hi, I was trying to do something like code below
Thanks for putting effort in this project.

proc addFile*(
  archive: ZipArchive, content: string, path: string
) {.raises: [IOError, OSError, ZippyError].} =
  ## Adds a single file to the archive.

  let (head, tail) = splitPath(path)
  if head.len > 0 and head notin archive.contents:
    archive.contents[(head & os.DirSep).toUnixPath()] =
      ArchiveEntry(kind: ekDirectory)
  archive.contents[path.toUnixPath()] = ArchiveEntry(
    kind: ekFile,
    contents: content,
    lastModified: getTime(),
    permissions: {},
  )

Code:

import pkg/zippy/tarballs

extractAll(tarPath = "./libressl-3.4.2.tar.gz", dest = "./libressl")

Log:

/home/snwmds/test_zippy.nim(3) test_zippy
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(63) extractAll
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(14) parseTarOctInt
[[reraised from:
/home/snwmds/test_zippy.nim(3) test_zippy
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(129) extractAll
]]
Error: unhandled exception: /home/snwmds/test_zippy.nim(3) test_zippy
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(63) extractAll
/home/snwmds/.nimble/pkgs/zippy-0.9.5/zippy/tarballs.nim(12) parseTarOctInt
/home/snwmds/.choosenim/toolchains/nim-1.6.2/lib/pure/strutils.nim(1143) parseOctInt
invalid oct integer: 100664  [ZippyError]

Tarball: https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-3.4.2.tar.gz

https://github.com/bung87/finder/tree/zippy

I previously use std zip lib which works fine and test passed, when switch to zippy I got Unhandled exception: Unsupported zip archive [ZippyError]

the zip file compression through mac context menu, am on
platform: macOS
nim version: 1.4.4

Thanks for the library.
From what I can see, it supports only extracting all or adding all to a zip archive.
I have potentially huge archives and want to be able to add a file at a time (and write immediately flush to disk) when writing

and also to extract a single file from the archive, into memory (without writing the uncompressed to disk) when reading.

Maybe it could be something like:

let archive = ZipArchive("/path/to/my.zip", fmRead)
var buffer = newString(777)
# or
# var buffer = newSeq[int32](2048)
archive.extractInto("internal/zip/path.bin", buffer) # read into buffer[0].pointer

and for writing it could just support adding a single file or Stream at a time along with the compression level (if any).

Would this be something you'd consider supporting?

It would be cool to be able to include/ignore files in createZipArchive/addDir with an optional glob-pattern:

Like this would include all top-level files in myfiles/, but exclude .DS_Store files:

createZipArchive("myfiles/*", "file.zip", ".DS_Store")

Or include all files in current directory recursively, except zip files:

createZipArchive("**/*", "file.zip", "*.zip")

If there is interest, I can PR for it.

Before 8c43184, tests/test_ziparchives.nim was writting files in system temporary directories. This seems to no longer be the case.

This is breaking nixpkgs package tests: NixOS/nixpkgs#176342

error: builder for '/nix/store/y81y71annic3isgqg417w2sbd5ckg430-zippy-0.9.9.drv' failed with exit code 1;
       last 10 log lines:
       >   inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vccenv.nim
       >   inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vccexe.nim
       >   inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vccvswhere.nim
       >   inflating: tmp/zip/gold/nim-1.6.2/tools/vccexe/vcvarsall.nim
       >   inflating: tmp/zip/gold/nim-1.6.2/tools/website.nimf
       >   inflating: tmp/zip/gold/nim-1.6.2/tools/winrelease.nim
       > fatal.nim:53             sysFatal
       > Error: unhandled exception: test_ziparchives.nim:28:16 `fileExists(zippyPath)`  [AssertionDefect]
       > Error: execution of an external program failed: '/build/nimcache/test_ziparchives_49AB4550FB1DC32554F842F393B0A633A4FA67C8 '
       > check phase failed

Is there a reason that this test is no longer using getTempDir?

Best regards,

It would be a nice feature to be able to choose a destination folder that already exists. For example:

someDir/
	- myZip.zip

And extract it like this:

someDir/
	- myZip.zip
	- the content of myZip.zip

The reason for this is the inconvenience of having to nest the destination in a new folder like this (plus the zip file may itself contain a folder as the top element):

someDir/
	- myZip.zip
	- myUnzippedFolder/
		- the content of myZip.zip

If you don't want to deal with overwriting files that already exist, you could perhaps demand from the user that they clean all old copies away before extracting. And otherwise you raise an exception if you stumble upon a clash.

What do you think? 😄

compilation command:

nimble js -o:public/main.js src/main.nim

nimble: v0.11.0

nim: v1.0.6

output:

/home/codespace/.nimble/pkgs/zippy-0.5.5/zippy/common.nim(137, 13) Error: type mismatch: got <int literal(16), uint8>
but expected one of: 
proc `-`(x: int8): int8
  first type mismatch at position: 2
  extra argument given
proc `-`(x: int16): int16
  first type mismatch at position: 2
  extra argument given
proc `-`(x, y: float32): float32
  first type mismatch at position: 2
  required type for y: float32
  but expression 'length' is of type: uint8
proc `-`(x: float32): float32
  first type mismatch at position: 2
  extra argument given
proc `-`(x, y: int32): int32
  first type mismatch at position: 2
  required type for y: int32
  but expression 'length' is of type: uint8
proc `-`(x: float): float
  first type mismatch at position: 2
  extra argument given
proc `-`(x: int32): int32
  first type mismatch at position: 2
  extra argument given
proc `-`(x, y: int8): int8
  first type mismatch at position: 2
  required type for y: int8
  but expression 'length' is of type: uint8
proc `-`[T: SomeUnsignedInt](x, y: T): T
  first type mismatch at position: 2
  required type for y: T: SomeUnsignedInt
  but expression 'length' is of type: uint8
proc `-`(x, y: int16): int16
  first type mismatch at position: 2
  required type for y: int16
  but expression 'length' is of type: uint8
proc `-`(x: int): int
  first type mismatch at position: 2
  extra argument given
proc `-`(x, y: float): float
  first type mismatch at position: 2
  required type for y: float
  but expression 'length' is of type: uint8
proc `-`(x: int64): int64
  first type mismatch at position: 2
  extra argument given
proc `-`(x, y: int64): int64
  first type mismatch at position: 2
  required type for y: int64
  but expression 'length' is of type: uint8
proc `-`(x, y: int): int
  first type mismatch at position: 2
  required type for y: int
  but expression 'length' is of type: uint8
1 other mismatching symbols have been suppressed; compile with --showAllMismatches:on to see them

Hi there,
thanks for this library!

At the moment I would be particularly interested in reading "gzipped" files but I didn't find the corresponding documentation. In particular, I need to read large files that would not fit in memory, either as streams or line by line.

Could you please help me with this?
Thank you very much

Andrea

This is not a call to arms, just something for me to remember that I need to look more into this. I am able to use 32-bit RISC-V just fine with zippy, but the signed shifts cause problems on 64-bit.

So what gives? Well, on RISC-V the arithmetic shifts (SRAI instructions primarily) will drag the sign bit with them down. So any time you have a signed value it will still be signed afterwards.
Example (just imagine that these are 64-bit signed integers instead of 16-bit):
0x8000 SRAI 8 == 0xFF80
0x8000 SRAI 12 == 0xFFF8
0x8000 SRAI 15 == 0xFFFF

Exactly which lines of code cause SRAI to be emitted is not known to me, but I believe it wouldn't be emitted for unsigned shifts.

It's a simple program that compresses and decompresses a string using gzip, it crashes on 32-bit machines (tested on Arch Linux 32 with Nim 1.4.2):

import zippy
let data = compress("Just for testing", BestCompression, dfGzip)
echo uncompress(data)

Crash log:

$ ./zippy_test                                                                                                     
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy.nim(84) zippy_test
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy.nim(38) compress
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy/deflate.nim(223) deflate
/home/victor/.nimble/pkgs/zippy-0.5.1/zippy/lz77.nim(62) lz77Encode
/usr/lib/nim/system/fatal.nim(49) sysFatal
Error: unhandled exception: value out of range: 2323877985 notin -2147483648 .. 2147483647 [RangeDefect]

This may be an issue with me being new to nim, but if I want to keep a reference to a ZipArchiveReader it's not exposed:

import zippy

type
  Null0Game* = object
    files*: ZipArchiveReader

proc newNull0Game*(filename: string): Null0Game =
  ## Create a new Game instance
  var game: Null0Game
  game.files = openZipArchive(filename)
  return game

I get undeclared identifier: 'ZipArchiveReader'. I think it just needs a few * to allow type-reference.

I am happy to PR for this, if this is the right way.

When using zippy to extract zip files, I've noticed it mess up file permissions on Linux. Not sure what's causing it, but here's an example of it happening.

example.zip

import zippy/ziparchives

extractAll("example.zip", "example")

results in

.-----x---    overflow1.png ... overflow10.png
.rw-r--r--    overflowneg1.png ... overflowneg10.png

instead of

.rw-r--r--    overflow1.png ... overflow10.png
.rw-r--r--    overflowneg1.png ... overflowneg10.png

Since 8c43184 (product of git bisect and tested on my local Mac) nimble test fails at

I am trying to store files so that they keep the desire structure within the zipfile.

I see that there is addDir with signature: proc addDir(archive: ZipArchive, base, relative: string). I think it would be a good idea to expose it.

Something similar for addFile would be appreciated. nim-lang/zip has it.

Extracting with tar:

$ tar --extract --file=./pcre-8.45.tar.gz
$ stat -c%a ./pcre-8.45/configure
755

Extracting with zippy:

import pkg/zippy/tarballs

extractAll(tarPath = "./pcre-8.45.tar.gz", dest = "./pcre")

Result:

$ stat -c%a ./pcre/pcre-8.45/configure
75

Tarball: https://megalink.dl.sourceforge.net/project/pcre/pcre/8.45/pcre-8.45.tar.gz

zip file from:
https://github.com/google/material-design-icons/archive/refs/heads/master.zip

import zippy/ziparchives, zip/zipfiles

# -----------------------------------

let filename = "material-design-icons-master.zip"
let z = openZipArchive(filename)
var ix : int64
echo "zippy"
for f in z.walkFiles:
    inc(ix)
echo ix," entries"

# -----------------------------------

ix = 0
var z1: zipfiles.ZipArchive
discard z1.open(filename)
echo "zip"
for f in z1.walkFiles:
    inc(ix)
echo ix," entries"

results

zippy
48629 entries
zip
952432 entries

environment

nim  1.6.2 stable
Linux Mint 64

Hey!

First time finally using your library. Thanks for your work first of all. :)

While trying to see how it performs on some of my larger .tar.gz archives, I noticed that for one of the larger ones extraction via extractAll fails with a size verification error. The issue seems to be that the archive is larger than int32.high, due to the size check here:

https://github.com/guzba/zippy/blob/master/src/zippy/gzip.nim#L76

Given that isize is already a uint32 why do we mod the length of dst by (1 shl 31) and not allow the full uint32 range?

My archive has 476,932 files with a total size of 3,497,146,858 bytes (according to du . -b in the extracted directory).

I also wondered what would happen if we fully exceed the size of uint32, but with a quick try I wasn't able to build a .tar.gz archive that actually enters the uncompressGzip proc. Instead it seems to go here https://github.com/guzba/zippy/blob/master/src/zippy/tarballs.nim#L51 instead (I didn't actually check that, I only noticed that uncompressedGzip wasn't being run for my ~8 GB deflated archive).

Code to reproduce is pretty much just:

import zippy/tarballs
const tmp = "/tmp/foobar"
extractAll("/path/to/file.tar.gz", tmp)

I didn't open a PR, because I assume the line mentioned above is the way it is for a good reason. If it helps I can also send you the archive via google drive or w/e.

Thanks!

Since zip and this library doesn't support reusing the same z_stream context for multiple messages.

In my own discord library, I would like to implement some transport compression which neither zip or zippy have this functionality of creating a zlib inflation context to run chunks through. (https://discord.com/developers/docs/topics/gateway#transport-compression-transport-compression-example)

Zip files created with zippy (at least on Windows 64) do not preserve the date/time of the original files. Instead, everything is dated 1980-00-00 00:00.

Another nim package, zip/zipfles, does preserve time/dates but it doesn't work on Windows 64. I'd really like to use zippy. Is there a way?

I'm really glad for this library in our ZIP-utils.

We have started to use the library to create ZIP archives, which works fine until the ZIP files are larger than the available memory. In those cases we are getting the "out of memory".

Are there any hacks to overcome this? Procedures for using a stream, adding individual files instead of directories, or is it there no solution to this yet?

I'd love to see zippy support bzip2 extraction, there doesn't seem to be a working bz2 solution for Nim at the moment

I have seen memory usage as high as 4gb for certain tarballs, for example https://github.com/nim-lang/csources/archive/64e34778fa7e114b4afc753c7845dee250584167.tar.gz.