brimworks / lua-zlib Goto Github PK

"_lua_type", referenced from:
_lz_filter_impl in lua_zlib.c.o
_lz_checksum in lua_zlib.c.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [zlib.so] Error 1
make[1]: *** [CMakeFiles/cmod_zlib.dir/all] Error 2
make: *** [all] Error 2

I'm getting this while trying to install.

> luarocks install lua-zlib 1.2-1;
Installing https://luarocks.org/lua-zlib-1.2-1.rockspec
Cloning into 'lua-zlib'...
fatal: remote error:
  The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.

Error: Failed cloning git repository.
root@millipede:~# luarocks install lua-zlib
Installing https://luarocks.org/lua-zlib-1.2-1.rockspec
Cloning into 'lua-zlib'...
fatal: remote error:
  The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.

I'm able to install lua-zlib using LuaRocks on my Windows system. However, when trying to load lua-zlib, using require("zlib"), I get an OS error dialog for lua.exe, saying "entry point crc32_combine could not be located in the dynamic link library zlib.dll".

Upon further inspection, I found out that Windows will not load the zlib.dll that is the dependency of (lua-)zlib.dll, because it thinks it's already loaded, and it will try to load the zlib entrypoints from the lua-zlib DLL. From MS docs: If a DLL with the same module name is already loaded in memory, the system uses the loaded DLL, no matter which directory it is in. The system does not search for the DLL.

There are two fixes for this, neither seems currently doable:

• rename lua-zlib's DLL to some other name (which also changes the "require string", as asked in #24)
• use static zlib

Can you advise what to do now? Thanks

Hi,

I have been using lua-zlib for last four months with lua-5.1 and Nginx , it is working fine too. But today when I installed LuaJit-2.0.2 and tried to use this module it gives me error.

error loading module 'zlib' from file '/usr/lib64/zlib.so':
/usr/lib64/zlib.so:1: unexpected symbol near 'char(127)'
stack traceback:

I use this file path when using with Lua-5.1 : /usr/lib64/lua/5.1/zlib.so .

After installing LuaJIT , lua script could not read file on above path so I copied it to /usr/lib64/zlib.so but now I am getting above mentioned error.

I am sure there must not be anything wrong with lua-zlib but something with my configuration. Would you help me track down the issue?

Thanks

I realise 8d07669 has been in this repo for quite a while, but it has an unfortunate side-effect when creating archives which is affecting downstream projects that are using lua-zlib (see mamedev/mame#3774 and libretro/mame#69).

Would it be possible to partially revert this change, removing $Format:%d$ so that git archive produces a stable result? This is important when downloading git archives and using a checksum to verify the download. As it stands now, the contents of the archive changes when additional commits are added to the repo, even though the rev used to retrieve the archive hasn't changed.

version 74d8451

-- The C compiler identification is GNU 4.5.4
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Found Lua51: /usr/lib64/liblua.so;/usr/lib64/libm.so (found version "5.1.5")
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7")
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/p/src/jabber/lua-zlib
Scanning dependencies of target cmod_zlib
[100%] Building C object CMakeFiles/cmod_zlib.dir/lua_zlib.c.o
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:18:30: error: expected '=', ',', ';', 'asm' or     '__attribute__' before 'OF'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:19:36: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'OF'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:27:42: error: expected declaration specifiers or '...' before 'checksum_t'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:27:63: error: expected declaration specifiers or '...' before 'checksum_combine_t'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c: In function 'lz_checksum':
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:293:9: error: 'checksum_combine_t' undeclared (first use in this function)
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:293:9: note: each undeclared identifier is reported only once for each function it appears in
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:293:28: error: expected ';' before 'combine'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:319:9: error: 'checksum_t' undeclared (first use in this function)
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:319:20: error: expected ';' before 'checksum'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c: At top level:
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:340:42: error: expected declaration specifiers or '...' before 'checksum_t'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:340:63: error: expected declaration specifiers or '...' before 'checksum_combine_t'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c: In function 'lz_checksum_new':
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:341:30: error: 'checksum' undeclared (first use in this function)
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:342:30: error: 'combine' undeclared (first use in this function)
/mnt/p/src/jabber/lua-zlib/lua_zlib.c: In function 'lz_adler32':
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:350:5: error: too many arguments to function 'lz_checksum_new'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:340:12: note: declared here
/mnt/p/src/jabber/lua-zlib/lua_zlib.c: In function 'lz_crc32':
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:354:5: error: too many arguments to function 'lz_checksum_new'
/mnt/p/src/jabber/lua-zlib/lua_zlib.c:340:12: note: declared here
make[2]: *** [CMakeFiles/cmod_zlib.dir/lua_zlib.c.o] Error 1
make[1]: *** [CMakeFiles/cmod_zlib.dir/all] Error 2
make: *** [all] Error 2

this library has the same 'require' string as lzlib

Installing this module can cause some weird failures around the place, including luarocks, which uses lzlib when installed

Could you rename the shared object lua-zlib gets installed as?

Hi,
I am trying to use lua-zlib to unzip the gzipped HTTP request that Nginx receives. Can you please help me with the steps i need to follow to add lua-zlib module and re-compile Nginx. I tried once and it said cannot find lua-zlib/config

the result of "lua test.lua" like this:

cc@ubuntu:~/git/lua-zlib$ lua test.lua
1..9
ok 1 - eof is true (true)
ok 2 - bytes in is greater than bytes out?
ok 3 - bytes out is the same size as deflated string length
ok 4 - bytes in is the same size as the input string length
ok 5 - Expected 8192 Xs got 8192
ok 6
ok 7 - 'abcdefghijklmnopqrstuv' == 'abcdefghijklmnopqrstuv'
ok 8 - large string
ok 9 - empty string
ok 10
ok 11 - InvalidInput error (InvalidInput: input string does not conform to zlib format or checksum failed at lua_zlib.c line 994)
EOF round
ok 12 - streaming works
ok 13 - IllegalState error (IllegalState: calling deflate function when stream was previously closed)
ok 14 - major version 1 == 1
ok 15 - minor version is number (2)
ok 16 - patch version is number (11)
ok 17 - Tom MacWright Test

And in the source code lua_zlib.c line 994. There is a comment:

do {
    stream->next_out  = (unsigned char*)luaL_prepbuffer(&buff);
    stream->avail_out = LUAL_BUFFERSIZE;
    result = filter(stream, flush);
    if ( Z_BUF_ERROR != result ) {
        /* Ignore Z_BUF_ERROR since that just indicates that we
         * need a larger buffer in order to proceed.  Thanks to
         * Tobias Markmann for finding this bug!
         */
        lz_assert(L, result, stream, __FILE__, __LINE__);
    }
    luaL_addsize(&buff, LUAL_BUFFERSIZE - stream->avail_out);
} while ( stream->avail_out == 0 );

Just ask this bug has been fixed or not?

I do some PR that I want to use in a chef cookbook.

I just want to know when this PR will be merged.
And more important, when a new release will be unleashed.

Hello ,I need help.When I install lua-zlib on mac, some error message as following:
[ 50%] Building C object CMakeFiles/cmod_zlib.dir/lua_zlib.c.o
[100%] Linking C shared module zlib.so
Undefined symbols for architecture x86_64:
"_luaL_argerror", referenced from:
_lz_checksum in lua_zlib.c.o
"_luaL_buffinit", referenced from:
_lz_filter_impl in lua_zlib.c.o
"_luaL_checklstring", referenced from:
_lz_checksum in lua_zlib.c.o
"_luaL_checkoption", referenced from:
_lz_filter_impl in lua_zlib.c.o
"_luaL_newmetatable", referenced from:
_lz_create_deflate_mt in lua_zlib.c.o
_lz_create_inflate_mt in lua_zlib.c.o

what should I do??

I used it as following
local zlib = require("zlib")
local inflate_stream = zlib.inflate()
local inflated_body = inflate_stream(body)

        local reverted_rbody, _ = string.gsub(inflated_body, "AAA", "BBB")
        local deflate_stream = zlib.deflate()
        body = deflate_stream(reverted_rbody, 'full')

I want to unzip it, replace something, then zip it again.
But the re-compressed content got another format, that was not a gzip again

Not sure why/how this is happening yet, but here's a test case:

lua code: https://gist.github.com/2411042

test file: http://dl.dropbox.com/u/68059/test_zlib

Test against ruby zlib:

ruby -rzlib -e 'print Zlib::Inflate.new.inflate(STDIN.read)' < test_zlib

lua-zlib output:

commit 234  true    168 245

ruby output:

ruby -rzlib -e 'print Zlib::Inflate.new.inflate(STDIN.read)' < test_zlib 
commit 234tree b0d387e58ba8f97e64e43e440d6e226a454cc60c
parent 7457837869dbf40e91719b5801301f465867d1c3
author Tom MacWright <[email protected]> 1313704231 -0400
committer Tom MacWright <[email protected]> 1313704231 -0400

And those ones.

Function test_tom_macwright() and test_amnon_david() in the file "test.lua", exec failed because src_dir is nil.
Maybe should use current pwd instead.

I compiled/installed via luarocks:

$ sudo luarocks install lua-zlib
Installing https://rocks.moonscript.org/lua-zlib-0.3-1.rockspec...
Using https://rocks.moonscript.org/lua-zlib-0.3-1.rockspec... switching to 'build' mode
Cloning into 'lua-zlib'...
remote: Counting objects: 17, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 17 (delta 0), reused 11 (delta 0)
Receiving objects: 100% (17/17), 11.82 KiB | 0 bytes/s, done.
Checking connectivity... done.
Note: checking out 'c0014bcbc4c3fd65ba3519b10965f0c184c1a059'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

gcc -O2 -fPIC -I/usr/include -c lua_zlib.c -o lua_zlib.o
gcc -shared -o zlib.so -L/usr/lib lua_zlib.o
Updating manifest for /usr/lib/luarocks/rocks-5.2
lua-zlib 0.3-1 is now built and installed in /usr (license: MIT)

However, when trying to run (e.g, via lua -lzlib), I get:

/usr/lib/lua/5.2/zlib.so: undefined symbol: inflate

The fix seems to be renaming the shared object, e.g. to lua-zlib.so.
I assume this is due to the dynamic linker getting confused?

nginx.conf:

location /test {
          echo "hello, world!";
          content_by_lua '
            local zlib = require "zlib"
            ngx.log(ngx.DEBUG, "test-------")
            local stream = zlib.deflate()("123456","finish")
            ngx.say(stream)
          ';
        }

How can I install Lua-zlib:

1. update INCDIR to "?= -I/usr/local/include/lua5.4" in Makefile
2. make macosx
3. cp zlib.so /usr/local/openresty/lualib/

when I send a request to 127.0.0.1/test, I got the following logs:

2022/04/07 18:10:21 [notice] 78887#0: signal 20 (SIGCHLD) received from 78901
2022/04/07 18:10:21 [alert] 78887#0: worker process 78901 exited on signal 11
2022/04/07 18:10:21 [notice] 78887#0: start worker process 78922
2022/04/07 18:10:21 [notice] 78887#0: signal 23 (SIGIO) received

I notice the following code snippet in lz_filter_impl function:

/*  Do the actual deflate'ing: */
if (lua_gettop(L) > 0) {
    stream->next_in = (unsigned char*)lua_tolstring(L, -1, &avail_in);
} else {
    stream->next_in = NULL;
    avail_in = 0;
}

however there is always a luaL_buffinit before this, so the stack may always larger than 0 ? Thanks for explanation :)

Thank you for providing this tool！
Already used in production environment。
Use it to process gzip body data。
But the processing speed drops by 60% after enabling。
Close the decompression program, processing capacity 600MB/S,but after turning it on, 200MB/S。
Do you have good suggestions and ways？
This is lua code：
ngx.ctx.max_chunk_size = tonumber(ngx.var.max_chunk_size) #：10kb
ngx.ctx.max_body_size = tonumber(ngx.var.max_body_size) #：1M

function inflate_chunk (stream, chunk)
return stream(chunk)
end

function inflate_body (data)
local stream = require("zlib").inflate()
local buffer = ""
local chunk = ""

for index = 0, data:len(), ngx.ctx.max_chunk_size do
chunk = string.sub(data, index, index + ngx.ctx.max_chunk_size - 1)
local status, output, eof, bytes_in, bytes_out = pcall(stream, chunk)
buffer = buffer .. output
end

return buffer
end

local content_encoding = ngx.req.get_headers()["Content-Encoding"]
if content_encoding == "gzip" then
ngx.req.read_body()
local data = ngx.req.get_body_data()

if data ~= nil then
local new_data = inflate_body(data)
ngx.req.set_body_data(new_data)
end
end

Building lua-zlib on windows 10

Building zlib using cmake on VS dev

using luarocks to install

I believe the issue lies withing the rockspec. Looking at the 1.2-2.rockspec

When replacing lines 37-41 of the rockspec with

platforms = { windows = { variables = { LUA_LIBRARIES = "$(LUA_LIBDIR)/$(LUALIB)" }}},

it fixes the build issue on windows.

I am unsure if this is an issue from my end (or even the correct fix), but wanted to mention I noticed this when building.

I'm using lua-zlib as shared library (zlib.so)

local zlib = require "zlib"
local stream = zlib.deflate()
local deflated, eof, bytes_in, bytes_out = stream(str, "finish")
...

everything goes fine, but lz_deflate_delete and lz_inflate_delete never be called, and it may raise memory leak ?

[test on Lua 5.3]

make linux
......
gcc -O -shared -fPIC -L/usr/local/lib -L/usr/local/lib lua_zlib.o -lz -llua -lm -o zlib.so
/usr/bin/ld: /usr/local/lib/liblua.a(lapi.o): relocation R_X86_64_32 against `luaO_nilobject_' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/liblua.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[1]: *** [zlib.so] Error 1

The current stable release of Lua is now 5.4.0. This module blocks anything newer than 5.3.X in the rockspec.

Can we please remove the upper bounds description and figure out what (if anything) else needs doing to support Lua 5.4.

Hi,

I'm using LuaJIT 2.1.0-beta3, when I try to run the tests I get the following output:

1..9
ok 1 - eof is true (true)
ok 2 - bytes in is greater than bytes out?
ok 3 - bytes out is the same size as deflated string length
ok 4 - bytes in is the same size as the input string length
ok 5 - Expected 1024 Xs got 1024
ok 6
ok 7 - 'abcdefghijklmnopqrstuv' == 'abcdefghijklmnopqrstuv'
ok 8 - large string
ok 9 - empty string
PANIC: unprotected error in call to Lua API (InvalidInput: input string does not conform to zlib format or checksum failed at lua_zlib.c line 994)

Running it with Lua 5.1 is OK, no panic and execution continues.

Can something be done to avoid panic?

Can I check for invalid input before calling inflate?

Thanks!

Can someone set me straight on these?

When running lua test.lua I get this error.
lua: test.lua:8: module 'zlib' not found:
no field package.preload['zlib']
no file './zlib.lua'
no file '/usr/local/share/lua/5.2/zlib.lua'
no file '/usr/local/share/lua/5.2/zlib/init.lua'
no file '/usr/local/lib/lua/5.2/zlib.lua'
no file '/usr/local/lib/lua/5.2/zlib/init.lua'
no file './zlib.lua'
no file '/usr/share/lua/5.2/zlib.lua'
no file '/usr/share/lua/5.2/zlib/init.lua'
no file './zlib.lua'
no file './zlib.so'
no file '/usr/local/lib/lua/5.2/zlib.so'
no file '/usr/lib/x86_64-linux-gnu/lua/5.2/zlib.so'
no file '/usr/lib/lua/5.2/zlib.so'
no file '/usr/local/lib/lua/5.2/loadall.so'
no file './zlib.so'
stack traceback:
[C]: in function 'require'
test.lua:8: in main chunk
[C]: in ?

When running make linux I get this error:
**make[1]: Entering directory `/home/josh/lua-zlib'
gcc -O -shared -fPIC -L/usr/lib -L/usr/lib lua_zlib.o -lz -llua -lm -o zlib.so
/usr/bin/ld: cannot find -llua
collect2: ld returned 1 exit status
make[1]: * [zlib.so] Error 1

I have installed lua5.1 and lua5.2 as well. Any help you can provide?

Currently the rockspec claims compatibility with only >= 5.1 < 5.3.

i don't know for sure, on which side is the problem:

luajit test.lua is crashing on the very last test 11 - IllegalState error (IllegalState: calling deflate function when stream was previously closed)

LuaJIT 2.0.0-beta6
OS: win7

when I do this

    local zlib = require ("zlib")

nginx error log would show:
[warn] 30914#30914: *24397 [lua] _G write guard:12: writing a global lua variable ('zlib') which may lead to race conditions between concurrent requests, so prefer the use of 'local' variables
stack traceback:
[C]: at 0x7f165255b755
[C]: in function 'require'

any light ?

Calling inflate with a numeric window_size argument (needed for extracting .zip files) does not work, because it switches into lzlib mode by accident in the current master (9ebfe31).

Pull request #39 fixes this.

I'm using lua-zlib for streaming decompression of gzipped data coming into OpenResty as multipart/form-data. While everything is working great for the vast majority of files I've tested, I have run into an odd edge case (see links at bottom).

Specifically, for some files, it looks like the stream(data) operation for an inflate stream is erroneously returning eof = true early, and then I get the following error message when trying to inflate the next chunk of the file:

2018/03/06 00:46:22 [error] 7#7: *2 lua entry thread aborted: runtime error: IllegalState: calling inflate function when stream was previously closed
stack traceback:
coroutine 0:
	[C]: in function 'stream'
	/app/./lua/gunzip_upload.lua:28: in function </app/./lua/gunzip_upload.lua:1>, client: 172.17.0.1, server: , request: "POST /gunzip_upload HTTP/1.1", host: "localhost:8080"

In looking at this further, it appears stream() is returning eof = true, despite being in the middle of the file.

Setup details
Specifically, I'm using lua-resty-upload and then just uncompressing the stream with some code that looks like this (modified from their example for here, this is just a content_by_lua block in OpenResty).

Here's a minimal example of my lua script (gunzip_upload.lua):

local zlib = require "zlib"
local upload = require "resty.upload"

local stream = zlib.inflate(31)
local chunk_size = 4096
local form = upload:new(chunk_size)
local file
while true do
    local typ, res, err = form:read()

    if not typ then
         ngx.say("failed to read: ", err)
         return
    end

    if typ == "header" then
        -- Contrived example to just unzip a file and write it out as "unzipped.txt"
        -- This will *only* work with gzipped uploads
        local file_name = "unzipped.txt"
        if file_name then
            file = io.open(file_name, "w+")
            if not file then
                ngx.say("failed to open file ", file_name)
                return
            end
        end

     elseif typ == "body" then
        if file then
            body, gzip_eof, bytes_in, bytes_out = stream(res)
            file:write(body)
        end

    elseif typ == "part_end" then
        file:close()
        file = nil

    elseif typ == "eof" then
        break

    else
        -- do nothing
    end
end

And an example nginx.conf:

worker_processes 1;
error_log logs/error.log;

events {
  worker_connections 256;  
}

http {
  server {
    listen 8080;
    client_max_body_size 5G;
    client_body_buffer_size 4M;
    keepalive_timeout 300s;
    proxy_read_timeout 600s;
    lua_code_cache off;  # For dev, turn `off` for hot reloading!

    location = /gunzip_upload {
      if ($request_method != POST ) {
        return 405;
      }

      content_by_lua_file ./lua/gunzip_upload.lua;
    }
  }
}

(Note you can actually just inline gunzip_upload.lua with a content_by_lua block)

Unfortunately the file I have where I can reliably reproduce this is 1.9GB. But this still triggers the failure for the first 40MB chunk, though note that's not actually a valid gzip file as it isn't complete.

• Full file (~2GB): link
• 40MB file: link

Many thanks in advance for the help, the library is otherwise working great for our needs! 👏