Non-deterministic output observed about lz4 HOT 7 CLOSED

Hey @pierrec, I'm working with Peter on Loki and today I tried the dig down more into the problem. It was indeed the hashtable not being reset causing this non deterministic output.

I have a repro and I was able to fix it by adding this in the reset function

	// reset hashtable to ensure deterministic output.
	for i := range z.hashtable {
		z.hashtable[i] = 0
	}

I'm sending you a PR, let me know what you think !

from lz4.

I'm attaching two LZ4 streams generated as described above (but ealier debug output is not related to these files).

files.zip

Here is program I've used to get lz4debug output:

// +build lz4debug

package main

import (
        "flag"
        "io"
        "log"
        "os"

        "github.com/pierrec/lz4"
)

func main() {
        flag.Parse()

        for _, f := range flag.Args() {
                openLz4(f)
        }
}

func openLz4(filename string) {
        f, err := os.Open(filename)
        if err != nil {
                log.Println("failed to open file", filename, "due to error:", err)
                return
        }
        defer f.Close()

        r := lz4.NewReader(f)

        _, err = io.Copy(os.Stdout, r)
        if err != nil {
                log.Println("error while reading file", filename, ":", err)
        }
}

from lz4.

Thanks. I will look into it.
Also, although I guess you would have mentioned but I ask anyway:
did you have a chance to disable reuse of writers and if so, do you observe any issue?

from lz4.

Thanks. I will look into it.
Also, although I guess you would have mentioned but I ask anyway:
did you have a chance to disable reuse of writers and if so, do you observe any issue?

Thanks. No, we only tried switching to Snappy, which helped. We haven't tried not reusing lz4 writers, but I'll see if we can try that.

from lz4.

Any news on this?

from lz4.

Any news on this?

Unfortunately not. I dropped the ball here and haven't yet tried to not reuse writers. Problem is that I cannot reproduce it locally, and trying it on our busy cluster, where we have observed this problem may blow up memory allocations too much. I still would like to try it at some point though.

from lz4.

No problem, thx for getting back to me. I am too busy atm to look at this in details, but any news is welcome!

from lz4.