luben / zstd-jni Goto Github PK

When trying to use zstd-jni in a docker container build from openjdk:alpine, I get this error:

Caused by: java.lang.UnsatisfiedLinkError: /tmp/libzstd3471839276809832200.so: Error relocating /tmp/libzstd3471839276809832200.so: __fprintf_chk: symbol not found
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
        at java.lang.Runtime.load0(Runtime.java:809)
        at java.lang.System.load(System.java:1086)
        at com.github.luben.zstd.util.Native.load(Native.java:78)
        at com.github.luben.zstd.ZstdInputStream.<clinit>(ZstdInputStream.java:22)

Problem seems to be that the libzstd.so bundled in the jar links to glibc. Adding one built with musl would allow it to work in a musl environment.

Hi,
sorry to bother again, but after some testing with the new 1.3.4-6 release I bumped into an issue I can't really explain.
In short I have a gradle project and the stacktraces I get in pre-prod are the following:

java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in you system.
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.github.luben.zstd.util.Native.load(Native.java:101)
	at com.github.luben.zstd.ZstdOutputStream.<clinit>(ZstdOutputStream.java:17)

and

java.lang.NoClassDefFoundError: Could not initialize class com.github.luben.zstd.ZstdOutputStream

Surprisingly everything seems to be working fine in my local linux environment and I can't reproduce the issue there.

In the effected environment the exception is getting raised reliable at the same places, indicating that something? (not sure what) is broken. I have run tests with several versions that are available on the maven repo and this is the current results based on the used version

1.3.4-6 - broken with exception
1.3.4-5 (seems non existent in git repo) - broken with exception below
1.3.4-4 (seems not to be aligned with version in git repo) - causes the IO Exception from #59
1.3.4-3 - causes the IO Exception from #59
1.3.4-2 - causes the IO Exception from #59

When I checkout the zstd-jni project locally and build it via sbt-1.1.4/sbt/bin/sbt publishLocal or even ./sbt publishLocal and use the artifact as dependency in my project via files('..../.ivy2/local/com.github.luben/zstd-jni/1.3.4-fixed/bundles/zstd-jni.jar') everything is working fine too. For me it seems that something? in the deployment is broken, or I'm not doing the same steps (not sure).
Would kindly appreciate any advice on how to debug this further.

using 1.3.1-1(newest version as of writing) on Mac OS X 10.10.5(64bit):

java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
Unsupported OS/arch, cannot find /darwin/x86_64/libzstd-jni.dylib or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in you system. ClassLoader.loadLibrary(ClassLoader.java:1867)

as I recall, previous versions of zstd-jni worked on my OS

Hello, will analogue for DeflateOutputStream SYNC_FLUSH (https://blogs.oracle.com/xuemingshen/entry/the_flushable_deflateroutputstream) be available anytime soon?
I'm using DeflateOutputStream and InflateInputStream in network communication to reduce cross-DC traffic and it works great but eats a lot of CPU
Tested ZStd. for data we are sending by network and it shows great compression levels (even better then zlib). But ZStdOutputStream.flush() method doesn't makes a flushed data be available for decompression at the Input side, unless buffer is full.

How long would it take to supply /windows_8.1/amd64 build with the library? Or where to find instructions how to build it myself?

I need to use compress with dictionary. After add samples to ZstdDictTrainer, and when calling trainSamples it crashes and makes this error:

art/runtime/java_vm_ext.cc:470] JNI DETECTED ERROR IN APPLICATION: jfieldID int java.nio.Buffer.capacity not valid for an object of class int[]
art/runtime/java_vm_ext.cc:470] in call to GetIntField
art/runtime/java_vm_ext.cc:470] from long com.github.luben.zstd.Zstd.trainFromBufferDirect(java.nio.ByteBuffer, int[], java.nio.ByteBuffer)
art/runtime/java_vm_ext.cc:470] "main" prio=5 tid=1 Runnable
art/runtime/java_vm_ext.cc:470] | group="main" sCount=0 dsCount=0 obj=0x74fac610 self=0xa988b400
art/runtime/java_vm_ext.cc:470] | sysTid=9405 nice=-10 cgrp=default sched=0/0 handle=0xadc3e534
art/runtime/java_vm_ext.cc:470] | state=R schedstat=( 0 0 0 ) utm=28 stm=41 core=3 HZ=100
art/runtime/java_vm_ext.cc:470] | stack=0xbf683000-0xbf685000 stackSize=8MB
art/runtime/java_vm_ext.cc:470] | held mutexes= "mutator lock"(shared held)
art/runtime/java_vm_ext.cc:470] native: #00 pc 00572f1e /system/lib/libart.so (_ZN3art15DumpNativeStackERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEEiP12BacktraceMapPKcPNS_9ArtMethodEPv+238)

It would be nice to extend the ByteBuffer interface with a way to stream through the byte buffer. At the moment, the whole ByteBuffer source is read, and especially with large files, this can require a too large target ByteBuffer.

The primary use case would be memory mapped byte buffers from nio. These are especially interesting for reading files. I was working on a input stream interface around these and noticed that the interface for ByteBuffers are not streaming aware.

I think it would be nice to have a destination buffer of 128K, and read from the src buffer in a streaming fashion, similar to the input stream.

I'm not sure what would be the best solution, create a version of decompressDirectByteBuffer where the long encodes 2 ints, the bytes read from input, and the bytes written to output? Or is the native zstd streaming interface more complicated than that, and do I need to create a similar class as the ZstdInputStream?

It was left behind from the 1.0.0 release. We need it in order to be able to consume files that are not yet finished.

Hi,
When I use 'zstd-jni' in Suse11Sp3x64 or CentOS7x64 , the error described in the title would be throwed.
Could you fix this problem? Thank you.
Best Regards.

Hi first of all thanks for this project!
I'm not quite sure if this is an edge case, but I came across an issue that seems to be caused when using the ZstdOutputStream with closeFrameOnFlush=true and issuing a flush before closing a BufferedWriter. In general when attempting to the read the data back again it causes a "Underlying input stream returned zero bytes" Exception. I feel it would be most help full to provide a full working minimal example (see blow) to reproduce this issue. I'm fully aware that close itself causes a flush, but in my case I want to "force-fully" flush every X-Elements. IMO the compression/decompression should should remain working when performing multiple flushes (potentially resulting in flush a NO-OP if no data need to be flushed). When removing the additional flush everything works fine.

Steps to Reproduce (for bugs only)

import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;

import com.github.luben.zstd.ZstdInputStream;
import com.github.luben.zstd.ZstdOutputStream;

public class Sample {
    private static final int ZSTD_CLEVEL_DEFAULT = 3;

    public static void main(String[] args) throws Throwable {
        final File compressedFile = createTempFile("random_compressed");
        final File uncompressed = createTempFile("random_extracted");
        decompressFile(compressedFile, uncompressed);

        Files.lines(uncompressed.toPath()).forEach(System.out::println);
    }

    private static void createCompressedFile(final File compressedFile){
        try (final BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
                new ZstdOutputStream(
                        Files.newOutputStream(compressedFile.toPath()),
                        ZSTD_CLEVEL_DEFAULT, true),
                StandardCharsets.UTF_8))) {
            for (int i = 0; i < 20; i++) {
                writer.write(i + System.lineSeparator());
            }

            // when using an additional flush before the try-with-resources closes the stream the execution yields "java.io.IOException: Underlying input stream returned zero bytes"
            writer.flush();
        } catch (final IOException e) {
            e.printStackTrace();
        }
    }

    private static void decompressFile(
            final File compressedFile, final File uncompressed) throws IOException {
        final ZstdInputStream zstdInputStream = new ZstdInputStream(
                Files.newInputStream(compressedFile.toPath())).setContinuous(true);
        final BufferedReader fileReader = new BufferedReader(
                new InputStreamReader(zstdInputStream, StandardCharsets.UTF_8));

        try (final BufferedWriter fileWriter = Files.newBufferedWriter(uncompressed.toPath(),
                StandardCharsets.UTF_8)) {
            String line;
            while ((line = fileReader.readLine()) != null) {
                fileWriter.write(line);
            }
        }
    }

    private static File createTempFile(final String prefix) throws IOException {
        final File tempFile = File.createTempFile(prefix, ".tmp");
        tempFile.deleteOnExit();
        return tempFile;
    }
}

Environment

com.github.luben:zstd-jni:1.3.3-1

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

Would you like to support linux_x86_64? Or I want to know what cause it. Thanks.

Hi,

Would be awesome to have javadoc published somewhere and/or have some examples in readme for quick getting started with the library.

Thanks

Can you add libzstd-jni.so for linux/ppc64le to a binary release? I have access to a ppc64le machine, so I can contribute it. Or, would you like to build it yourself?

Hi,
I'm pretty sure this is one of the annoying corner cases again :-P

In short I think the handling of the decompression of empty files should be handled more gracefully.
I bumped into a similar issue when attempting to decompress empty files with zstd and continuous=true. I now use zstd for another use-case where I don't necessarily need continuous=true and noticed that zstd fails with Read error or truncated source while trying to decompress an empty file with continuous=false .

I think the root cause is residing in the fact that the ZstdInputStream gets initialized with frameFinished = false even though we never read a frame. When opening an empty file with Files.newInputStream(originalFile.toPath()) no exception is thrown and I believe this is exactly how ZstdInputStream should handle the case with continuous=false (implementation still can fail with continuous=true). As of now it seems that ZstdInputStream only can handle files with at least one frame.

Sample-Code

package com.nuance.stark.internal.dataflow;

import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;

import com.github.luben.zstd.ZstdInputStream;

public class Sample {
    public static void main(String[] args) throws Throwable {
        final File file = createTempFile("random_compressed");

        try {
            readFileWithFis(file);
        } catch (final IOException e) {
            System.out.println("Exception 1: " + e.getMessage());
        }

        try {
            readFileWithZstd(file, true);
        } catch (final IOException e) {
            System.out.println("Exception 2: " + e.getMessage());
        }

        try {
            readFileWithZstd(file, false);
        } catch (final IOException e) {
            // OOF: IMHO not needed to throw an exception
            System.out.println("Exception 3: " + e.getMessage());
        }
    }

    private static void readFileWithFis(final File originalFile)
            throws IOException {
        final InputStream inputStream = Files.newInputStream(originalFile.toPath());
        try (final BufferedReader fileReader = new BufferedReader(
                new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
            String line;
            while ((line = fileReader.readLine()) != null) {
                System.out.println(line);
            }
        }
    }

    private static void readFileWithZstd(final File originalFile, boolean continuous)
            throws IOException {
        final ZstdInputStream zstdInputStream = new ZstdInputStream(
                Files.newInputStream(originalFile.toPath())).setContinuous(continuous);
        try (final BufferedReader fileReader = new BufferedReader(
                new InputStreamReader(zstdInputStream, StandardCharsets.UTF_8))) {
            String line;
            while ((line = fileReader.readLine()) != null) {
                System.out.println(line);
            }
        }
    }

    private static File createTempFile(final String prefix) throws IOException {
        final File tempFile = File.createTempFile(prefix, ".tmp");
        tempFile.deleteOnExit();
        return tempFile;
    }
}

Hello,
I was looking for File compression and File Decompression code in Java using BROTLI and ZStandard.
But all i found was a sample code using Byte[] and compressing a single string as byte buffer.

If Possible try a File Compression and File Decompression in Brotli and ZStd using Java.
Here File means any type of file without any specification.

Hi,

I get the following error when trying to compress and then uncompress random bits with 0.8.0 release. Apart from the native library loading error, what seems odd is the input stream decoder seems to detect zstd 0.6.

[info] - zstd should compress the content
[info] Exception encountered when attempting to run a suite with class name: com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest *** ABORTED ***
[info]   java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdInputStreamV06.createDCtx()J
[info]   at com.github.luben.zstd.ZstdInputStreamV06.createDCtx(Native Method)
[info]   at com.github.luben.zstd.ZstdInputStreamV06.<init>(ZstdInputStreamV06.java:57)
[info]   at com.github.luben.zstd.ZstdInputStream.<init>(ZstdInputStream.java:80)
[info]   at com.gu.contentapi.porter.integration.ZstdCompression$.uncompress(Compression.scala:59)
[info]   at com.gu.contentapi.porter.integration.ZstdCompression$.uncompress(Compression.scala:68)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$3.apply$mcV$sp(ZstdCompressionTests.scala:17)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$3.apply(ZstdCompressionTests.scala:15)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$3.apply(ZstdCompressionTests.scala:15)

My code:

import com.github.luben.zstd.ZstdInputStream
import com.github.luben.zstd.ZstdOutputStream

object ZstdCompression {

  def compress(data: Array[Byte]): Array[Byte] = {
    val bos = new ByteArrayOutputStream()
    val out = new ZstdOutputStream(bos)
    out.write(data)
    out.close()
    bos.toByteArray
  }

  def uncompress(buffer: ByteBuffer): ByteBuffer = {
    val bos = new ByteArrayOutputStream(8192)
    val in = new ZstdInputStream(new ByteBufferBackedInputStream(buffer))
    IOUtils.copy(in, bos)
    in.close()
    bos.close()
    ByteBuffer.wrap(bos.toByteArray())
  }

  def uncompress(data: Array[Byte]): Array[Byte] = {
    val buffer = ByteBuffer.wrap(data)
    val byteBuffer = uncompress(buffer)
    val result = new Array[Byte](byteBuffer.remaining)
    byteBuffer.get(result)
    result
  }

}

the test:

import com.gu.contentapi.porter.integration.ZstdCompression._
import org.scalatest.FunSuite
import org.scalatest.Matchers
import scala.util.Random

class ZstdCompressionTest extends FunSuite with Matchers {

  test("zstd should compress the content") {
    val blob = Array.fill(2048) { Random.nextInt(100).toByte }
    compress(blob).length should be < blob.length
  }

  test("ztsd and then unzstd should return the same content") {
    val blob = Array.fill(2048) { Random.nextInt(100).toByte }
    uncompress(compress(blob)) should be(blob)
  }
}

Hi,

It would be great to have support for dictionary based compression and decompression methods that operate on (direct) ByteBuffers. This would avoid additional copying when operating on off-heap buffers.

Thanks,
Viktor

It will improve the compression ratio for the small files

hi luben, I try to use zest-jni (1.1.4) lib in my android project in Android studio IDE, so I import this lib with gradle like:
dependencies {
compile fileTree(include: ['*.jar'], dir: 'libs')
androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
exclude group: 'com.android.support', module: 'support-annotations'
})
compile 'com.android.support:appcompat-v7:23.4.0'
compile 'com.android.support:design:23.4.0'
testCompile 'junit:junit:4.12'
compile 'com.github.luben:zstd-jni:1.1.4'
}
then I build my project and I can find this lib under external Libraries , I can import Zstd.java in my test java class file, here is my test code:

==================

String testJson = "{"name":"line_location", "location":"1231231231231231231231231231231323123123"}";
byte[] encoded = com.github.luben.zstd.Zstd.compress(testJson.getBytes());
Log.d(tag, "in onClick encoded result is " + encoded);

===================
but when I run my app , this test code catch exception as shown below:

Caused by: java.lang.UnsupportedOperationException: Unsupported OS/arch, cannot find /linux/aarch64/libzstd.so. Please try building from source.
at com.github.luben.zstd.util.Native.load(Native.java:55)
at com.github.luben.zstd.Zstd.(Zstd.java:11)
at com.example.hangl.zstdtest.ZstdTestActivity.onClick(ZstdTestActivity.java:68) 
at android.view.View.performClick(View.java:5212) 
at android.view.View$PerformClick.run(View.java:21214) 
at android.os.Handler.handleCallback(Handler.java:739) 
at android.os.Handler.dispatchMessage(Handler.java:95) 
at android.os.Looper.loop(Looper.java:148) 
at android.app.ActivityThread.main(ActivityThread.java:5628) 
at java.lang.reflect.Method.invoke(Native Method) 
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:853) 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:737) 

It seems like getResourceAsStream() in Native.java return null obj, then expiation throws, what's the problem with this?

We noticed crashes like this:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00007f562ea0575f, pid=32000, tid=0x00007f5646e99700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libzstd6655353904108682045.so+0x3175f]  HUF_compress4X_repeat+0x68f
#

It was only on machines with AMD Opteron(tm) Processor 4170. To fix it the compiler flag can be changed in build.sbt from -msse4 to -msse3.

-  case "amd64"|"x86_64"   => Seq("-msse4")
-  case "i386"             => Seq("-msse4")
+  case "amd64"|"x86_64"   => Seq("-msse3")
+  case "i386"             => Seq("-msse3")

get error on android 8.1, it seems can not find the so file.

byte[] bytes_c = Zstd.compress(bytes);

Process: **, PID: 32609
                                                   java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/
***
G2A1lahdmnHjLbIRu2ZjeQ==/split_lib_slice_9_apk.apk!/lib/arm64-v8a, /system/lib64, /vendor/lib64, /product/lib64]]] couldn't find "libzstd-jni.so"
                                                   Unsupported OS/arch, cannot find /linux/aarch64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in you system.
                                                       at java.lang.Runtime.loadLibrary0(Runtime.java:1011)
                                                       at java.lang.System.loadLibrary(System.java:1660)
                                                       at com.github.luben.zstd.util.Native.load(Native.java:69)
                                                       at com.github.luben.zstd.Zstd.<clinit>(Zstd.java:11)
                                                       at com.github.luben.zstd.Zstd.compress(Zstd.java:348)
                                                       at com.huawei.hiai.vision.mobilextest.MainActivity.compress(MainActivity.java:53)
                                                       at com.huawei.hiai.vision.mobilextest.MainActivity.onCreate(MainActivity.java:39)
                                                       at android.app.Activity.performCreate(Activity.java:7354)
                                                       at android.app.Activity.performCreate(Activity.java:7345)
                                                       at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1219)
                                                       at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3110)
                                                       at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3268)
                                                       at android.app.ActivityThread.-wrap12(Unknown Source:0)
04-25 21:46:53.432 32609-32609/? E/AndroidRuntime:     at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1888)
                                                       at android.os.Handler.dispatchMessage(Handler.java:109)
                                                       at android.os.Looper.loop(Looper.java:218)
                                                       at android.app.ActivityThread.main(ActivityThread.java:7384)
                                                       at java.lang.reflect.Method.invoke(Native Method)
                                                       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:469)
                                                       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:963)

currently there is no way to enable checksum, which is unsafe. could we have an option to enable checksum? thanks.

Again I fear this might be another edge-case where I assume that this might even be supported in the native library (or caused by shared stuff between reader and writer). In general what I'm currently testing if it is possible to launch reader and writer in parallel by making use of the "closeFrameOnFlush=true" and "continuous=true" (see full code below). However it seems that from the java side the frame is correctly closed and thus invokes a flushStream in the native library (https://github.com/luben/zstd-jni/blob/master/src/main/java/com/github/luben/zstd/ZstdOutputStream.java#L114). This then somehow triggers a "Context should be init first" exception.

Open Question: Is this (even) supported? Logic wise the reader should read a full frame (since the writer invoked flush and should have caused the frame to close).

Stacktrace:

org.gradle.CompressionTest > runTest FAILED
    java.io.IOException: Compression error: Context should be init first
        at com.github.luben.zstd.ZstdOutputStream.flush(ZstdOutputStream.java:116)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
        at java.io.BufferedWriter.flush(BufferedWriter.java:254)
        at org.gradle.CompressionTest.runTest(CompressionTest.java:53)

Steps to Reproduce

import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;

import com.github.luben.zstd.ZstdInputStream;
import com.github.luben.zstd.ZstdOutputStream;

import org.junit.Test;

import static org.junit.Assert.*;

public class CompressionTest {
    private static final int ZSTD_CLEVEL_DEFAULT = 3;

    @Test
    public void runTest() throws Throwable {
        try {
            final File compressedFile = createTempFile("random_compressed");
            final File uncompressed   = createTempFile("random_extracted");

            // create writer
            final BufferedWriter writer = new BufferedWriter(
                    new OutputStreamWriter(new ZstdOutputStream(
                            Files.newOutputStream(compressedFile.toPath()),
                            ZSTD_CLEVEL_DEFAULT, true),
                            StandardCharsets.UTF_8));
            // create reader
            final BufferedReader reader = new BufferedReader(
                    new InputStreamReader(
                            new ZstdInputStream(
                                Files.newInputStream(compressedFile.toPath())
                            ).setContinuous(true),
                            StandardCharsets.UTF_8));

            // feed content into writer
            for (int i = 0; i < 20; i++) {
                writer.write(i + System.lineSeparator());
            }
            // flush to cause a new frame
            writer.flush();

            // read content (should be possible since frame is closed?)
            for (int i = 0; i < 20; i++) {
                String line;
                while ((line = reader.readLine()) != null) {
                    System.out.println(line);
                }
            }

            // close
            writer.flush(); // flush or close causes an Exception
            writer.close();
            reader.close();
        } catch (final Throwable t) {
            // t.printStackTrace();
            throw t;
        }
    }

    private File createTempFile(final String prefix) throws IOException {
        final File tempFile = File.createTempFile(prefix, ".tmp");
        tempFile.deleteOnExit();
        return tempFile;
    }
}

I like the approach of having one jar with all the binaries! Makes it much nicer to distribute.

However, we would like to switch back to gzip when zstd is not available for the current platform. This is not possible at the moment, since the static initializer of Zstd class calls the Native.load method that throws the UnsupportedOperationException.

Would it be possible to have something like this:

• Put the load in a place that can be catched (for example the constructors of the Input/OutputStreams)
• Have a boolean method that returns if there is a zstd implementation available (to prevent the exception in the first place).

I have no problem adding this, but in that case I would prefer some discussion on the desired interface (to avoid having that after I implemented it).

Hi,

Thanks for the nice wrapper! Could you please also provide native builds for OS X?

Thanks,
Viktor

The current implementation supports only Java managed byte[] buffers. In my application I'm working with memory-mapped I/O buffers, in other words ByteBuffers for which isDirect() returns true. This is a special case which has an optimized JNI API for accessing the underlying void * very quickly, without pinning an array on the heap.

It would be very useful to be able to use this optimized I/O pattern with the zstd library. I have a working implementation to submit shortly.

Hi I am trying to run a jar which uses RTI connext dependencies . When I run it I get following stack trace :
java.lang.UnsatisfiedLinkError: C:\Users\477047\Downloads\configService\nddsjava.dll: Can't find dependent libraries
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at com.rti.dds.util.NativeInterface.loadNativeLibrary(Unknown Source)
at com.rti.dds.util.NativeInterface.loadNativeLibraries(Unknown Source)
at com.rti.ndds.config.Logger.(Unknown Source)
at com.ge.hc.lcs.axonenet.logging.LoggerFactory.configureDDSLogger(LoggerFactory.java:179)
at om.ge.hc.lcs.axonenet.logging.LoggerFactory.getLogListenerWorkerThread(LoggerFactory.java:113)
at com.ge.hc.lcs.axone.configservice.ConfigServiceController.initializeLogger(ConfigServiceController.java:285)
at com.ge.hc.lcs.axone.configservice.ConfigServiceController.(ConfigServiceController.java:100)
at com.ge.hc.lcs.axone.configservice.ConfigServiceController.createConfigServiceController(ConfigServiceController.java:
at com.ge.hc.lcs.axone.configservice.ConfigServiceMain.start(ConfigServiceMain.java:263)
at com.ge.hc.lcs.axone.configservice.ConfigServiceMain.main(ConfigServiceMain.java:125)

I have all the DLL in the folder where I am executing this JAR the java version is JDK1.8

Following update to 1.0.0 I still see an error like on 0.8.0 (see #9 for details)

[info] Exception encountered when attempting to run a suite with class name: com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest *** ABORTED ***
[info]   java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdOutputStream.recommendedCOutSize()J
[info]   at com.github.luben.zstd.ZstdOutputStream.recommendedCOutSize(Native Method)
[info]   at com.github.luben.zstd.ZstdOutputStream.<clinit>(ZstdOutputStream.java:29)
[info]   at com.gu.contentapi.porter.integration.ZstdCompression$.compress(Compression.scala:51)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$1.apply$mcV$sp(ZstdCompressionTests.scala:12)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$1.apply(ZstdCompressionTests.scala:10)
[info]   at com.gu.contentapi.porter.integration.flexible.ZstdCompressionTest$$anonfun$1.apply(ZstdCompressionTests.scala:10)
[info]   at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)

Hi,
sorry to bother again, but unfortunately I bumped into an issue where I'm actually not quite sure what is causing this and how to fix it. Even though the issue is reproducible (in production) and always points a specific piece of code I'm currently unable to tell what is causing this behaviour.

In a nutshell what I'm currently doing is that I create a writer with closeFrameOnFlush=true and a reader with continuous=true. Reader and writer might run in parallel threads. The writer-Thread flushes every x-Elements (e.g. every 100 elements) and notifies the reader that he can start reading. Since the zstd-Frame is closed at a flush on the writer side and continuous-Mode is enabled on reader side this should technically just give the reader a complete frame to read and should not cause an Exception.

Implementation wise the reader and writer are created in the following way (quite similar to what I outlined in #52 just with threads)

            private static final int ZSTD_CLEVEL_DEFAULT = 3;
            // create writer
            final BufferedWriter writer = new BufferedWriter(
                    new OutputStreamWriter(new ZstdOutputStream(
                            Files.newOutputStream(compressedFile.toPath()),
                            ZSTD_CLEVEL_DEFAULT, true),
                            StandardCharsets.UTF_8));
            // create reader
            final BufferedReader reader = new BufferedReader(
                    new InputStreamReader(
                            new ZstdInputStream(
                                Files.newInputStream(compressedFile.toPath())
                            ).setContinuous(true),
                            StandardCharsets.UTF_8));

The Exception that currently pops up is the following:

java.io.IOException: Decompression error: Unknown frame descriptor
	at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:111)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)

I currently can't reproduce the issue with testcases (so the tests and code seems to work fine), but the "production" code is producing this exception reliably. Cleaning up any produced files and have them regenerated on-the-fly does not fix the issue.

I can reproduce the issue with 1.3.3-4 and 1.3.4-1

When looking at the zstd-project the issue seems to be related with "Support for Legacy format". Since I can also reproduce the issue with a clean work environment the files generated are not "old" hence I don't see any relation.

Just a side note:
I can open the files with the zstd command-line tools without any error-messages.

Any help on how to troubleshoot this further or what might go wrong is appreciated.

Zstd library cannot be loaded at some of our customers due to a missing dependency. The customers use Windows 7 & 8.1 with 32-bit Java 8.

Stack trace:
java.lang.UnsatisfiedLinkError: C:\Users\gaileri\AppData\Local\Temp\libzstd5953396159493472756.dll: Can't find dependent libraries
at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_101]
at java.lang.ClassLoader.loadLibrary0(Unknown Source) ~[na:1.8.0_101]
at java.lang.ClassLoader.loadLibrary(Unknown Source) ~[na:1.8.0_101]
at java.lang.Runtime.load0(Unknown Source) ~[na:1.8.0_101]
at java.lang.System.load(Unknown Source) ~[na:1.8.0_101]
at com.github.luben.zstd.util.Native.load(Native.java:78) ~[commons3.jar:na]
...

Maybe some MS Visual C runtime is needed for loading the zstd dll? Could it be included in the jar file?

We are currently adding zstd compression support to Spark and we are using ztsd-jni for that. We are facing Spark unit tests failure because the library is compiled with a newer glibc requirement than the amplab machines have. Can you tweak your compilation to support older Linux distros?

See apache/spark#18805 (comment) for more details.

Many builds with natives have support for maven classifiers (used to provide separate jars depending on OS). This reduces download size etc. Would be good to add this as more and more platforms are added.

Hello

I don't find how to include zstd to my Android JNI Project.
Can you please provide a guide how to build it ?

Thank you

Running zstd-jni 1.3.0-1 in the openjdk:8-jre-alpine container on an ubuntu host produces a java.lang.UnsatisfiedLinkError.

java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in you system.
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at com.github.luben.zstd.util.Native.load(Native.java:101)
        at com.github.luben.zstd.ZstdOutputStream.<clinit>(ZstdOutputStream.java:17)

The debian based versions, openjdk:slim and openjdk:<version> both work fine.

From the description of openjdk:alpine:

The main caveat to note is that it does use musl libc instead of glibc and friends, so certain software might run into issues depending on the depth of their libc requirements.

Looks like zstd-jni falls into this category.

Native.java selects /linux/amd64/libzstd-jni.so. I'm not sure what'd you need to do to determine that you should swap in a different module, but I'd be happy to gather more info about the environment from the jvm's perspective if it'd help.

An acceptable resolution to this issue might just be to update the readme to say that alpine is unsupported to save the next person some debugging time. Happy to submit a PR for that too, if you'd like me to.

Hi,

Looks like there's a native memory leak in zstd-jni. The code in question is a straightforward reader in a server, where a GZIPInputStream was simply replaced by a ZstdInputStream, which clearly caused a native memory leak (but not a JVM heap leak). The same code was very stable with GZIPInputStream.

I'll spend some time to reproduce and diagnose later, but wanted to report this in the meantime.

Hi, @luben why is int returned in java side when the signature in c version's compressStream/endStream is size_t?

There are currently no checks in the close() methods of ZstdInputStream/ZstdOutputStream. When attempting to close an already closed stream, the freeDStream/freeCStream native methods are called again, resulting in a segmentation fault and crashing the JVM.

The streams should behave like the default DeflaterOutputStream/InflaterInputStream by maintaining a closed flag and doing nothing if it has been set. It could also be used to prevent potential crashes in the read/write methods, by throwing an exception if the stream has already been closed.

We will need separate build for Android

Thanks for maintaining this immensely useful project!

It would be great, if you could retroactively add tags to this repository the zstd-jni releases on Maven Central are based on.

Hi, first of all, thanks for your nice wrapper.
I notice the implementation in ZstdInputStream would decode only one zstd frame. Would the multiple zstd frames support be added in the near feature?

Hi luben :)
Thank you for your project.

Can you provide a example about compress a media file or any format file ?
if would be better if you can provide usage also!

Thank you

Maven artefacts does not seems to support non 64bits ARM architecture.
ZStandard library does, is it possible to build zstd-jni for ARM 32 bits architectures like ARM7 ? How ?
Thanks

I have recently added Android support. Zstd-jni sources can be imported in Adroid Studio and used from Android apps. I am not familiar with the dependency management on Android and best practices for distributing libraries containing native shared objects. Any suggestions how improve over using zstd-jni as git sub-module will be highly appreciated.

In sbt build file, "com.github.luben.zstd-jni" is set as automatic module name. However, jdk module name cannot contains dash('-') char. Maybe we can use "com.github.luben.zstd_jni".

Can you add useful InputStream class for dict compress/decompress?

Buffers pinned with GetPrimitiveArrayCritical get corrupted in the ZstdInputStream under certain garbage collectors when the tests are run in parallel. G1GC reliebly works without fault.

Another option is to ise GetByteArrayElements for the destination buffer (it solves the problem) but it is detremental of the performance because each get/release is making a new copy.

Needs further investigation.

Function ZSTD_decompress() expects to get exact size of a single compressed frame.
In case of decompressing data stored on heap in byte-array it is not always possible to use long decompress(byte[] dst, byte[] src) binding function. Because byte[] src might not represent single compressed frame (e.g. data was read from socket in temporary buffer which contains several neighbouring frames).

It will be great to have overloaded decompress() method with next signature:

long decompress(byte[] dst, int dstOff, int dstLen, byte[] src, int srcOff, int srcLen)

I've looked everywhere I could to find some form of example to use this with Java

I understand using something like this for compression
byte[] data = ...
byte[] compressed = Zstd.compress(data)

But decompression is baffling me, how are these used in order to decompress?
long originalSize = Zstd.decompress(byte[] dest, byte[] src)
OR
byte[] dest = Zstd.decompress(byte[] src, int originalSize)

Thank you for any help.

Hi, I'm using version 1.2.0, which is the newest version available as of writing this,
And I found no open or closed issue regarding this question.

com/github/luben/zstd/Zstd.java around line 695:

    long size = decompressDirectByteBuffer(dstBuf, //decompress into dstBuf
    dstBuf.position(), //write decompressed data at offset position()
 ***dstBuf.capacity() - dstBuf.position(), //write no more than capacity() - position()
    srcBuf, //read compressed data from srcBuf
    srcBuf.position(), //read starting at offset position()
    srcBuf.limit() - srcBuf.position()); //read no more than limit() - position()

and later in line 704: dstBuf.position(dstBuf.position() + (int)size);
also in corresponding JNI native code, it seems that the dstBuf's limit is not respected.

this will not only cause runtime exception as shown below:

Exception in thread "main" java.lang.IllegalArgumentException
    at java.nio.Buffer.position(Buffer.java:244)
    at com.github.luben.zstd.Zstd.compress(Zstd.java:354)
    ...

but it will also cause data corruption in violation to ByteBuffer's contract.

So my question here: is this a design decision? if so, why?

Unable to work and use