Git Product home page Git Product logo

pjflate's Introduction

Pure Java Flate

Java standard library's java.util.zip.Inflater is a wrapper of zlib, and it uses JNI. This is one of those attempts to implement the deflate format's inflater in pure Java.

The code is mostly a direct translation of Go's compress/flate. Huge thanks to the Go authors and contributors for the readable and annotated implementation.

This is not an official Google product. This is the author's hobby project. Not for production use.

Motivations

  • This is not an attempt to make it fast. In fact, it's much slower than the Java standard Inflater. This is more like to avoid the GCLocker impact.

    JNI code in general is not GC aware, and in order to avoid arrays to be moved by JVM GC, it needs to pin primitive arrays. While pinning, JVM GC is paused. After this pinning is done, JVM can trigger a GC (GCLocker initiated GC). Because no GC is done while this pinning, depending on the heuristics JVM can trigger a full GC. For certain applications, it's more favorable to keep concurrent GCs running to prevent a full GC.

  • There are some pure Java deflate implementations out there, but the author wanted to have a ByteBuffer friendly interface. It's easy to make an adapter though (see the test).

    Starting from Java SE 11, the standard Inflater can take a ByteBuffer. In Open JDK implementation, it checks if it's a direct buffer and if it is, it doesn't lock GC. If GCLocker is a concern and Open JDK 11 or later is used, it's better to use a direct buffer with the standard Inflater. It gives you a better speed than pure Java implementations and you can avoid the GCLocker issue.

  • It seems it's a good size for a weekend hobby project.

Build & Test

Use Bazel. Run download_testdata.sh before running the test. It downloads the test data.

Speed comparison

This is not a serious benchmark, but it seems that it's 4x slower than the standard library Inflater (see the test).

For 100MiB file (enwik8):

java.util.zip.InflaterStream: 361ms
java.util.zip.Inflater: 326ms
Inflater(HeapBuffer): 1201ms
Inflater(DirectBuffer): 1097ms

For 1GiB file (enwik9):

java.util.zip.InflaterStream: 2953ms
java.util.zip.Inflater: 2889ms
Inflater(HeapBuffer): 9772ms
Inflater(DirectBuffer): 9783ms

The first implementation was actually using ByteBuffer#array, and it was 3x slower than the standard library. The extra slowness came from the DirectBuffer support.

pjflate's People

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.