lhns / fs2-compress Goto Github PK

View Code? Open in Web Editor NEW

34.0 2.0 1.0 198 KB

Compression Algorithms for Fs2

License: Apache License 2.0

Scala 100.00%

bzip2 compression fs2 gzip scala zip zstd tar

fs2-compress's Introduction

fs2-compress

Integrations for several compression algorithms with Fs2.

Usage

build.sbt

libraryDependencies += "de.lhns" %% "fs2-compress-gzip" % "2.0.0"
libraryDependencies += "de.lhns" %% "fs2-compress-zip" % "2.0.0"
libraryDependencies += "de.lhns" %% "fs2-compress-zip4j" % "2.0.0"
libraryDependencies += "de.lhns" %% "fs2-compress-tar" % "2.0.0"
libraryDependencies += "de.lhns" %% "fs2-compress-bzip2" % "2.0.0"
libraryDependencies += "de.lhns" %% "fs2-compress-zstd" % "2.0.0"
libraryDependencies += "de.lhns" %% "fs2-compress-brotli" % "2.0.0"

Example

import cats.effect.IO
import de.lhns.fs2.compress.{GzipCompressor, GzipDecompressor}
import fs2.io.compression._
import fs2.io.file.{Files, Path}

implicit val gzipCompressor: GzipCompressor[IO] = GzipCompressor.make()
implicit val gzipDecompressor: GzipDecompressor[IO] = GzipDecompressor.make()

for {
  _ <- Files[IO].readAll(Path("file"))
    .through(GzipCompressor[IO].compress)
    .through(Files[IO].writeAll(Path("file.gz")))
    .compile
    .drain
  _ <- Files[IO].readAll(Path("file.gz"))
    .through(GzipDecompressor[IO].decompress)
    .through(Files[IO].writeAll(Path("file")))
    .compile
    .drain
} yield ()

License

This project uses the Apache 2.0 License. See the file called LICENSE.

fs2-compress's People

Contributors

Stargazers

Watchers

Forkers

mrdziuban

fs2-compress's Issues

Add BrotliCompressor API

The Brotli module only has a decompressor.

zip decompressor is not cancellable

Thanks for this great library, I been using it for several projects.

I just have a small issue that zip module to decompress a file on the fly, it works great but it seems I can't cancel it (just by ctrl+c).

//> using scala "3.4.1"
//> using toolkit typelevel:0.1.25
//> using dep de.lhns::fs2-compress-zip:2.0.0

import cats.effect.{ IO, IOApp }
import cats.syntax.all.*
import org.http4s.*
import org.http4s.client.Client
import org.http4s.implicits.*
import org.http4s.ember.client.EmberClientBuilder

object Main extends IOApp.Simple:
  val downloadUrl = uri"http://ratings.fide.com/download/players_list.zip"
  lazy val request = Request[IO](
    method = Method.GET,
    uri = downloadUrl
  )

  def run =
    EmberClientBuilder
      .default[IO]
      .build
      .use:
        _.stream(request)
          .switchMap(_.body)
          .through(Decompressor.decompress)
          .compile
          .drain

object Decompressor:

  import de.lhns.fs2.compress.*
  import fs2.Pipe
  val defaultChunkSize = 1024 * 4

  def decompress: Pipe[IO, Byte, Byte] =
    _.through(ArchiveSingleFileDecompressor(ZipUnarchiver.make[IO](defaultChunkSize)).decompress)

It works fine with zip4j module for example with these changes:

@@ -1,6 +1,6 @@
 //> using scala "3.4.1"
 //> using toolkit typelevel:0.1.25
-//> using dep de.lhns::fs2-compress-zip:2.0.0
+//> using dep de.lhns::fs2-compress-zip4j:2.0.0
 
 import cats.effect.{ IO, IOApp }
 import cats.syntax.all.*
@@ -34,4 +34,4 @@ object Decompressor:
   val defaultChunkSize = 1024 * 4
 
   def decompress: Pipe[IO, Byte, Byte] =
-    _.through(ArchiveSingleFileDecompressor(ZipUnarchiver.make[IO](defaultChunkSize)).decompress)
+    _.through(ArchiveSingleFileDecompressor(Zip4JUnarchiver.make[IO](defaultChunkSize)).decompress)

Tar and zip archivers read entries fully into memory

The use of chunkAll in TarArchiver#archive and in ZipArchiver#archive causes all the bytes for each entry to be read into memory, which can lead to out-of-memory errors for large entries.

I believe the main limitation/reason for why this is necessary is so tarEntry.setSize can be called with the correct size. I tried using chunks instead of chunkAll along with updating the size with each chunk, but it didn't work properly because tarOutputStream.putArchiveEntry(tarEntry) is executed while the size is still 0, so I ended up with an error when writing to tarOutputStream.

Can you think of any other ways to work around this? Here's my attempt in case it's helpful: main...mrdziuban:tar-zip-read-into-memory

lhns / fs2-compress Goto Github PK

fs2-compress's Introduction

fs2-compress

Usage

build.sbt

Example

License

fs2-compress's People

Contributors

Stargazers

Watchers

Forkers

fs2-compress's Issues

Add BrotliCompressor API

zip decompressor is not cancellable

Tar and zip archivers read entries fully into memory

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent