Git Product home page Git Product logo

Comments (11)

martint avatar martint commented on July 22, 2024 1

The formula matches the one in the reference implementation, so this must be a bug somewhere else. It would help a lot to have a dataset that reproduces the problem.

#define ZSTD_COMPRESSBOUND(srcSize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /* margin, from 64 to 0 */ : 0))

from aircompressor.

martint avatar martint commented on July 22, 2024 1

I've been able to reproduce this locally. I'm investigating.

from aircompressor.

findepi avatar findepi commented on July 22, 2024

Thanks @miniway for the report!
Can you perhaps try to reproduce this in a synthetic way, eg using tpch data as the input?
That would also help verify the problem still exists in the latest release.

from aircompressor.

miniway avatar miniway commented on July 22, 2024

Unfortunately I could not easily reproduce the issue, but my gut feeling is it doesn't seem to happen at 0.15 but it happens rarely at 0.18

from aircompressor.

dongjoon-hyun avatar dongjoon-hyun commented on July 22, 2024

Hi, All. Apache ORC community also is blocked by this issue. This causes UT failures consistently.
Apache ORC community has been using 0.16 without this issue. New 0.17 and 0.18 seems to have this issue.

The relevant change of airlift is this commit.

Caused by: java.lang.IllegalStateException: Overflow detected
	at io.airlift.compress.zstd.Util.checkState(Util.java:59)
	at io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
	at io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
	at io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
	at io.airlift.compress.zstd.ZstdFrameCompressor.encodeLiterals(ZstdFrameCompressor.java:333)
	at io.airlift.compress.zstd.ZstdFrameCompressor.compressBlock(ZstdFrameCompressor.java:224)
	at io.airlift.compress.zstd.ZstdFrameCompressor.compressFrame(ZstdFrameCompressor.java:172)
	at io.airlift.compress.zstd.ZstdFrameCompressor.compress(ZstdFrameCompressor.java:145)
	at io.airlift.compress.zstd.ZstdCompressor.compress(ZstdCompressor.java:45)
	at org.apache.orc.impl.AircompressorCodec.compress(AircompressorCodec.java:68)
	at org.apache.orc.impl.OutStream.spill(OutStream.java:290)
	at org.apache.orc.impl.OutStream.write(OutStream.java:263)
	at org.apache.orc.impl.SerializationUtils.writeLongBE(SerializationUtils.java:840)
	at org.apache.orc.impl.SerializationUtils.unrolledBitPackBytes(SerializationUtils.java:674)
	at org.apache.orc.impl.SerializationUtils.unrolledBitPack32(SerializationUtils.java:644)
	at org.apache.orc.impl.SerializationUtils.writeInts(SerializationUtils.java:497)
	at org.apache.orc.impl.RunLengthIntegerWriterV2.writeDirectValues(RunLengthIntegerWriterV2.java:370)
	at org.apache.orc.impl.RunLengthIntegerWriterV2.writeValues(RunLengthIntegerWriterV2.java:174)
	at org.apache.orc.impl.RunLengthIntegerWriterV2.write(RunLengthIntegerWriterV2.java:804)
	at org.apache.orc.impl.writer.IntegerTreeWriter.writeBatch(IntegerTreeWriter.java:87)
	at org.apache.orc.impl.writer.StructTreeWriter.writeRootBatch(StructTreeWriter.java:56)
	at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:704)
	... 60 more

from aircompressor.

martint avatar martint commented on July 22, 2024

Thanks for narrowing down the issue to that commit. I’ll take a look soon.

from aircompressor.

martint avatar martint commented on July 22, 2024

Also, do you have a data set that reproduces it consistently? I’d like to investigate whether there’s a bug elsewhere that is being surfaced by that change.

from aircompressor.

dongjoon-hyun avatar dongjoon-hyun commented on July 22, 2024

Update: Apache ORC upgrades to 0.19 finally. Thanks!

from aircompressor.

melin avatar melin commented on July 22, 2024

aircompression 0.20， The same problem occurred

image

from aircompressor.

dongjoon-hyun avatar dongjoon-hyun commented on July 22, 2024

Thank you for reporting, @melin . If you don't mind, could you file an Apache ORC JIRA issue at https://issues.apache.org/jira/projects/ORC with the above information, please?

from aircompressor.

dongjoon-hyun avatar dongjoon-hyun commented on July 22, 2024

Thank you, @martint !

from aircompressor.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.