Many thanks for providing these excellent compressors as open source software! I do not believe this is a bug report, but an inquiry on the characteristics of the compression algorithm and how to interpret the compression ratio.

I have made a very simple experiment to better understand the characteristics of fpzip's compression algorithm:

// Helper function for losslessly compressing a 1D array of doubles and returning the compression ratio
func compress(data: [Double], toFPZ fpz: UnsafeMutablePointer<FPZ>?) -> (Double, Int) {
    let nx: Int32 = Int32(data.count)
    let size = Int32(MemoryLayout<Double>.size)

    fpz!.pointee.type = FPZIP_TYPE_DOUBLE
    fpz!.pointee.prec = 64 // Int32(CHAR_BIT * size), this is full 64 bits of precision, which is lossless
    fpz!.pointee.nx = nx
    fpz!.pointee.ny = 1
    fpz!.pointee.nz = 1
    fpz!.pointee.nf = 1

    // write header
    fpzip_write_header(fpz)

    // perform actual compression
    let outbytes = fpzip_write(fpz, data)

    fpzip_write_close(fpz)

    let ratio = Double(nx) * Double(size) / Double(outbytes)
    return (ratio, outbytes)
}

var rng = ThreefryRandomNumberGenerator(seed: [42]) // This is provided by Swift for Tensorflow
let randomDistribution = UniformFloatingPointDistribution(lowerBound: 0.0, upperBound: Double(count)) // Also from Swift for Tensorflow
let tensorflowRandomValues: [Double] = (0..<Int(count * 10)).map { _ in randomDistribution.next(using: &rng) }
let tensorflowCompressionRatio = compressionRatio(data: tensorflowRandomValues)
print("uniform compression ratio:", tensorflowCompressionRatio)

// I expect the compression ratio to be better for a non-uniform distribution
let normalDistributionGenerator = NormalDistribution(mean: 0.0, standardDeviation: 1.0)
let normalValues: [Double] = (0..<count).map { _ in normalDistributionGenerator.next(using: &rng) }
let normalCompressionRatio = compressionRatio(data: normalValues)
print("normal compression ratio:", normalCompressionRatio)

Please excuse the fact that the code is written in Swift. I wrote a Swift->C interface to call fpzip from Swift, but the same should be readily achievable directly in C. This code generates random data from a uniform distribution and from a normal distribution, compresses each independently as 1-dimensional arrays, and then reports the compression ratios. It produces this output:

uniform compression ratio: 1.1626494222504358
normal compression ratio: 1.0636699538500198

Two questions arise:

• Why is the compression ratio on the uniformly distributed random data greater than 1.0? I would expect it to be exactly 1.0 given that it's random data, unless perhaps the data aren't properly random. I tried generating the random data using various different methods: Swift's built-in Double.random, TensorFlow's ThreefryRandomNumberGenerator, as well as downloading truly random values produced by quantum physical processes. These all gave me a compression ratio of ~1.1625541314267445. I would hypothesize perhaps fpzip's minimum compression ratio on floats is 1.16 no matter how incompressible the data are due to some improvement over the floating point representation, but that doesn't hold up against the compression ratio of the normally distributed data:
• Why is the compression ratio on the normally distributed random data worse than on the uniformly distributed data? I would expect the opposite. The normally distributed points are more likely to be clumped together around 0. Indeed, a differential entropy calculation of these distributions shows that the uniform distribution has a higher entropy:

#!/usr/bin/env python3

from scipy import stats

print(stats.uniform(loc=0.0, scale=100000).entropy()) # uniform distribution from 0 to 100,000
print(stats.norm(loc=0.0).entropy()) # std-deviation 1.0 normal distribution

Producing output:

11.512925464970229
1.4189385332046727

As expected, the uniform distribution is higher entropy than the normal distribution. Similar results are obtained when calculating discrete entropies on discretized samples from uniform and normal distributions.

I realize attempting to compress random data is a fool's errand, this is merely an experiment to aid my understanding of how the compression works.

Are there any plans to support half precision floats, using the __fp16 type?

https://en.wikipedia.org/wiki/Half-precision_floating-point_format

Dear writers,

I have a question on the algorythm . what kind of lossless techique (ie entropy, dictionary type) is fzip?

Thanks in advance!
Best regards,
EP

I would like to refactor the cmake scripts to make this project more modern in general and more suitable for Conan packaging. Will you accept a PR?

fpzip is not supported by Python3.11

Hi Dr. Lindstrom,

I was working on integrating a WASM fpzip decoder into a web viewer (Neuroglancer). However, the maintainer felt it would be best if fpzip performed bounds checking on arrays for the functions. It can cause buffer overflow issues if the number of bytes is not provided.

• fpzip_read_from_buffer(const void* buffer, const size_t num_bytes)
• fpzip_read(FPZ* fpz, void* data, const size_t num_bytes)

There may be others, but those are the big ones. It seems like this would be backwards incompatible without adding new functions, so perhaps fpzip_read_from_buffer2 and fpzip_read2 would be better?

Thanks so much for your work. Let me know if you need any updates to https://github.com/seung-lab/fpzip . I could potentially help a PR for this as well.

Will

I am using BSD Make to build fpzip. It does not define $^ macro which is
not in POSIX [1].

libpzip builds without error but it does not include any object files
because $^ expands to an empty string.

ar rc ../lib/libfpzip.a

It gives me very hard to trace errors if I link with the libray.

I think Makefiles (https://github.com/LLNL/fpzip/blob/master/src/Makefile#L14) can be portable if one does

$(LIBDIR)/libfpzip.a: $(OBJECTS)
       rm -f $@
       ar rc $@ $(OBJECTS)

instead of

$(LIBDIR)/libfpzip.a: $(OBJECTS)
       rm -f $@
       ar rc $@ $^

[1] https://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html

I don't see any mention of endianness in the docs or fpzip -h, and I'm trying to understand if it supports either endianness and if so how to toggle it. I tried it on both little-endian and big-endian versions of my data and got a much better compression ratio for big-endian data, so perhaps it only supports big-endian? Curious to understand better.