How to use snappy? about parquet-go HOT 8 CLOSED

Hi @seaguest,

Looking at the snippet you shared it doesn't look like you are using segmentio/parquet-go.

Using the following :

package main

import (
        "os"

        "log"

        segmentparquet "github.com/segmentio/parquet-go"
)

func write() {
        f, err := os.OpenFile("outputs.parquet", os.O_APPEND|os.O_WRONLY|os.O_CREATE|os.O_TRUNC, os.ModePerm)
        if err != nil {
                log.Println(err)
                return
        }
        defer f.Close()

        writer := segmentparquet.NewWriter(f)

        type record struct {
                Format   string `parquet:"format,snappy"`
                DataType int32  `parquet:"data_type,snappy"`
                Country  string `parquet:"country,snappy"`
        }

        num := 1000
        for i := 0; i < num; i++ {
                stu := record{
                        Format:   "Test",
                        DataType: 1,
                        Country:  "IN",
                }

                writer.Write(stu) // here argument can only be []byte
        }
        // Closing the writer is necessary to flush buffers and write the file footer.
        if err := writer.Close(); err != nil {
                log.Println(err)
        }

}

func main() {
        write()
}

ends up creating a file of 1060 bytes so about 1KB.

from parquet-go.

I tried to use

	type record struct {
		Format   string `parquet:"format,snappy"`
		DataType int32  `parquet:"data_type,snappy"`
		Country  string `parquet:"country,snappy"`
	}

but the output file is still very large, about 34M，while with other 2 libraries, they are only 1KB

from parquet-go.

@Pryz

Indeed it works now.
But I am curious why should we put snappy annotation for each field, usually we won't have different compression type for different fields in one struct.

I saw other library has an option like this

	pw.CompressionType = parquet2.CompressionCodec_GZIP

why this library doesn't have such an option?

from parquet-go.

usually we won't have different compression type for different fields in one struct.

We're planning to use different compression types for different fields in one struct (tracing data), which is why we thought that choice was a good fit.

from parquet-go.

I'm going to close this - thanks for the issue report and glad you got it working!

from parquet-go.

are you planning to provide an option of compression type for all fields in the future?
If we have hundreds of fields, it would be a disaster to add "snappy" for each, and that is meaningless in case we need only one compression type.

thanks for your quick reply~

from parquet-go.

+1 on this. It would be great to provide an alternative way to pass compression config while initialising an writer.

from parquet-go.

Created #124 as a follow up

from parquet-go.