The go-tfdata is a Go library helping to work with tar/tgz archives and files in TFRecord and tf.Example formats, including converting TAR files to TFRecord files. It provides interfaces and their default implementations on each intermediate step between tar and TFRecord format. Additionally, it includes easy to use utilities to convert and augment data in intermediate steps.

The library is designed with simplicity, speed and extensibility in mind. The goal is not to support multiple, complicated communication protocols for remote data handling or complex algorithms implementations, it's rather giving ability for users to extend it in any possible way.

Full documentation

go-tfdata provides default implementations for manipulating tar and TFRecord files. It includes:

• FromTar(io.Reader) - read Samples from io.Reader in Tar format
• TransformSamples(transformations) - transform each Sample according to provided transformations (either predeclared in go-tfdata or provided by a user)
• SampleToTFExample(reader, [typesMapping] - default transformation from Sample to TFExample format. If typesMapping provided, maps sample to TFExample accordingly to types.
• TransformTFExamples(transformations) - transform each TFExample according to provided transformations
• ToTFRecord(io.Writer) - write serialized TFExamples to io.Writer in TFRecord file format
• FilterEmptyExamples(reader), FilterEmptySamples(reader) - filter reader from empty TFExamples / Samples

go-tfdata provides basic Samples and TFExamples transformations and selections, which can be easily applied to the data

• ByKey(key) - selects entry which key equals to key
• ByKeyValue(key, value) - selects entry which key equals key and value equals value
• ByPrefix(name), BySuffix(name), BySubstring(name) - selects entries which key is prefix, suffix or substring of name
• BySampleF(f), ByExampleF(f) - selects entries which keys are in subset returned by function f
• TBA...

• RenameTransformation(dest string, src []string) - renames src fields into dest field
• SampleF(f func(core.Sample) core.Sample) - transforms Sample based on specified function f
• TFExampleF(f func(*core.TFExample) *core.TFExample) - transforms TFExample based on specified function f

pipeline := NewPipeline().FromTar(inFile).SampleToTFExample().ToTFRecord(outFile)
pipeline.Do()

pipeline := NewPipeline().FromTar(inFile)
pipeline.SampleToTFExample(core.TypesMap{
    "cls": core.FeatureType.INT64,
    "jpeg": core.FeatureType.BYTES,
})
pipeline.ToTFRecord(outFile).Do()

type Logger struct {
    reader TFExampleReader
    cnt    int
}

func (l *Logger) Read() (*TFExample, bool) {
    cnt++
    if cnt % 10 == 0 { log.Infof("read %d examples", cnt) }
    return l.reader.Read()
}

pipeline := NewPipeline().WithTFExampleStage(func(reader TFExampleReader) TFExampleReader {
    return &Logger{reader: reader}
}).FromTar(inFile).SampleToTFExample().ToTFRecord(outFile)

pipeline.Do()

pipeline := NewPipeline().TransformSamples(
    transform.ExampleSelections(selection.ByKey("image"))
).FromTarGz(inFile).SampleToTFExample().ToTFRecord(outFile)
pipeline.Do()

type FAASClient struct { 
    reader SamplesReader
    ...
}

func (c *FAASClient) Read() (Sample, bool) {
    sample, ok := c.reader.Read()
    if !ok { return nil, false }
    id := c.Send(sample)
    c.Receive(id, &sample)
    return sample, true
}

pipeline := NewPipeline().WithSamplesStage(func(reader SamplesReader) SamplesReader {
    return FAASClient{reader: reader} 
}).FromTar(inFile).SampleToTFExample().ToTFRecord(outFile)
pipeline.Do()

To see fully working implementation of some examples see go-tfdata/tests package.

pipeline is abstraction for TAR-to-TFRecord process. pipeline is made of stages. Default pipeline implementation has 5 stages:

With this approach, evaluation can be (but doesn't have to be) lazy, meaning that each of the stages process the data when final consumer - TFRecordStage - decides to consume a TFExample

Pipeline is high-level abstraction and can be replaced, extended or limited. For each stage, default implementation can be used (or none at all for optional stages), or custom implementation can be provided by a user via pipeline.With[STAGE] method

There exists two types of readers interfaces - SamplesReader, TFExamplesReader. Their methods:

TFExampleReader interface {
    Read() (ex *TFExample, ok bool)
}

SampleReader interface {
    Read() (sample Sample, ok bool)
}

It's up to Reader implementation how it behaves on creation or Read calls. It might be executing a transformation only when Read method is called (lazy) or Reader can drain internal Reader and do transformations immediately. It can as well prefetch part of internal Reader data. Each of approaches has it's advantages and should be considered per use-case.

TFExample format is based on TensorFlow example.proto files. Thanks to Go Protobuf API v2, a structure of TFExamples in TFRecord files is determined automatically. Learn more about TFExample.