exponent-io / jsonpath Goto Github PK

Extends the Go runtime's json.Decoder enabling navigation of a stream of json tokens.

Home Page: https://godoc.org/github.com/exponent-io/jsonpath

License: MIT License

Go 100.00%

jsonpath's Introduction

jsonpath

This package extends the json.Decoder to support navigating a stream of JSON tokens. You should be able to use this extended Decoder places where a json.Decoder would have been used.

This Decoder has the following enhancements...

The Scan method supports scanning a JSON stream while extracting particular values along the way using PathActions.
The SeekTo method supports seeking forward in a JSON token stream to a particular path.
The Path method returns the path of the most recently parsed token.
The Token method has been modified to distinguish between strings that are object keys and strings that are values. Object key strings are returned as the KeyString type rather than a native string.

Installation

go get -u github.com/exponent-io/jsonpath

Example Usage

SeekTo

import "github.com/exponent-io/jsonpath"

var j = []byte(`[
  {"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}},
  {"Space": "RGB",   "Point": {"R": 98, "G": 218, "B": 255}}
]`)

w := json.NewDecoder(bytes.NewReader(j))
var v interface{}

w.SeekTo(1, "Point", "G")
w.Decode(&v) // v is 218

Scan with PathActions

var j = []byte(`{"colors":[
  {"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10, "A": 58}},
  {"Space": "RGB",   "Point": {"R": 98, "G": 218, "B": 255, "A": 231}}
]}`)

var actions PathActions

// Extract the value at Point.A
actions.Add(func(d *Decoder) error {
  var alpha int
  err := d.Decode(&alpha)
  fmt.Printf("Alpha: %v\n", alpha)
  return err
}, "Point", "A")

w := NewDecoder(bytes.NewReader(j))
w.SeekTo("colors", 0)

var ok = true
var err error
for ok {
  ok, err = w.Scan(&actions)
  if err != nil && err != io.EOF {
    panic(err)
  }
}

jsonpath's People

Contributors

Stargazers

Watchers

Forkers

nagesh4193

jsonpath's Issues

How to seek forward to the end of the outer block

In this example:

import "github.com/exponent-io/jsonpath"

var j = []byte(`[
  {"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}},
  {"Space": "RGB",   "Point": {"R": 98, "G": 218, "B": 255}}
]`)

w := json.NewDecoder(bytes.NewReader(j))
var v interface{}

w.SeekTo(1, "Point", "G")
w.Decode(&v) // v is 218

How would one SeekTo the closing "]" to ensure all the content was read? Or is that not possible?

Note: in this case it isn't really feasible to look for an EOF. The actual JSON payload looks more like this and the goal is to seek to the second array after processing the first one:

[
  {"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}},
  {"Space": "RGB",   "Point": {"R": 98, "G": 218, "B": 255}}
]
[
  {"Space": "YCbCr", "Point": {"Y": 255, "Cb": 0, "Cr": -10}},
  {"Space": "RGB",   "Point": {"R": 98, "G": 218, "B": 255}}
]

-- Greg

Finding start of items array

I need to upload into database json files are quite large, like 3+ Gb, I would simply use Decoder to stream them but they have a weird structure { "someKey": [ .. item1 .. item2 .. item3 .. ]}.

I am trying to do something like this by seeking for a start of array and then storing items into file in a streaming mode, is this possible with the library?

func ExtractJSONArrayIter(filePath, arrayKey string) error {
    fmt.Printf("Extracting items from array key %s\n", arrayKey)

    // Open the input file for streaming
    file, err := os.Open(filePath)
    if err != nil {
        return fmt.Errorf("opening input file: %w", err)
    }
    defer file.Close()

    // Create a new JSONPath Decoder
    decoder := jsonpath.NewDecoder(bufio.NewReader(file))

    // Open the output file for writing NDJSON
    outputFile, err := os.Create("output.ndjson")
    if err != nil {
        return fmt.Errorf("opening output file: %w", err)
    }
    defer outputFile.Close()
    writer := bufio.NewWriter(outputFile)

    // Define PathActions to extract each item in the array
    var actions jsonpath.PathActions
    actions.Add(func(d *jsonpath.Decoder) error {
        // Decode the current item
        var item interface{}
        if err := d.Decode(&item); err != nil {
            return err
        }
        
        // Marshal the item to JSON and write to the output file
        if itemBytes, err := json.Marshal(item); err == nil {
            if _, err := writer.Write(itemBytes); err != nil {
                return err
            }
            if _, err := writer.WriteString("\n"); err != nil {
                return err
            }
        }
        return nil
    }, arrayKey)

    // Seek to the start of the specified array
    decoder.SeekTo(arrayKey)

    // Scan the JSON stream and process each item
    var ok = true
    for ok {
        ok, err = decoder.Scan(&actions)
        if err != nil && err != io.EOF {
            return fmt.Errorf("error during JSON scan: %w", err)
        }
    }

    // Flush the writer to ensure all data is written to the file
    if err := writer.Flush(); err != nil {
        return fmt.Errorf("error flushing output file: %w", err)
    }

    fmt.Println("Extraction completed")
    return nil
}

exponent-io / jsonpath Goto Github PK

jsonpath's Introduction

jsonpath

Installation

Example Usage

SeekTo

Scan with PathActions

jsonpath's People

Contributors

Stargazers

Watchers

Forkers

jsonpath's Issues

How to seek forward to the end of the outer block

Finding start of items array

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent