tsouza / yajs Goto Github PK

View Code? Open in Web Editor NEW

11.0 2.0 4.0 120 KB

Yet Another JSON Streaming Tool

License: MIT License

JavaScript 1.08% TypeScript 97.51% ANTLR 1.41%

json streaming javascript

yajs's Introduction

YAJS: Yet Another JSON Streaming Tool

YAJS is a tool for filtering a portion of json files.

Motivation

The reason I built this tool is that I could not find a proper json stream processor with the features I needed without sacrificing speed and memory.

There is a also a benchmark of this tool comparing with oboe.js and JSONStream. See benchmark.

Documentation

Head over to Wiki for more information on how to use it.

Example

Pipe a text stream of json into YAJS and select 'author' property:

const yajs = require('yajson-stream');
const { createReadStream } = require('fs');

createReadStream('./package.json').
    pipe(yajs('$.author')).
    on('data', data => {
        console.log(data.path); // outputs [ 'author' ]
        console.log(data.value); // outputs 'Thiago Souza <[email protected]>'
    });

Command line tool

Call it from a shell:

$ npm install -g yajson-stream
$ cat package.json | yajs '$.author'
"Thiago Souza <[email protected]>"

YAJS Selector Syntax

YAJS selector syntax is jsonpath-like, yet it's not jsonpath.

YAJS Selector	Description
`$`	The root object/element
`*`	Wildcard matching all objects/elements regardless
`.`	Child member operator
`..`	Recursive descendant operator
`..[<path filter>]<key>`	Recursive descendant operator if path filter evaluates to true (see example below)
`<key>{keys filter}`	Will emit only if keys filter evaluates to true. Only supported in the end of the expression (see example below)

Example of `..[<filter keys>]<key>`

Given the following json:

{
    "array": [
        {
            "key1": {
                "child": "value1"
            }
        },
        {
            "key2": {
                "child": "value2"
            }
        }
    ]
}

Select only the second child entry with:

$ cat test.json | yajs '$..[!key1]child'
"value2"

Example of `<key>{<keys filter>}`

Given the following json:

[
    {
        "object1": {
            "key1": "value1"
        }
    },
    {
        "object1": {
            "key2": "value1"
        }
    }
]

Will emit only the first object1:

$ cat test.json | yajs '$.object1{key1}'
{"key1":"value1"}

Bugs and Feedback

For bugs, questions and discussions please use the Github Issues.

Acknowledgements

Thanks to wanglingsong for his awesome work with JsonSurfer which inspired me to create YAJS.
Thanks to creationix for his crazy fast sax-based json parser.

LICENSE

Code and documentation released under The MIT License (MIT).

yajs's People

Contributors

Stargazers

Watchers

Forkers

juancaicedo suresh44t phaedrusthegreek touv

yajs's Issues

ES5 compatibility is broken

After yarn add yajson-stream in our non-typescript (babel) project, we got the following error when trying to use:

module.js:540
    throw err;
    ^

Error: Cannot find module 'yajson-stream'
    at Function.Module._resolveFilename (module.js:538:15)
    at Function.Module._load (module.js:468:25)
    at Module.require (module.js:587:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (.../aggregator/src/data/parse.js:3:14)
    at Module._compile (module.js:643:30)
    at loader (.../node_modules/babel-register/lib/node.js:144:5)
    at Object.require.extensions.(anonymous function) [as .js] (.../node_modules/babel-register/lib/node.js:154:7)
    at Module.load (module.js:556:32)
    at tryModuleLoad (module.js:499:12)

I think this is because main.js is not getting included in the dist/ folder. When I change main.js to main.ts, it is detected by gulp and properly packaged -- I think this is the fix, but I'm not too sure.

infinity loop

with single object, no problem:

echo '{"a": 1}' | yajs '$'

with single string, no problem:

echo '"a"' | yajs '$'

with single number, just one error

echo '1' | yajs '$'

with syntax error, an infinity of errors is triggered:

echo '1..' | yajs '$'
echo '#' | yajs '$'
echo '{"a":a}' | yajs '$'

ES6 bundle being used in node?

Hi there! I'm playing around with yajs and had a problem loading it in node. The main function doesn't seem to be available here

const yajs = require('yajson-stream')
yajs('$') // TypeError: yajs is not a function

I think this might be because the bundle being used by node is bundled for use by ES6 imports

const yajs = require('yajson-stream').default
yajs('$') // works fine

I put together this glitch to show what I'm seeing

A solution I thought of while reading this article would be to set "main" in package.json to point to a node bundle and "module" to point to an ES6 module.

Alternately, the documentation could just be updated to use require('yajson-stream').default.

Let me know if you want any help with this 😀

Multiple key filters

Feel free to close this if it's no a feature you want to support, no hard feelings 😀

Right now I don't think it's possible to apply multiple key filters. I put together this example in which I try to apply two filters, but only the first gets applied.

yajs('$.array{x}{special}') // does not throw error, but only applies {}

The API that oboe supports is like this. I think that shorthand makes a lot of sense

oboe(source)
  .node('$.array.{x, special}')

Let me know if I can provide more info or you want help implementing this!