vtrushin / json-to-ast Goto Github PK

I see that the source is already using JS modules. Could you publish them to npm too?

JSON5 adds many additional features to JSON. Can this package support the new syntax added?

The JSON5 project has a reference parser that has code very similar (but not identical) to the tokenize.js and parse.js files in this project.

I wonder, because it seems a bit like it was probably used for testing the parser? If it's just a repeat of
value, maybe better to remove it?

The property x" is represented in the AST as x\":

{ "x\"": 0 }

becomes:

{
  "type": "object",
  "children": [
    {
      "type": "property",
      "key": {
        "type": "identifier",
        "value": "x\\\"",
      },
  ...
}

Try this to verify:

> let sample = JSON.stringify({ 'x"': 0 })
> JSON.parse(sample)
{ 'x"': 0 }
> parse(sample).children[0].key.value
'x\\"'

Hi @vtrushin

I have a fork of json-to-ast which I've extended to firstly support comments and secondly to allow the JSON to be modified and written back out again while preserving comments and all whitespace. This is really useful for us (we are the Qooxdoo framework - https://github.com/qooxdoo) because it allows .json configuration files to be tweaked on the fly by code (eg we're developing a command line tool that will work similar to npm install module).

Is this something that you would be interested in receiving as a pull request? You can see the work to date at my personal repo here: https://github.com/johnspackman/json-to-ast/tree/editable-json

The changes to date are principally:

• Tokeniser is modified to collect whitespace and comments; instead of returning a tokenList array, an instance of a class Tokenizer is used which can selectively skip the whitespace and comments
• Parser is modified to use Tokenizer instance, add startToken and endToken properties that record where the AST node is in the token stream, and add leadingComments and trailingComments
• Add src/stringify.js that adds a reprint method to output a POJO object, using the whitespace and comments found in the original AST. Also included is a prettyPrint method that pretty prints from an AST tree
• Gulp process was modified to support source maps and as part of this the source files are compiled separately instead of into one parser.js; the new main.js retains backwards compatibility with the existing json-to-ast API.

At the moment, the code is functional and passing units tests and we're about to start testing in anger as we add it to our tools; there's no documentation (not that this really requires much), and this should be 100% backwards compatible with your existing release.

If you would like to incorporate this I would be willing to make changes to meet your requirements, eg coding standards, documentation, etc; alternatively my thoughts are that I would either incorporate it into our tool as is, or possibly release it as another npm module under another name (with full credits to you of course)

Regards
John

When an unexpected symbol error occurs at an emoji character, the resulting error message doesn't display the correct character.

try {
  jsonToAst('💩');
} catch (error) {
  console.log(error.rawMessage);
  // > "Unexpected symbol <�> at 1:1"
}

To be clear, I was expecting to see the following error message: Unexpected symbol <💩> at 1:1.

From what I can gather, this happens because String.prototype.charAt (which is being used to get the offending character) doesn't support these special characters.

Example here: https://runkit.com/bsonntag/5c1b96384702bf0012a73396

2.0.5 release changed target node version to 6, so resulting source code is not transforming to es5 anymore. However, json-to-ast is using by projects that support node.js below 6, in particular css-tree which test is failing now. It's a bad practice to change env support in a patch version, so instead of 2.0.5 there should be 3.0.0.

Now root node is JSON's top level node, should to be a document

case objectStates.PROPERTY: {
    if (token.type === tokenTypes.RIGHT_BRACE) {
        if (settings.verbose) {
            object.loc = location(
                startToken.loc.start.line,
                startToken.loc.start.column,
                startToken.loc.start.offset,
                token.loc.end.line,
                token.loc.end.column,
                token.loc.end.offset,
                settings.source
            );
            return {
                value: object,
                index: index + 1
            };
        }

Should be

case objectStates.PROPERTY: {
    if (token.type === tokenTypes.RIGHT_BRACE) {
        if (settings.verbose) {
            object.loc = location(
                startToken.loc.start.line,
                startToken.loc.start.column,
                startToken.loc.start.offset,
                token.loc.end.line,
                token.loc.end.column,
                token.loc.end.offset,
                settings.source
            );
        }
        return {
            value: object,
            index: index + 1
        };

Would it be possible to add support for comments in JSON?

Hi! @vtrushin .
I've used it in my project, but it's a little unclear how to convert it back to json after the ast changes.

1. Not make sense, absolutely. This inconsistency around is not good.

{ type: 'property',
  key: 
   { type: 'key',
     value: 'foo',
     position: { start: [Object], end: [Object], human: '2:3 - 2:6 [4:7]' } },
  value: 
   { type: 'string',
     value: 'sa',
     position: { start: [Object], end: [Object], human: '2:7 - 2:11 [8:12]' } } }
{ type: 'property',
  key: { position: { start: [Object], end: [Object], human: '3:3 - 3:9 [16:22]' } },
  value: 
   { type: 'true',
     value: null,
     position: { start: [Object], end: [Object], human: '3:11 - 3:15 [24:28]' } } }
{ type: 'property',
  key: { position: { start: [Object], end: [Object], human: '4:3 - 4:9 [32:38]' } },
  value: 
   { type: 'string',
     value: 'hyge',
     position: { start: [Object], end: [Object], human: '4:11 - 4:17 [40:46]' } } }

How we'll know what's the name of the key that contains true value? How we'll know what's the name of the key that contains string with value hyge (the third/last node)? Btw, in that second node here also have something wrong. Value of type: 'true' is null (same applies for type: false)? Not make sense.

It's correct/valid only for type: null

{ type: 'null',
  value: null,
  position: 
   { start: { line: 22, column: 10, char: 263 },
     end: { line: 22, column: 14, char: 267 },
     human: '22:10 - 22:14 [263:267]' } }

* maybe make sense to rename type of the null, true and false to literal so

{
  type: 'literal'
  value: true
}
{
  type: 'literal'
  value: null
}
{
  type: 'literal'
  value: false
}

2. Also, it would be better to rename position[start/end].char to offset.

3. Third inconsistency is that .properties and .items thing. It's not comfortable for walkers and etc - you always should check the .type of the coming node.

You may also be interested in https://github.com/finnp/json-lexer. Try to use it?
And we should implement finnp/json-lexer#3 (positions).