Git Product home page Git Product logo

node-jsonc-parser's Introduction

jsonc-parser

Scanner and parser for JSON with comments.

npm Package NPM Downloads Build Status License: MIT

Why?

JSONC is JSON with JavaScript style comments. This node module provides a scanner and fault tolerant parser that can process JSONC but is also useful for standard JSON.

  • the scanner tokenizes the input string into tokens and token offsets
  • the visit function implements a 'SAX' style parser with callbacks for the encountered properties and values.
  • the parseTree function computes a hierarchical DOM with offsets representing the encountered properties and values.
  • the parse function evaluates the JavaScript object represented by JSON string in a fault tolerant fashion.
  • the getLocation API returns a location object that describes the property or value located at a given offset in a JSON document.
  • the findNodeAtLocation API finds the node at a given location path in a JSON DOM.
  • the format API computes edits to format a JSON document.
  • the modify API computes edits to insert, remove or replace a property or value in a JSON document.
  • the applyEdits API applies edits to a document.

Installation

npm install --save jsonc-parser

API

Scanner:

/**
 * Creates a JSON scanner on the given text.
 * If ignoreTrivia is set, whitespaces or comments are ignored.
 */
export function createScanner(text: string, ignoreTrivia: boolean = false): JSONScanner;
    
/**
 * The scanner object, representing a JSON scanner at a position in the input string.
 */
export interface JSONScanner {
    /**
     * Sets the scan position to a new offset. A call to 'scan' is needed to get the first token.
     */
    setPosition(pos: number): any;
    /**
     * Read the next token. Returns the token code.
     */
    scan(): SyntaxKind;
    /**
     * Returns the zero-based current scan position, which is after the last read token.
     */
    getPosition(): number;
    /**
     * Returns the last read token.
     */
    getToken(): SyntaxKind;
    /**
     * Returns the last read token value. The value for strings is the decoded string content. For numbers it's of type number, for boolean it's true or false.
     */
    getTokenValue(): string;
    /**
     * The zero-based start offset of the last read token.
     */
    getTokenOffset(): number;
    /**
     * The length of the last read token.
     */
    getTokenLength(): number;
    /**
     * The zero-based start line number of the last read token.
     */
    getTokenStartLine(): number;
    /**
     * The zero-based start character (column) of the last read token.
     */
    getTokenStartCharacter(): number;
    /**
     * An error code of the last scan.
     */
    getTokenError(): ScanError;
}

Parser:

export interface ParseOptions {
    disallowComments?: boolean;
    allowTrailingComma?: boolean;
    allowEmptyContent?: boolean;
}
/**
 * Parses the given text and returns the object the JSON content represents. On invalid input, the parser tries to be as fault tolerant as possible, but still return a result.
 * Therefore always check the errors list to find out if the input was valid.
 */
export declare function parse(text: string, errors?: {error: ParseErrorCode;}[], options?: ParseOptions): any;

/**
 * Parses the given text and invokes the visitor functions for each object, array and literal reached.
 */
export declare function visit(text: string, visitor: JSONVisitor, options?: ParseOptions): any;

/**
 * Visitor called by {@linkcode visit} when parsing JSON.
 * 
 * The visitor functions have the following common parameters:
 * - `offset`: Global offset within the JSON document, starting at 0
 * - `startLine`: Line number, starting at 0
 * - `startCharacter`: Start character (column) within the current line, starting at 0
 * 
 * Additionally some functions have a `pathSupplier` parameter which can be used to obtain the
 * current `JSONPath` within the document.
 */
export interface JSONVisitor {
    /**
     * Invoked when an open brace is encountered and an object is started. The offset and length represent the location of the open brace.
     */
    onObjectBegin?: (offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath) => void;

    /**
     * Invoked when a property is encountered. The offset and length represent the location of the property name.
     * The `JSONPath` created by the `pathSupplier` refers to the enclosing JSON object, it does not include the
     * property name yet.
     */
    onObjectProperty?: (property: string, offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath) => void;
    /**
     * Invoked when a closing brace is encountered and an object is completed. The offset and length represent the location of the closing brace.
     */
    onObjectEnd?: (offset: number, length: number, startLine: number, startCharacter: number) => void;
    /**
     * Invoked when an open bracket is encountered. The offset and length represent the location of the open bracket.
     */
    onArrayBegin?: (offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath) => void;
    /**
     * Invoked when a closing bracket is encountered. The offset and length represent the location of the closing bracket.
     */
    onArrayEnd?: (offset: number, length: number, startLine: number, startCharacter: number) => void;
    /**
     * Invoked when a literal value is encountered. The offset and length represent the location of the literal value.
     */
    onLiteralValue?: (value: any, offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath) => void;
    /**
     * Invoked when a comma or colon separator is encountered. The offset and length represent the location of the separator.
     */
    onSeparator?: (character: string, offset: number, length: number, startLine: number, startCharacter: number) => void;
    /**
     * When comments are allowed, invoked when a line or block comment is encountered. The offset and length represent the location of the comment.
     */
    onComment?: (offset: number, length: number, startLine: number, startCharacter: number) => void;
    /**
     * Invoked on an error.
     */
    onError?: (error: ParseErrorCode, offset: number, length: number, startLine: number, startCharacter: number) => void;
}

/**
 * Parses the given text and returns a tree representation the JSON content. On invalid input, the parser tries to be as fault tolerant as possible, but still return a result.
 */
export declare function parseTree(text: string, errors?: ParseError[], options?: ParseOptions): Node | undefined;

export declare type NodeType = "object" | "array" | "property" | "string" | "number" | "boolean" | "null";
export interface Node {
    type: NodeType;
    value?: any;
    offset: number;
    length: number;
    colonOffset?: number;
    parent?: Node;
    children?: Node[];
}

Utilities:

/**
 * Takes JSON with JavaScript-style comments and remove
 * them. Optionally replaces every none-newline character
 * of comments with a replaceCharacter
 */
export declare function stripComments(text: string, replaceCh?: string): string;

/**
 * For a given offset, evaluate the location in the JSON document. Each segment in the location path is either a property name or an array index.
 */
export declare function getLocation(text: string, position: number): Location;

/**
 * A {@linkcode JSONPath} segment. Either a string representing an object property name
 * or a number (starting at 0) for array indices.
 */
export declare type Segment = string | number;
export declare type JSONPath = Segment[];
export interface Location {
    /**
     * The previous property key or literal value (string, number, boolean or null) or undefined.
     */
    previousNode?: Node;
    /**
     * The path describing the location in the JSON document. The path consists of a sequence strings
     * representing an object property or numbers for array indices.
     */
    path: JSONPath;
    /**
     * Matches the locations path against a pattern consisting of strings (for properties) and numbers (for array indices).
     * '*' will match a single segment, of any property name or index.
     * '**' will match a sequence of segments or no segment, of any property name or index.
     */
    matches: (patterns: JSONPath) => boolean;
    /**
     * If set, the location's offset is at a property key.
     */
    isAtPropertyKey: boolean;
}

/**
 * Finds the node at the given path in a JSON DOM.
 */
export function findNodeAtLocation(root: Node, path: JSONPath): Node | undefined;

/**
 * Finds the most inner node at the given offset. If includeRightBound is set, also finds nodes that end at the given offset.
 */
export function findNodeAtOffset(root: Node, offset: number, includeRightBound?: boolean) : Node | undefined;

/**
 * Gets the JSON path of the given JSON DOM node
 */
export function getNodePath(node: Node): JSONPath;

/**
 * Evaluates the JavaScript object of the given JSON DOM node 
 */
export function getNodeValue(node: Node): any;

/**
 * Computes the edit operations needed to format a JSON document.
 * 
 * @param documentText The input text 
 * @param range The range to format or `undefined` to format the full content
 * @param options The formatting options
 * @returns The edit operations describing the formatting changes to the original document following the format described in {@linkcode EditResult}.
 * To apply the edit operations to the input, use {@linkcode applyEdits}.
 */
export function format(documentText: string, range: Range, options: FormattingOptions): EditResult;

/**
 * Computes the edit operations needed to modify a value in the JSON document.
 * 
 * @param documentText The input text 
 * @param path The path of the value to change. The path represents either to the document root, a property or an array item.
 * If the path points to an non-existing property or item, it will be created. 
 * @param value The new value for the specified property or item. If the value is undefined,
 * the property or item will be removed.
 * @param options Options
 * @returns The edit operations describing the changes to the original document, following the format described in {@linkcode EditResult}.
 * To apply the edit operations to the input, use {@linkcode applyEdits}.
 */
export function modify(text: string, path: JSONPath, value: any, options: ModificationOptions): EditResult;

/**
 * Applies edits to an input string.
 * @param text The input text 
 * @param edits Edit operations following the format described in {@linkcode EditResult}.
 * @returns The text with the applied edits.
 * @throws An error if the edit operations are not well-formed as described in {@linkcode EditResult}.
 */
export function applyEdits(text: string, edits: EditResult): string;

/**
 * An edit result describes a textual edit operation. It is the result of a {@linkcode format} and {@linkcode modify} operation.
 * It consist of one or more edits describing insertions, replacements or removals of text segments.
 * * The offsets of the edits refer to the original state of the document.
 * * No two edits change or remove the same range of text in the original document.
 * * Multiple edits can have the same offset if they are multiple inserts, or an insert followed by a remove or replace.
 * * The order in the array defines which edit is applied first.
 * To apply an edit result use {@linkcode applyEdits}.
 * In general multiple EditResults must not be concatenated because they might impact each other, producing incorrect or malformed JSON data.
 */
export type EditResult = Edit[];

/**
 * Represents a text modification
 */
export interface Edit {
    /**
     * The start offset of the modification.
     */
    offset: number;
    /**
     * The length of the modification. Must not be negative. Empty length represents an *insert*.
     */
    length: number;
    /**
     * The new content. Empty content represents a *remove*.
     */
    content: string;
}

/**
 * A text range in the document
*/
export interface Range {
    /**
     * The start offset of the range. 
     */
    offset: number;
    /**
     * The length of the range. Must not be negative.
     */
    length: number;
}

/** 
 * Options used by {@linkcode format} when computing the formatting edit operations
 */
export interface FormattingOptions {
    /**
     * If indentation is based on spaces (`insertSpaces` = true), then what is the number of spaces that make an indent?
     */
    tabSize: number;
    /**
     * Is indentation based on spaces?
     */
    insertSpaces: boolean;
    /**
     * The default 'end of line' character
     */
    eol: string;
}

/** 
 * Options used by {@linkcode modify} when computing the modification edit operations
 */
export interface ModificationOptions {
    /**
     * Formatting options. If undefined, the newly inserted code will be inserted unformatted.
    */
    formattingOptions?: FormattingOptions;
    /**
     * Default false. If `JSONPath` refers to an index of an array and `isArrayInsertion` is `true`, then
     * {@linkcode modify} will insert a new item at that location instead of overwriting its contents.
     */
    isArrayInsertion?: boolean;
    /**
     * Optional function to define the insertion index given an existing list of properties.
     */
    getInsertionIndex?: (properties: string[]) => number;
}

License

(MIT License)

Copyright 2018, Microsoft

node-jsonc-parser's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-jsonc-parser's Issues

Upgrading from 2.1.1 to 2.2.0 has diagnostic for file with only comments

The following jsonc file returns a diagnostic in 2.2.0:

/// test
//////asdf

Diagnostic:

ValueExpected:  (20)

I know this is a rare scenario, but I believe this is a valid jsonc file and a diagnostic should not be created.

Additionally, in this scenario the return value of parser.parseTree is undefined yet the return value is not defined as possibly returning that value.

Adding format and edit files breaks imports in TS projects using node module resolution

After this commit, I'm unable to figure out how to get webpack to play nicely with this library.

Adding the dependency and doing nothing else, I get Uncaught Error: Cannot find module ".". If I suppress the require function using imports-loader, everything used to work, but now I get an error because require is no longer a function.

This may be an issue with my understanding of how Webpack/TS handle different module resolution strategies being used concurrently, but I'm now somewhat stuck.

v3.2.1 has a breaking change

Specifically, it uses the ?? operator, which doesn't work in node 12 (and breaks eslint-plugin-import tests).

It was added here: #81 (comment)

Please remove use of this operator and publish a v3.2.2?

(I'm happy to make a PR if that's needed)

[question]: `ParseErrorCode` why `const enum`?

Just a question: Why did you switch from enum to const enum? I guess you had a good reason. Is there a better way to map the error code number to its "human-readable" name? Below snippet used to work before this change. Thanks!

if (parsingErrors && errors) {
    for (const error of parsingErrors) {
        errors.push(`${parser.ParseErrorCode[error.error]} at ${error.offset} offset of ${error.length} length`);
    }
}

Source


Problems with const enum:

enum E {
    A = 0,
    B = 1
}

const enum CE {
    CA = 2,
    CB = 3
}

console.log(E[E.A]); // `A`
console.log(CE[CE.CA]); // A const enum member can only be accessed using a string literal.

Feature Request: Provide endLine and endColumn for objects, arrays and values

I would like to get the line and column an object, array or value ends to be able to get the object at a specific location (line and column) in a document. I have a text based search and would want to get the object at the search location.

It would be awesome if the end line and end column would be provided in the visitor functions in the same ways as the start line and column:
jsoncParser.JSONVisitor.onObjectEnd
jsoncParser.JSONVisitor.onArrayEnd
jsoncParser.JSONVisitor.onLiteralValue

Publish types for setProperty

We use the setProperty function by itself. Right now, we use a fork of node-jsonc-parser because the main package doesn't publish types for non-main modules. Would you accept a PR that publishes types for all of the .js files in the npm package?

We use setProperty for the quick action buttons in the JSON settings editors on Sourcegraph, in case anyone's curious about the specific use case:
image

Question: Why are lineNumbers / column not a part of Node?

I'm looking at both the Monaco Editor's source code, which uses lineNumber and column to describe the position of text data.

However, this json parser keeps track of offset for Nodes.

Is there a specific reason for this? Other parsers like this return the tree with lineNumber and column info too.

Format: ability to provide custom property ordering.

It would be great if new object additions could be specified with a defined property ordering (in formatting options). Something like the insertion function that exists, but for total ordering of existing objects.

This way adding new objects could retain a canonical property ordering with a single call to modify, without having to piece together multiple modify removals and individual modify inserts.

Thanks!

Clarify whether / how `Edit[]` can be concatenated

The documentation currently does not make it clear whether or how Edit[] from multiple modify() or format() calls can be concatenated before being applied using applyEdits().

Both the documentation for modify() and format() contains the following:

Edits can be either inserts, replacements or removals of text segments. All offsets refer to the original state of the document. No two edits must change or remove the same range of text in the original document. However, multiple edits can have the same offset, for example multiple inserts, or an insert followed by a remove or replace. The order in the array defines which edit is applied first.

Personally I think this documentation should be moved to applyEdits() respectively to the documentation of the type Edit instead since it is more relevant there.

The issue is that the output of multiple modify() calls cannot be concatenated without risking to produce malformed JSON when calling applyEdits(). Here are two examples (from #48 (comment)):

  • Removing subsequent array elements / object properties.
    This will cause both elements / properties to remove the same ,, which in the end results in one character too much being removed:
    const json = '{"a":1,"b":2}'
    let edits = jsoncParser.modify(json, ["a"], undefined, {})
    edits = edits.concat(jsoncParser.modify(json, ["b"], undefined, {}))
    
    // Prints only `{`; the closing `}` is missing
    console.log(jsoncParser.applyEdits(json, edits))
  • Removing array elements / object properties in reverse order:
    const json = '{"a":1,"b":2,"c":3}'
    let edits = jsoncParser.modify(json, ["c"], undefined, {})
    edits = edits.concat(jsoncParser.modify(json, ["a"], undefined, {}))
    
    // Prints `{"b":2,"c":3`
    console.log(jsoncParser.applyEdits(json, edits))

So currently the only safe usage of modify() and applyEdits() is to call both functions for every modification you want to make. It is not safe to try concatenating the result of multiple modify() calls. This is rather inefficient when a user wants to remove multiple properties.

In case this is intended, then it should ideally be clarified in the documentation for the modify() (and format()?) function.

insertSpaces formatting option doesn't always work

Steps to reproduce. Run the following program using ts-node or similar.

import { format, applyEdits } from 'jsonc-parser';
const JSONString = `{"hello": \t"\tworld"}`;
const edit = format(JSONString, undefined, { insertSpaces: true});
const res = applyEdits(JSONString, edit);
console.log(res);

We would expect the following output

{
    "hello": "       world"
}

But we actually get

{
    "hello":    "       world"
}

Interestingly, if we remove the tab inside the quotes, the tab outside is removed as expected. This seems like an error since formatting should only care about tabs outside quotes.

Support multi line strings

related issue:
microsoft/vscode#15140

My proposal

Support multi line strings with python-like syntax

{
  "prefix": "foo",
  "body": """ 
int main() {
  return 0;
}"""
}
// Equals to
/*
{
  "prefix": "foo",
  "body": "\nint main() {\n  return 0;\n}"
}
*/

End of lines are automatically included in the string, but it’s possible to prevent this by adding a \ at the end of the line.

{
  "prefix": "foo",
  "body": """\
int main() {
  return 0;
}"""
}
// Equals to
/*
{
  "prefix": "foo",
  "body": "int main() {\n  return 0;\n}"
}
*/

Motivation

When we create a snippet with multi line, we have 2 choices.

  1. wrap the string of "body" every line. (it's bothersome)
{
  "prefix": "foo",
  "body": [
    "int main() {",
    "  return 0;",
    "}"
  ]
}
  1. join with \n (the longer the code, the harder it is to read)
{
  "prefix": "foo",
  "body": "int main() {\n  return 0;\n}"
}

There is no problem if the number of lines is small or if there are few opportunities to create snippets,
but there is room for improvement in terms of labor and readability for snippets heavy use.

I hope this proposal is taken into account. Thanks!

Unescaped tabs in strings should not be supported

The following throws in JS:

JSON.parse ( `"tab ->	"` );

But it parses with this library:

require ( 'jsonc-parser' ).parse ( `"tab ->	"` );

IMO that breakage from the spec is a bug, as it goes beyond what jsonc-parser says it does (namely parsing JSON with comments and trailing commas basically).

Convert Location / Segment[] to offsets/range

When writing semantic validators for JSON documents, it would be useful to be able to pass in a Segment[] and get back either character offsets or line and character numbers. This may be outside the charter of this library, but I couldn't find anything that fulfilled a similar need when reviewing the code for monaco-json or for the language server, but did find dependencies from them on this project.

Add dev container

I have a general preference for working with VS Code dev containers to keep tools and dependencies isolated between projects. Would you be open to a PR contributing a dev container?

parseTree - Option to keep comments

Thanks for this library! I wrote a jsonc code formatter using it.

Right now I'm using the scanner to get the comments around the nodes in the tree. It would be slightly faster to not rescan these out and instead get all the comments when calling parseTree.

A simple solution would may be to have an option that returns all the comments in an array. Maybe something like...

export function parseTree(text: string, errors: ParseError[] = [], options: ParseOptions = ParseOptions.DEFAULT): Node;

Goes to:

export function parseTree(text: string, options: ParseOptions = ParseOptions.DEFAULT): ParseResult;

interface ParseResult {
	file: Node;
	errors: ParseError[];
	/** Included when `options.includeComments` is true. **/
	comments?: Comment[];
};

Then:

export interface ParseOptions {
	disallowComments?: boolean;
	allowTrailingComma?: boolean;
	allowEmptyContent?: boolean;
}

Goes to:

export interface ParseOptions {
	disallowComments?: boolean;
	includeComments?: boolean;
	allowTrailingComma?: boolean;
	allowEmptyContent?: boolean;
}

Thoughts? I could submit a PR for this, but will just need some direction on how the public API should look.

Feature request: Add JSON path parameter to `JSONVisitor` functions

Rationale

The JSONVisitor is quite useful for validating the content of a JSON document (and getting locations of incorrect data). However, currently it is rather cumbersome to find out at which JSON path in the document the parser is when a visitor function is called.

Suggestion

It would therefore be good if some of the JSONVisitor functions had an extra parameter for the JSONPath of the currently visited element:

  • onObjectBegin, onArrayBegin, onLiteralValue
  • onObjectProperty: Here the property which is visited should probably not be part of the JSON path

For the other functions adding a JSONPath would probably either not add much value (onObjectEnd, onSeparator, onArrayEnd), or its behavior cannot be defined very well (onComment, onError).

However, because JSONPath is a mutable array (and is therefore modified when the parser proceeds), it would be sanest to only hand out copies of it to the visitor functions. A performant way in which this could be implemented is to have a supplier function of JSON path so the copy is only created when the user actually uses the JSON path. For example:

export interface JSONVisitor {
    onObjectBegin?: (
        offset: number,
        length: number,
        startLine: number,
        startCharacter: number,
        pathSupplier: () => JSONPath
    ) => void;

    ...
}

Based on the TypeScript documentation adding an extra function parameter should not be an issue.

What do yo think?

Workaround

Users implementing JSONVisitor can manually maintain a JSON path array, updating it in the respective visitor functions.

Demo implementation (might contain bugs)
const path = []

jsoncParser.visit(jsonContent, {
    onError(error, offset, length, startLine, startCharacter) {
        // ... error handling
    },
    onArrayBegin() {
        path.push(0)
    },
    onArrayEnd() {
        path.pop()
    },
    onSeparator() {
        // Check if inside of array (and therefore path element is array index)
        if (typeof path[path.length - 1] === "number") {
            path[path.length - 1]++
        }
    },
    onObjectBegin() {
        // Push temporary placeholder for object property names
        path.push(undefined)
    },
    onObjectEnd() {
        path.pop()
    },
    onObjectProperty(property, offset, length, startLine, startCharacter) {
        path[path.length - 1] = property
    },
    onLiteralValue(value, offset, length, startLine, startCharacter) {
        // ... process value
    }
})

parseTree() returns `undefined` on empty string input

I ran into an issue with jsonc.parseTree when my input file was an empty file thus causing the text parameter to be an empty string. My code did something very similar to the following:

const rawFileContents = (await readFile(myFilePath)).toString();
const root = jsonc.parseTree(rawFileContents, parseErrors);

if (root.type === MY_EXPECTED_TYPE) {/* do something. */}
else { /* handle invalid file */ }

This causes an exception at root.type when the file at myFilePath is empty because root is undefined.

Is this WAI or should the function signature below include undefined or the function itself return an empty Node when text is the empty string?

/**
 * Parses the given text and returns a tree representation the JSON content. On invalid input, the parser tries to be as fault tolerant as possible, but still return a result.
 */
export declare function parseTree(text: string, errors?: ParseError[], options?: ParseOptions): Node;

Enums are not documented

The only documentation for this library appears to be the README. However, this documentation is incomplete. The README documentation references several "enum-type" types, such as SyntaxKind (an enum{}) or NodeType (a union type supporting seven strings). Without knowing the allowed values of these enums, the library is not usable.

Clicking around in the github repo, I found src/main.ts, an extremely readable file with comments documenting the entire interface including the enums. I was able to use this to write my code. However a less experienced coder might not think to look for documentation in the src/ folder.

Expected behavior

All items from src/main.ts should be included in the README documentation, including the enums.

Error objects should also contain location information

Consider the following snippet:

{
  "one": 1
  "two": 2
}

If you run parseTree on it, it parses fine (which is cool), but the error that is reported has no other information than the error code. It would be nice if each error also contained the location information so they could be found in source (for example in an editor or IDE).

Information that would be handy to have:

  1. the starting offset of the error
  2. the length of the bad stuff
  3. the column start of the error
  4. the line number

Feature request: visit() is not useable with modify()

node-jsonc-parser presents several convenient ways of interacting: The scanner, the visit interface, or traversing a parse tree. Unfortunately, only the parse tree interface is compatible with modify/Edit/applyEdits. This means there are essentially three interfaces, but two are read-only.

Context

I have a lot of JSON files, from which I want to remove all instances of a particular deprecated property. I intended to use node-jsonc-parser to write a script to find those instances and delete them. At first I thought the Visitor interface would offer an incredibly simple way to do this; I could write a single JSONVisitor:

let edits:Edit[] = []
visit(jsonString, {
	onObjectProperty: function(property: string, offset: number, length: number, startLine: number, startCharacter: number) {
		if (property == "cursedPropertyName") {
			// Do something here to add "remove this property" to edits?
		}
	}
})
jsonString = applyEdits(jsonString, edits)

The realization I quickly hit was there is no way to create the Edit to move the property. modify() requires a JSONPath, obtaining a JSONPath (eg, findNodeAtLocation) requires providing a node or a root node (which means running the parse interface). I could create an Edit object manually with the offset and length of the property, but this might mean creating a noncompliant JSON document (eg, if I removed a property but not the preceding comma).

Expected behavior

There should be some way to use the convenient visitor-style interface with modify()/applyEdits(). Two ways I can think of to do this would be

  1. Add a Node.visit() interface. The reason I would prefer to use the visitor interface rather than parseTree is parseTree required me to write code to recursively traverse the tree of node children, a somewhat complicated construction for a simple find/replace script. However, jsonc-parser could just as easily provide a visitor interface to node trees, calling a visitor function and passing in the appropriate node for each node in the DOM tree, eg calling onObjectProperty for each NodeType="property" node.

  2. Add some variant of modify() that works with the offset/length arguments, but still knows how to produce edits that transform from a compliant JSON document to a compliant JSON document, eg, it also removes incidental material like whitespace and commas as appropriate. This option might be harder due to ambiguity about what to remove.

Another thing that would help would be simply being clearer in the documentation about what is supported. Writing my script once to use visitor and then starting over with parsetree was a bit frustrating, but if I had known to start with parsetree, I could have skipped the steps of trying to make it work with scanner and visitor and that would not have been frustrating. The documentation could make this clearer by, in the "Why" section, between the third and fourth bullet points, adding the non-bulleted text "Using parseTree enables the following extra functionality:".

Please publish new version?

I'd like to use printParseErrorCode but it's not included in the latest published version 2.0.2. Could you please publish a new version?

Parse errors make parsed tree useless

I have the following code snippet that can be run in node:

const { parseTree } = require('jsonc-parser');

const correctObj = '{"prop1":"foo","prop2":"foo2","prop3":{"prp1":{}}}';
console.log(parseTree(correctObj));

That produces the following correct output:

<ref *1> {
  type: 'object',
  offset: 0,
  length: 50,
  children: [
    {
      type: 'property',
      offset: 1,
      length: 13,
      parent: [Circular *1],
      children: [Array],
      colonOffset: 8
    },
    {
      type: 'property',
      offset: 15,
      length: 14,
      parent: [Circular *1],
      children: [Array],
      colonOffset: 22
    },
    {
      type: 'property',
      offset: 30,
      length: 19,
      parent: [Circular *1],
      children: [Array],
      colonOffset: 37
    }
  ]
}

However if I run it on a slightly malformed object like so (two double quotes in prp1):

const malformedObj = '{"prop1":"foo","prop2":"foo2","prop3":{"prp1":{""}}}';
console.log(parseTree(malformedObj));

I get a tree with two nodes which represent key and value of prop1:

<ref *1> {
  type: 'property',
  offset: 1,
  length: 13,
  children: [
    {
      type: 'string',
      value: 'prop1',
      offset: 1,
      length: 7,
      parent: [Circular *1]
    },
    {
      type: 'string',
      offset: 9,
      length: 5,
      parent: [Circular *1],
      value: 'foo'
    }
  ],
  colonOffset: 8
}

That makes it impossible to figure out which node is for example at offset 48.

My use case: I'm writing a VSCode extension for JSON completion. When the user starts typing, the object will always be malformed so that I get no information on the node which he is typing in.

Have you similar use cases and maybe solved it differently or is there any way to increase fault tolerance of the parser? My current workaround would be to remove the current word range from the text and parse the tree based on that. I'm by far not sure if this works in every possible situation.

Feature request: `modify` and `applyEdits` on parsed AST

I'm trying to keep a JSON datastructure in sync with its text representation for a custom VS Code editor.

I was thinking maybe it'd make sense to directly apply edit operations on the parsed tree rather than reparsing JSON every time?

getNodePath/modify do not work for property nodes (capture too much content)

When run on a property node, getNodePath appears to instead returns a path to the enclosing dictionary of the property. This results in some serious problems:

  • findNodeAtLocation(tree, getNodePath(node)) is a different node from node.
  • Calling modify() on a property node removes additional content other than the property node targeted.

Repro steps

Check out the jsonc-parser-demo branch of this github repo.

Run these steps:

npm install && npm run build
node . -d transformBackground testcase.json > testcase-post.json

src/app.ts in this repro is a short script that processes a json file and removes all instances of the property whose key name is given as the -d argument, then prints the file out to STDOUT. To demonstrate the bug more clearly, it currently removes only the first instance before bailing out, and once it has identified the property to be removed it console.warns to STDERR (1) the node it intends to remove (2) the getNodePath of the target node and (3) the roundtrip result of findNodeAtLocation(tree, getNodePath(node)) on the target node. The important code in this example is the code immediately following the comment "PERFORM OPERATIONS HERE"; everything else only serves to load the file and walk the node tree.

My expected behavior running this test script on the included testcase JSON is that on STDERR the "DELETING" and "BUT REALLY DELETING" nodes will be the same (ie, findNodeAtLocation(tree, getNodePath(node)) should be a noop); and on STDOUT the single property "transformBackground" in the dictionary levels/0/screen will be removed.

My observed behavior is quite different:

  1. The path printed is [ 'levels', 0, 'screen' ]. In other words, as printed, it points to the enclosing dictionary, not the transformBackground property.
  2. The "DELETING" prints as expected the property I'm looking for, but the "BUT REALLY DELETING" is different and contains much more content— it does appear to encompass the entire dictionary stored under "screen".
  3. After applying the edits, jsondiff shows this change for the output file: [{"path": "/levels/0", "value": {"color": [0.3, 0.37, 1], "png": "level/test/trigger/trigger.png"}, "op": "replace"}]. In other words, it shows the "screen" property and its value were deleted from /levels/0.

Analysis

I am able to get the "expected behavior" I desire by, instead of passing at.node to getNodePath in the "PERFORM OPERATIONS HERE" section, passing in either of its children. If I do this I get the path [ 'levels', 0, 'screen', 'transformBackground' ] as expected and this is indeed the content that gets removed. However this feels wierd, and additionally it contradicts the documentation. The comment/doc for modify says: " * @param path The path of the value to change. The path represents either to the document root, a property or an array item." This implies modify should accept property nodes.

Remove dependency to `vscode-nls`

This is a great parser.

However the fact that it relies on vscode-nls make it less suitable in a browser environment. vscode-nls pulls fs (node filesystem core lib) which is not available in browser. Also it's doing some dynamic require which is not wepback-safe:

{
 "node": {
        "fs": "empty"
    }
}

WARNING in ./~/vscode-nls/lib/main.js
118:23-44 Critical dependency: the request of a dependency is an expression

Without the empty config for fs I would get:

WARNING in ./~/vscode-nls/lib/main.js
118:23-44 Critical dependency: the request of a dependency is an expression

ERROR in .//vscode-nls/lib/main.js
Module not found: Error: Can't resolve 'fs' in '/Users/jattali/dev/mnubo/front-end/node_modules/vscode-nls/lib'
@ ./
/vscode-nls/lib/main.js 7:9-22
@ ./~/jsonc-parser/lib/main.js

Since vscode-nls is only used for localizations of error, I would say it's not critical to remove it.

What do you think @aeschli ?

Non-standard whitespace handling

The scanner in node-jsonc-parser allows multiple non-standard whitespace chars to be accepted as a whitespace:

function isWhiteSpace(ch: number): boolean {

However, JSON specification allows only a handful of whitespace chars:

  ws = *(
          %x20 /              ; Space
          %x09 /              ; Horizontal tab
          %x0A /              ; Line feed or New line
          %x0D )              ; Carriage return

Difference in whitespace handling leads to interop problems with other JSON parsers, where input successfully parsed by node-jsonc-parser would fail to parse in Node or Python.

Support "replace" utility function

Similar to the "modify" function, but allowing the key to be changed too.

The two use cases I have in mind for this is:

  • Renaming a key in-place while retaining the existing value
  • Replacing an existing key/value with new key/value in the same place

Of course, the existing modify function could be overloaded to support this, but I suggest a new function because:

  • It's not valid to supply the root for a replace
  • The meaning of undefined value param changes

Example of how this would look if "replace" is implemented:

/**
 * Computes the edit operations needed to replace a key/value in the JSON document.
 * 
 * @param documentText The input text 
 * @param path The path of the value to change. The path represents either to a property or an array item.
 * If the path points to an non-existing property or item, an error will be thrown. 
 * @param key The new key name for the specified property or item.
 * @param value The new value for the specified property or item. If the value is undefined,
 * the existing value will be retained.
 * @param options Options
 * @returns The edit operations describing the changes to the original document, following the format described in {@linkcode EditResult}.
 * To apply the edit operations to the input, use {@linkcode applyEdits}.
 */
export function replace(text: string, path: JSONPath, key: string, value: any, options: ModificationOptions): EditResult;

Option to keep spaces before inline comments

Add a keepSpaces or keepSpacesBeforeLineComments similar too keepLines that preserves the spaces between values and inline comments at the end of the line.

This will help to avoid unnecessary changes when formatting tsconfig.json files:

Original:

    /* Language and Environment */
    "target": "es2016",                                  /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
    // "lib": [],                                        /* Specify a set of bundled library declaration files that describe the target runtime environment. */

Formatted:

    /* Language and Environment */
    "target": "es2016", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
    // "lib": [],                                        /* Specify a set of bundled library declaration files that describe the target runtime environment. */

Diff:

     /* Language and Environment */
-    "target": "es2016",                                  /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
+    "target": "es2016", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
     // "lib": [],                                        /* Specify a set of bundled library declaration files that describe the target runtime environment. */

ncc & v3.1.0

Hello,

I'm using ncc to bundle a vscode extension which is using this library.
With v3.0.0, I don't have any issue.
With v3.1.0, ncc doesn't produce any errors but when running the extension, I get Cannot find module './format'

Allow the visitor to abort or cease recursion

I'm working on a code lens that should decorate npm scripts. For this, I'm parsing a JSON file, and only want the top level "scripts" key. I'm entirely uninterested in other properties, and once I have that I'm happy to abort parsing.

It'd be nice to be able to return a value from JSONVisitor methods which could indicate that we want to cease recursion on the current node, or abort parsing entirely.

Example?

A simple example of using this library would go a long way.

It would be nice to see:

  1. parse a string
  2. modify a single field
  3. stringify

Remove Property is not working as expected.

It's such a great tool to handle JSON (with comments). Thanks!

But I find that it can not pass the following test:

test('remove property', () => {
    let content = '{\n  "x": "y",\n  // This is a comment\n  "test": "1"\n}';
    let edits = removeProperty(content, ['x'], formatterOptions);
    assertEdit(content, edits, '{\n  // This is a comment\n  "test": "1"\n}');
});

I've added a comment right after the property x. What I want is to remove the property but keep the comment. But jsonc-parser removes the comment too, which I think this maybe a bug or something?

P.S. You can test this in edit.test.ts.

Formatting valid json content is causing an invalid json

Format following valid json

{"version": 1,"setttings": // This is some text
{
	// some comment
	"workbench.settings.editor": "json",
	"workbench.settings.useSplitJSON": true,
	"workbench.colorTheme": "Default Light+",
}
}

Output:

{
	"version": 1,
	"setttings": // This is some text {
		// some comment
		"workbench.settings.editor": "json",
		"workbench.settings.useSplitJSON": true,
		"workbench.colorTheme": "Default Light+",
	}
}

difficulty bundling with esbuild

I want to use node-jsonc-parser in a vscode extension and, per the guidance, I am using esbuild as the bundler. Unfortunately it doesn't work 'out of the box'; the internally required files (in src/impl/) are not bundled. I filed evanw/esbuild#1619 with esbuild. They explained the reason and suggested a work around, but also mentioned 'making the code (ie. node-jsonc-parser) more bundler friendly'.
I'd appreciate any advice on whether this is possible/straightforward to do, thanks in advance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.