jsonata-js / jsonata Goto Github PK

Hi,
We started using JSONata for one of our projects and discovered that we will need an ability to find the indices of matching predicates in the JSON path. Just wanted to see if we can add that ability.
Thanks.

We have a use-case that requires a JSONata function to cast hex-encoded binary data embedded inside a JSON document into a 32-bit float.

Our input is JSON similar to the following:

{
   "payload_cleartext":"540a41b1999a",
   ...
}

For instance, we need a function like:

$readFloat(payload_cleartext, 16, 4)

The first parameter is a hex string, the second defines the encoding of the input string (hex=base16), and the third is an optional start index into the input string (default:0). Here, the function would read 8 hex characters starting at index 4 ("41b1999a"), cast this into a float, and return the value 22.2.

More generally, we should consider use-cases with input types other than hex and output types other than (BE) 32-bit floats. To this end, we propose one of the following:

1. a suite of functions covering all permutations of input and output types,
2. a single function that accepts parameters to specify the input/output types,
3. or something inbetween 1 and 2

Option 3 can provide a good balance between function count and function complexity: i.e. one function per numeric type, with each function accepting a "base" parameter specifying how to interpret the input string (e.g. 16 for hex):

# float
$readFloatBE(input, base, [offset=0])
$readFloatLE(input, base, [offset=0])

# double
$readDoubleBE(input, base, [offset=0])
$readDoubleLE(input, base, [offset=0])

# signed ints
$readInt8BE(input, base, [offset=0])
$readInt8LE(input, base, [offset=0])
$readInt16BE(input, base, [offset=0])
$readInt16LE(input, base, [offset=0])
$readInt32BE(input, base, [offset=0])
$readInt32LE(input, base, [offset=0])

# unsigned ints
$readUInt8BE(input, base, [offset=0])
$readUInt8LE(input, base, [offset=0])
$readUInt16BE(input, base, [offset=0])
$readUInt16LE(input, base, [offset=0])
$readUInt32BE(input, base, [offset=0])
$readUInt32LE(input, base, [offset=0])

This layout mimics that of the widely used NodeJS Buffer library and so should be familiar to many. If this seems like too many functions, we could collapse groups down by adding parameters to control endianness and signedness.

Any feedback / suggestions would be greatly appreciated!

In many packages where callbacks are optional (e.g., mongodb), a function called without a callback returns a promise. jsonata seems to buck this trend and reverts to a purely synchronous invocation scheme (as discussed in #77) when invoked without a callback.

In order to activate async processing and return a Promise to the result, one needs to wrap the call to evaluate in a new Promise(...) call (as seen in the tests).

I realize that you have an established API, but you might wish to consider changing the semantics of evaluate to return a promise when invoked without a callback or, alternatively, add an additional evaluateAsync method that returns a promise (i.e., implements the wrapping once inside the library sparing users the burden to do this themselves).

I would be happy to submit a PR if either of these if they would be accepted...

Hi,
We are trying to filter all matching predicate based on the current context value.
for e.g.
Phone[number=$[0].number] . The expectation is to have number filtered by the 0th index value.
But when i try this, i am getting all Phone objects. Nothing seems to be filtered. Is this the expected behavior?
Thanks.

In the JSONata Exerciser, using the default JSON data structure, I am able to use the expression
Account.Order.Product."Product Name".$split($, /[ ,.\n]+/)
to generate a list of words used in the Product Names. Using a regex is a great feature, since not all strings can be split on just a list of characters.

While the string_functions.html documentation states that this feature is available, the functions.md file does not allow this:

The optional separator parameter specifies the characters within the str about which it should be split. If separator is not specified, then the empty string is assumed, and str will be split into an array of single characters. It is an error if separator is not a string.

Is the regex feature newer than the docs? The reason I'm asking is that the ace editor embedded in node-red uses the ace/mode/jsonata validators (rules?) that will not allow me to use a regex. I'm not seeing where that validation occurs in node-red, so I'm just wondering if it requires a change in either this library or the Ace editor library. Thanks for any help on who to contact to enable this feature in node-red.

Hello Andrew,

If a ES6 class definition is in the path, it is not picked up.

In the code 1800:

        } else if (input !== null && typeof input === 'object') {

This means that classes cannot be used. I propose the following addition:

} else if (input !== null && ( typeof input === 'object' || isClass(input))) {

where isClass is defined as:

    function isClass(func) {
        return typeof func === 'function' 
            && /^class\s/.test(Function.prototype.toString.call(func));
    }

/Martin

"skus":[{"a":1,"c":3},{"b":2}]
to
skus:{ "a":1,"b":2,"c":3 }

I've been trying to work on node-red project but running into problems trying to understand this piece of code from editor.js

Object.keys(jsonata.functions).forEach(function(f) {
     funcSelect.append($("<option></option>").val(f).text(f));
})

I've been looking at the jsonata.json for version 1.3 being reference in the package.json

Any idea what it could be?

I am playing around with jsonata, and I like it a lot. However, I have run into a problem I do not understand. There might be something I have misunderstood. So my apologies if this is a false negative.

I tested this at http://try.jsonata.org/ (Not sure which version is being used on the site)

Concider the following JSON:

{
  "Contacts" : [
  {
    "type": "contact",
    "id": "69496314488018A264F30C62D6FE7E97",
    "firstName": "Name 1",
    "lastName": "Surname 1",
    "address1": "Adress 1",
    "address2": null,
    "zipCode": "Zip1",
    "city": "Country1",
    "countryCode": "NO",
    "phonePrivate": null,
    "phoneWork": null,
    "phoneMobile": "phone1",
    "fax": "",
    "email": "[email protected]"
  },
  {
    "type": "contact",
    "id": "4742A63CF0B7F6861ACFC7A3543B9622",
    "firstName": "Name2",
    "lastName": "Surname2",
    "address1": "Adress2",
    "address2": null,
    "zipCode": "Zipcode2",
    "city": "City2",
    "countryCode": "NO",
    "phonePrivate": "phone2",
    "phoneWork": null,
    "phoneMobile": "mobile2",
    "fax": "fax2",
    "email": "[email protected]"
   }
],
  
"Relations" : [
  {
    "apiId": "69496314488018A264F30C62D6FE7E97",
    "system": "SystemX",
    "systemId": "SystemX.Id.1"
  },
  {
    "apiId": "4742A63CF0B7F6861ACFC7A3543B9622",
    "system": "SystemX",
    "systemId": "SystemX.Id.2"
  }
 ]
}

I would think that the following jsonata would filter the relation tab from the contact:

$.{
	"Contacts" : $.Contacts[].{
        "ifsId" : $$.Relations[apiId = $.id],
        "id" : $.id
    }
}

But the result is this:

{
  "Contacts": [
    {
      "id": "69496314488018A264F30C62D6FE7E97"
    },
    {
      "id": "4742A63CF0B7F6861ACFC7A3543B9622"
    }
  ]
}

In other words, using the $.id does not result in a hit in the Relation tab as I would expect. If I replace $.id in the filter like this: Relations[apiId = "4742A63CF0B7F6861ACFC7A3543B9622"], the Relations tab is filtered as expected. It seems that there is a problem using a dynamic value for the filtering.

There seems to be an issue with jsonata transforms that have a literal \T in them (such as a value for a JSON attribute that is a windows path C:\Test\test.txt. Interestingly this works correctly in the JSONata exerciser using a transform like

(
  $path := "\"C:\\Test\\test.txt\"";
  {
    "path": $path
  }
)

But it fails when I pass that transform as a string to the transformer

const transform = "{\"foo\": \"C:\\Test\\test.txt\"}";
// Compile the transform
  let transformer;
  try {
    transformer = jsonata(transform);
  }
  catch(error) {
    // Compile error.  Try to provide more useful information abut the error
    if (error.position) {
      throw new Error(`Transform compilation error: ${error.message} around "${transform.substring(error.position - 20, error.position + 20)}"`)
    }
  }

with Error: Transform compilation error: Unsupported escape sequence: "T" around "{"foo": "C:\Test\test.txt"}"

It also fails with C:\pest and C:\sest, but not with C:\fest, so it appears it is trying to treat \t, \s, and \p (case insensitive) as escape sequences.

Is it possible to add functions like formatNumber and formatDates?
E.g.,
formatNumber('1.2345',2,3) = 01.234
formatNumber('1',1,3) = 1.000
formatDate('2017-04-24T15:51:49', DD/MM/YYYY HH:mm:ss) = 24/04/2017 15:51:49

Hi I am currently in a situation where I need to run the Jsonata script under a Java environment. Currently we are using nashorn to do this but we heard that a new library called J2V8 is much faster in running JS inside Java. I was wondering if anybody here has any experience running J2V8 with the JSONATA.js script. I ask this because I am currently trying to run jsontata.js with j2v8 but keep getting a "java is not defined error". Any input would be great. I have also taken the time to provide our current nashorn code and j2v8 code in case anybody is familiar with the 2.

Nashorn:

`

    Invocable inv = (Invocable) scriptEngine;
    scriptEngine.put("input", jsonString);
    Object inputjson = scriptEngine.eval("JSON.parse(input);");

    // query the data
    Object expr = inv.invokeFunction("jsonata", jsonPath);
    Object resultjson = inv.invokeMethod(expr, "evaluate", inputjson);

    // JSON.stringify the result
    scriptEngine.put("resultjson", resultjson);
    result = scriptEngine.eval("JSON.stringify(resultjson);");

`

J2V8

`
runtime.executeScript(provided_js_script);
V8Function jsonataFunction = (V8Function)runtime.getObject("jsonata");

	V8Array arrayJsonata = new V8Array(runtime).push(jsonPath[0]); 		

	V8Object jsonataResult = (V8Object) jsonataFunction.call(null, arrayJsonata);

`

Hello I am currently trying to use the JSONATA $map statement that contains a conditional statement with my json path. An example of the code I am working with is provided below.

$map(--somejsonpath--, function($v, $k) {($v.example.value = "test" and $v.example.value.data = "device" and $v.example.value.valueInt > 1) ? $k})
Currently when I run this code I get an error that notes:

"Error\n at validate (<anonymous>:1881:29)\n at evaluateComparisonExpression (<anonymous>:1904:17)\n at evaluateBinary (<anonymous>:1604:30)\n at next (native)\n at evaluate (<anonymous>:1332:26)\n at next (native)\n at evaluateBinary (<anonymous>:1586:19)\n at next (native)\n at evaluate (<anonymous>:1332:26)\n at next (native)","value":"32","value2":1,"position":5,"token":"map","message":"The values \"32\" and 1 either side of operator \"map\" must be of the same data type"}

With the expression $v.example.value.valueInt I am essentially trying to acces the valueInt and see if it is greater than 1. The issue is however that it is in a set of nested arrays. I understand the params for the $map function are an array but for some reason I cannot get it to work with nested arrays. Is there any way I could get it to work with the current nested arrays I have now? Any help is appreciated, thanks.

sample data:

{
  "docId1": {
      "tags":["sales", "markting", "house"],
      "name": "How to sale house."
    },

  "docId2": {
      "tags":["tech", "smart home", "house"],
      "name": "How to make house smart."
    }
}

I want select doc by tag, returns like:

[
  {
    "name": "How to sale house.",
    "key": "docId1"
  },
  {
    "name": "How to make house smart.",
    "key": "docId2"
  }
]

My try is:

$.*["house" in tags].{"name": name, "key": ? }

I don't know how to reference wildcard key's name? Can you help me?

By the way, I know "$each" function is the answer. I just want to know is there some operators can fix my situation.

Thanks a lot.

In the following code:

if(isFunction(lhs)) {
    // this is function chaining (func1 ~> func2)
    // λ($f, $g) { λ($x){ $g($f($x)) } }
    result = yield * apply(chain, [lhs, func], environment, null);
} else {
    result = yield * apply(func, [lhs], environment, null);
}

apply is called with four arguments. But it only takes three. It doesn't hurt anything and I've fixed it in some code I'm working on. But I just wanted to point it out so I could reference this issue in a future PR.

Dear,

I was wondering if there is any way to use Jsonata for changing Json data, or returning a transformed version of the JSON data (containing the changes).

For example assume following input:

{
    "TOPIC1": {
        "last_value": "old value"
    },
    "TOPIC2": {
        "last_value": "old value"
    }
}

That needs to be transformed to following output (based on some Jsonata expression):

{
    "TOPIC1": {
        "last_value": "old value"
    },
    "TOPIC2": {
        "last_value": "new value"
    }
}

Since Jsonata is a query and transformation language (and not a manipulation language), I assume it is not possible... I have been debugging the entire expression evaluation, but couldn't find anything relevant.

But perhaps there is a workaround e.g. using a user defined function (that logs everything from the original data, except from the changes )?

Thanks in advance for your time !!!
Bart Butenaers

Changes in #51 seem to have resulted in:

Need to have another think about how to do Browser and Node.js safe btoa in a way that continues to work in the browser, but doesn't cause the browserify issues.

If you go to https://jsonata.slack.com/signup it is hardcoding @ibm.com as a domain.

Feel free to close this issue if it's working as intended. 😃

Also, thank you so much for a wonderful library. It is working out really well so far.

Per our discussion on the slack channel: here are some ideas for new functions related to

• Timestamps -- potentially providing such variants as Unix time (secs. since epoch), JavaScript time (millis since epoch), and Microsoft filetime (just kidding)
• Unique IDs -- variants such as UUID/GUID, or my favorite originally based on Firebase PushIDs

Although version 1.2 includes the $now() function, this returns an ISO 8601 formatted date string, which in a bit unwieldy to calculate differences between dates. I was looking for something more like the Unix timestamp, that is sortable and avoids timezone and locale formatting issues.

Hi,
We are stuck with a problem and am not sure if I missed something. I am trying to filter an array by matching an attribute in another array but it doesnt seem to work.
For eg.,
Accounts.Order[OrderID = Account.Shipping.OrderID] should result in the Order array filtered by matching orderID. In this example, Shipping is an object as well. I tried to use the $filter function as well with no success. Any help on how to solve this would be helpful. If this is a known limitation, please suggest a workaround.

Hello again. I have another issue I would need some input on.

I work with knex, a sql builder that uses chained methods to build the sql and then at the end uses a promise.
Converted jsonata to work with async/await simply by copy paste whenever it was generator functions. This works reasonably well.

Now I need to use both in the same evaluation, first chain a few methods in sync, run await and then await on all the following expressions.

Could we do a ~([expression]) addition to the syntax or something of the sort to tell jsonata that we should await?

Many thanks
Martin

let json2 = {
    a:[
        {$type:'aa', value:1},
        {$type:'aa', value:2},
        {$type:'bb', value:3},
    ]
}

let ret = jsonata("a[$type='aa'][]").evaluate(json2);
console.log(ret);

I am expecting [{$type:'aa', value:1}, {$type:'aa', value:2}]. But the result is undefined. How to write this? I have tried "a[\$type='aa'][]" "a[$type='aa'][]" "a['$type'='aa'][]" "a['\$type'='aa'][]", None of them actually works.

Hello there.

Is there any plans for code completion? Maybe based on source and target json schemas?

If you have a grammar for jsonata, I could maybe start working on one using peg.js?

(Aside: this is an awesome project; I'm enjoying using it in Node-RED!)

Description

In the docs, on the page about result structures, in the section on array constructors, the third example (non-wildcard multiple selection) doesn't seem to be working as described when run against http://try.jsonata.org/ 's Address dataset.

Current Behavior:

Desired Behavior:

Do you know of any GUI implementations, or plans to make any?

I am interested in building one in angular.

Excellent library! Thanks!

I see that this project includes a package-lock.json file. That file is meant to work with npm version 5.x.x which only comes with Node 8.x. So, can we assume Node 8 as the development platform? (package.json says Node>=4)

Keep in mind, I'm not proposing that we discontinue support for ES5. That is a separate issue. Just whether we can assume Node 8.x for development work.

Comments?

JSONata, as a query system should be tolerant to finding ‘nothing’ as a routine thing. This is reflected by the fact that passing the undefined value (the Javascript value used to represent 'nothing' here) to any operator or function does not raise an error, instead just itself returning undefined.

For instance:
Where the path x is not on the context:
x + 1 => undefined (no error is raised)

(To my knowledge, the only meaningful exception to this behaviour is $exists(undefined) => false, but there may be others).

This behaviour is not observed by the negate operator. E.g.
-x raises an error (Cannot negate a non-numeric value: undefined).

Is this another deliberate exception to this behaviour, or is this a bug?

I'm looking for something like a $distinct(array) function, that squashes an array into a unique list of values. Alternatively, I would think it could also be called $unique(...)

I've managed to write a couple expressions that build an object with keys, then gets the list of key names, but that seems too heavy handed to be very useful.

Ideally, this function would also take a second argument, a function that returns true/false (like $sift or $filter), to determine whether each element is unique. Arguments to this optional arg would probably follow the pattern of the $reduce(...) function, so the test logic can see what element is being checked, what its index is in the original array, and what the output array contains. But this is all just off the top of my head, so we have a starting point for any further discussions.

The incoming weather data object may contain some fields with null values -- like this:

{
  "temp": 22.7,
  "wind": 7,
  "gust": null,
  "timestamp": 1508971317377
}

I'm using the $lookup() function to find the "gust" value, with this expression:

{
	"x": $$.timestamp,
	"y": $lookup($$, "gust")
}

Instead of getting the property "y": null in the output object, this error is thrown instead:
Invalid operand to 'in': Object expected

Since I can use $$.gust to return the value null, this error is inconsistent and probably should be caught. I would prefer the null value to be returned, but it would also be acceptable to not create that field on the output object at all -- and would be consistent with the current behavior when trying to lookup a field that does not exist.

I see, from looking at the code, that Pratt's algorithm is being used for constructing the AST. But one downside is that I don't see any EBNF-like grammar here so it is hard to really understand the grammar rules.

Now I'm guessing that since you are using Pratt's algorithm, the operators are central to the parsing and perhaps that doesn't map well onto EBNF. But it would still be useful to understand how to form a grammatically correct expression.

Am I missing something?

I'm trying to do something that I thought would be easy, but running into some roadblocks...

As an example, let's try to convert an input Month abbreviation into the Date month number (0-based). It's easy to find out if the incoming string is in the array of months, using payload in $months, but finding out what position it occupies in the array is not so easy:

(
    $months := [
        "Jan", "Feb", "Mar", "Apr", "May", "Jun",
        "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"
    ];
    $months~>$map(function($val, $idx) {
        $val = payload ? $idx : undefined
    })
)

A bit clunky, perhaps, but it returns the index number for every array element that is equal to the incoming string. I tried using the $reduce($array, function($total, $value)) -- but unlike the Javascript function of the same name, there is no index value passed as the 3rd argument. At a minimum, I think the $reduce() function should be extended to also pass $index as an optional argument.

But perhaps a pair of new functions would be more intuitive and simpler to use, such as:

• $find(array, input) and the overloaded $find(array, function($value))
• $indexOf(array, input) and the overloaded $indexOf(array, function($value))

Both functions take an array of "things", and either a similar "thing" to find, or a function that determines which element is a match (returns true or false).

The documentation includes a description of a $merge function

http://docs.jsonata.org/object-functions.html#mergearrayobject

$merge(array)
Merges an array of objects into a single object containing all the key/value pairs from each of the
objects in the input array. If any of the input objects contain the same key, then the returned object
will contain the value of the last one in the array. It is an error if the input array contains an item
that is not an object.

but using it returns a "Invalid JSONata expression: Attempted to invoke a non-function" error.

Looking through the code, it does not appear to have been implemented.

It would be terribly helpful if we were able to access environment variables from within JSONata.

That would make it easier to mix in environmental settings such as the calling user's home folder or whether we are in a Dev/Prod environment.

The former is particularly challenging in a cross-platform environment since some OS's use the HOME environment variable but a Windows cmd shell uses HOMEPATH.

Working within JSONata would let us easily and automatically choose between them.

@bmpvieira has commented that JSONata "Doesn't do Streaming... yet...". I just wanted to kick of a discussion about how JSONata stream wrappers for Newline Delimited JSON data might look. Could it be a bit like this?

fs.createReadStream('data.txt') // Read a file
  .pipe(ndjson.parse()) // Convert to stream of JavaScript objects
  .pipe(JSONataStream.transform( // Wrapper for mutating the stream
    // Expression would be compiled once and evaluated against every object,
    // emitting the result
    expression: "{day: date}" 
  ))
  .pipe(JSONataStream.filter( // Wrapper for filtering out objects
    // Under the covers, the expression would become '$boolean(expression)' so as
    // to use JSONata's cast to boolean.  Only objects that are truthy would be emitted
    expression: "day > 5"
  ))
  .pipe(<next thing>)

For the above, through2-map and through2-filter could do the heavy lifting, resulting in a relatively simple implementation.

It would need a little bit of thought as to what happens when errors occur (both parsing the expression and at runtime), handling null (which is valid JSON, but would terminate the stream) and doubtless a few other things. Also we would need to find someone to actually do the work.... But at this stage, I just wanted a common understanding of what might be considered useful.

I've created a set of tests to reproduce a problem I've been having with evaluate. This demonstrates that if the input value is a Promise, things work mostly. But one case (involving sorting) seems to somehow deviate from the synchronous case.

Here is the code of the tests (you should be able to just drop it into jsonata-test.js):

const base_input = [
    { uri: "/foo/bar", metadata: { classes: ["foo", "bar"] } },
    { uri: "/bar", metadata: { classes: ["bar"] } },
  ];

describe.only("Bug in async?", () => {
    const asyncEval = (expr, input) => {
        return new Promise((resolve, reject) => {
            expr.evaluate(input, {}, (err, result) => {
                if (err) reject(err);
                else resolve(result);
            })
        })
    }
    it("should return an array synchronously", () => {
        var expr = jsonata("$");
        var result = expr.evaluate(base_input);
        expect(result).to.deep.equal(base_input);
    });
    it("should return a filtered array synchronously", () => {
        var expr = jsonata("$[\"bar\" in metadata.classes]");
        var result = expr.evaluate(base_input);
        expect(result).to.deep.equal([
            { uri: "/foo/bar", metadata: { classes: ["foo", "bar"] } },
            { uri: "/bar", metadata: { classes: ["bar"] } },
          ]);
    });
    it("should return a filtered and sorted array synchronously", () => {
        var expr = jsonata("$[\"bar\" in metadata.classes]^(>uri)");
        var result = expr.evaluate(base_input);
        expect(result).to.deep.equal([
            { uri: "/foo/bar", metadata: { classes: ["foo", "bar"] } },
            { uri: "/bar", metadata: { classes: ["bar"] } },
          ]);
    });
    it("should return an array asynchronously", () => {
        var expr = jsonata("$");
        return asyncEval(expr, base_input)
        .then((result) => {
            expect(result).to.deep.equal(base_input);            
        })
    });
    it("should return a filtered array asynchronously", () => {
        var expr = jsonata("$[\"bar\" in metadata.classes]");
        return asyncEval(expr, Promise.resolve(base_input))
        .then((result) => {
            expect(result).to.deep.equal([
                { uri: "/foo/bar", metadata: { classes: ["foo", "bar"] } },
                { uri: "/bar", metadata: { classes: ["bar"] } },
              ]);            
        })
    });
    it("should return a filtered and sorted array asynchronously", () => {
        var expr = jsonata("$[\"bar\" in metadata.classes]^(uri)");
        return asyncEval(expr, Promise.resolve(base_input))
        .then((result) => {
            expect(result).to.deep.equal([
                { uri: "/foo/bar", metadata: { classes: ["foo", "bar"] } },
                { uri: "/bar", metadata: { classes: ["bar"] } },
              ]);            
        })
    });
});

I'm trying to use jsonata in my front-end webpack project, importing it like this:

import jsonata from 'jsonata'

It works in a dev environment, but when I try to do a production build it fails with:

Creating an optimized production build...
Failed to compile.

Failed to minify the code from this file:

    ./node_modules/jsonata/jsonata.js:1438

Read more here: http://bit.ly/2tRViJ9

As per the referenced link, the main entry point is ES6 rather than ES5.

Hello I was wondering if there was a better way of calling evaluate. To expand, currently in Jsonata to evaluate on a set of data we execute the following code:

`

var jsonata = require("jsonata");

var data = {
example: [
{value: 4},
{value: 7},
{value: 13}
]
};
var expression = jsonata("$sum(example.value)");
var result = expression.evaluate(data); // returns 24

`

As we can see in this code, when we call expression.evaluate we are calling it on a set of js/json data. Currently I am writing an internal script which calls the expression.evalute on a set of the same data each time a new expression is put in the jsonata function to get the expression. I was wondering if there was perhaps a way to simple call our expression.evaluate on a set of data once, and then call jsonata(--some expression--) multiple times as our data is never changing and only are input expressions are. If further clarification is needed I can provide it.

I should note I understand I can assign variables to certain values and hold it in memory, but was wondering if I could do this similar thing to entire json files. Again if this is confusing I can provide examples of my issue.

During discussion of another issue, it became clear there is a desire to be able to embed comments within multiline expressions.

Options discussed included aligning with other scripting languages that use # to indicate the start of a comment, or Javascript/Java/C that use // or /* ... */ for comments.

This issue can be used to host any further discussion on this requirement.

JSONata has an implicit contract to return a position index when there is an error. Consider the following test which is an extension of an existing test, updated to check the position:

    describe('Error cases - (Evaluate)', function () {
        describe('General', function () {
            it.only('Closing brackets are mandatory', function () {
                expect(function () {
                    jsonata('$lowercase("Missing close brackets"').evaluate(person);
                }).to.throw()
                    .to.deep.contain({token: '(end)', value: ')', position: 35})
                    .to.have.property('message').to.match(/Syntax error: expected \'\)\' before end of expression/);
            });

It fails with:

  1) #evaluate Error cases - (Evaluate) General Closing brackets are mandatory:
     AssertionError: expected { Object (message, stack, ...) } to have a property 'position' of 35, but got undefined
      at Assertion.include (node_modules/chai/lib/chai/core/assertions.js:221:47)
      at Assertion.assert (node_modules/chai/lib/chai/utils/addChainableMethod.js:84:49)
      at Context.<anonymous> (test/jsonata-test.js:7434:30)

JSONata should be updated so that it returns a position in this situation. The position could either be the unmatched token (10) or the end of the expression (35).

I'm not sure what is going on in this function:

/**
 * Check if value is a finite number
 * @param {float} n - number to evaluate
 * @returns {boolean} True if n is a finite number
 */
export function isNumeric(n) {
    var isNum = false;
    if (typeof n === "number") {
        var num = parseFloat(n);
        isNum = !isNaN(num);
        if (isNum && !isFinite(num)) {
            throw {
                code: "D1001",
                value: n,
                stack: new Error().stack,
            };
        }
    }
    return isNum;
}

According to the JSDoc, n is presumed to be a number (not a string?). This is substantiated by the typeof n === "number" conditional. While parseFloat can take a number as an argument, what is the point if we know that n is already a number (which we know is the case because of the type check)?

Using this object:
{
"input": {
"text": "get weather at jfk"
}
}

If I call a function that expects an array as input:
$reverse(input.text)

it works, returning:
[
"get weather at jfk"
]

But, I would have expected it to reject the input as it is not an array (Array.isArray(input.text) is false. Interesting enough, because the function expects an array as input, it gets converted to an array somehow.

I ran across this trying to add a function to allow assignment of a value to an array cell by passing in the array and index and value and attempting: function assignCell(array,index,value)
{array[index]=value;return array;} and hadn't realized the input being sent to this function was a String by mistake. Of course the assignment failed (but no errors where thrown because the input had been transformed into an array, but the returned updated array was not accepted back.

I had used anx:a as the signature for the assignCell function above.

Even after setting the exerciser to use J:use 1.4

This fails:
Account.Order~>|Product.Description|{'Weight':(Weight * 16)}|

However this works:
Account~>|Order.Product.Description|{'Weight':(Weight * 16)}|

From the closed issue #70 I was hoping what was shown in the example image below was supported in the new 1.4 release:

Maybe the Exerciser use of the 1.4 branch didn't pick up the released 1.4 code yet?

Consider the following

const jsonata = require('jsonata')
try {
  jsonata('$replace("foo", "o, "rr")')
} catch (e) {
  console.log(e.message)
}

The error message printed out is:

Expected ")", got "(name)"

Which is not what I would have expected.

I am working on a Java implementation of the JSONata engine, and I'm struggling to understand JSONata's treatment of singleton arrays...

From http://docs.jsonata.org/complex.html:

Within a JSONata expression or subexpression, any value (which is not itself an array) and an array containing just that value are deemed to be equivalent.

Firstly, if so - why does the following hold?

[1] = 1 => false (should this not be true?)

So, outside of the '=' operator itself, I suspect the singleton/primitive equivalence is effectively achieved by "unwrapping" singleton arrays as they are referenced by expressions.

I see the following behaviour:

{'a': 1 }.a => 1
{'a': [1] }.a => 1
{'a': [[1]] }.a => 1
{'a': [[[1]]] }.a => [1]
{'a': [[[[1]]]] }.a => [[1]]

so it seems to me that JSONata is applying (at most) two levels of flattening to nested singleton arrays pulled out of an associative array via a path. Why 2? Why does it not completely flatten the output array in cases like this?

Next, if I wrap the statements above in an array constructor, I see the the following:

[{'a': 1 }.a] => [1]
[{'a': [1] }.a]=> [1]
[{'a': [[1]] }.a] => [1]

these results seem reasonable (given that the embedded statement returns 1 in those cases), however:

[{'a': [[[1]]] }.a] => [1]

should this not be [[1]]? (given than the statement inside the array constructor evaluates to [1]?

Hello there.

What happened to the comments? It was in the issue #75 that got closed but I cannot see that comments are allowed in version 1.4.

Regards
Martin

HI,

I've came into a problem with updating JSONata version that is used in Java Nashorn.
Simple expressions (without {}) are properly evaluated, but the execution of complex expressions (with {}) ends with an exception: Caused by: jdk.nashorn.internal.runtime.ECMAException: TypeError: Cannot call undefined

Simple expression is evaluated correctly.

@Test
  public void testJsonMerge(){

	String actual = nashornJsonataEvaluator.evaluateString("{\"name\":\"John\"}", "$.name");
	Assert.assertEquals("Name should be john", "John", actual);
	
  }

Complex expression that fails.

  @Test
  public void testJsonMergeComplex(){

	String actual = nashornJsonataEvaluator.evaluateString("{\"name\":\"John\"}", "$.{\"name\": name}");
	Assert.assertEquals("Name should be JSOn with name John", "{\"name\":\"John\"}", actual);
	
  }

Below there is a stacktrace of failed execution:

	at jdk.nashorn.internal.scripts.Script$Recompilation$245$90543A$jsonata_es5.L:1$L:1-1$L:1$L:2$jsonata$evaluateGroupExpression$evaluateGroupExpression$(/jsonata/jsonata-es5.js:2384) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$209$262558AAA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$tryCatch(/jsonata/jsonata-es5.js:6925) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$208$270231AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$makeInvokeMethod$invoke(/jsonata/jsonata-es5.js:7163) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$207$264651A$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$defineIteratorMethods$L:6975$L:6976(/jsonata/jsonata-es5.js:6977) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$209$262558AAA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$tryCatch(/jsonata/jsonata-es5.js:6925) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$236$272741AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$maybeInvokeDelegate(/jsonata/jsonata-es5.js:7225) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$208$270231AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$makeInvokeMethod$invoke(/jsonata/jsonata-es5.js:7137) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$207$264651A$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$defineIteratorMethods$L:6975$L:6976(/jsonata/jsonata-es5.js:6977) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$209$262558AAA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$tryCatch(/jsonata/jsonata-es5.js:6925) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$236$272741AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$maybeInvokeDelegate(/jsonata/jsonata-es5.js:7225) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$208$270231AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$makeInvokeMethod$invoke(/jsonata/jsonata-es5.js:7137) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$207$264651A$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$defineIteratorMethods$L:6975$L:6976(/jsonata/jsonata-es5.js:6977) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$209$262558AAA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$tryCatch(/jsonata/jsonata-es5.js:6925) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$236$272741AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$maybeInvokeDelegate(/jsonata/jsonata-es5.js:7225) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$208$270231AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$makeInvokeMethod$invoke(/jsonata/jsonata-es5.js:7137) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$207$264651A$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$defineIteratorMethods$L:6975$L:6976(/jsonata/jsonata-es5.js:6977) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$209$262558AAA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$tryCatch(/jsonata/jsonata-es5.js:6925) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$236$272741AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$maybeInvokeDelegate(/jsonata/jsonata-es5.js:7225) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$208$270231AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$makeInvokeMethod$invoke(/jsonata/jsonata-es5.js:7137) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$207$264651A$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$defineIteratorMethods$L:6975$L:6976(/jsonata/jsonata-es5.js:6977) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$209$262558AAA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$tryCatch(/jsonata/jsonata-es5.js:6925) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$236$272741AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$maybeInvokeDelegate(/jsonata/jsonata-es5.js:7225) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$208$270231AA$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$makeInvokeMethod$invoke(/jsonata/jsonata-es5.js:7137) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$207$264651A$jsonata_es5.L:1$L:1-1$L:6859$L:6860$L:6871$defineIteratorMethods$L:6975$L:6976(/jsonata/jsonata-es5.js:6977) ~[?:?]
	at jdk.nashorn.internal.scripts.Script$Recompilation$232$188977AAA$jsonata_es5.L:1$L:1-1$L:1$L:2$jsonata$jsonata$evaluate(/jsonata/jsonata-es5.js:5096) ~[?:?]

I'n using jsonata-es5.js file and nashorn-commonjs-modules.

The previous version of JSONata (currently 1.1.1) works well and this issue is blocking migration to higher version of JSONata library.

Thanks for help

This may be an issue with other node types too, but I noticed it with path when writing some evaluation code and I think I've seen at least two cases of it "in the wild".

In the evaluation code, you have the following code:

            if(ii === 0 && step.consarray) {
                resultSequence = yield * evaluate(step, inputSequence, environment);
            } else {
                resultSequence = yield * evaluateStep(step, inputSequence, environment);
            }

Basically this is saying the when faced with a path, you should treat them as transformations of the input value (because resultSequence becomes inputSequence as you loop over this code) unless the first one is an array constructor. In that case, you should set the result to the result of evaluating the array constructor.

It seems to me (and I could be totally wrong here), that this is actually a defect in the AST design. The reason I use the word "defect" is because when I looked at this code I thought of other ways it could fail and I think I'm able to demonstrate at least two different ones. But I can imagine an alternative way of designing the AST that could avoid this special handling (and the two failures I've seen).

The issue is with the implicit assumption in the path type that the current context value as the initial value for all each step. The array constructor breaks this because it defines an entirely new context. But the problem I see is...it isn't alone.

Consider, for example, evaluating this:

[{"size": 1}, {"size": 2}].size

From this we get [1, 2]. However, wrapping our array construction in a block using parentheses, i.e.,

([{"size": 1}, {"size": 2}]).size

Now gives us this: [1,2,1,2,1,2]. ??? (even though if we drop the .size from both expressions, we get exactly the same answer).

Note it isn't just block nodes either. I suspect a function evaluation could trigger the same issue in some circumstances. I suspect there are other cases as well.

It seems to me that a clean way to address this would be to have an explicit head AST node associated with each path node. This head node would represent the initial value in the loop over the steps and in most cases, its value would be the equivalent of $ (the current context). But the key thing here is that the parser could determine this not the evaluator. I don't know for certain because I'm not that familiar with the grammar, but my guess is that the parser can determine unambiguously what the "head" of the expression is and encode that in the AST so that we don't need to worry about these special cases while tree-walking.

Or am I completely off base here? In any case, I think there is a bug here (see my examples above). My point is that I think this can be addressed cleanly at the AST level.

Given input like:

{
  "msg": {
    "payload": {
      "originalInput":"",
      "input": {
        "text": "get weather for jfk"
      }
    }
  }
}

and a rule using J:branch 1.4
msg~>|payload|{"originalInput":(msg.payload.input)}|

I expected:

{
  "payload": {
    "originalInput":{
      "text": "get weather for jfk"
    },
    "input": {
      "text": "get weather for jfk"
    }
  }
}

but got:

{
  "payload": {
    "originalInput": "",
    "input": {
      "text": "get weather for jfk"
    }
  }
}

even though (msg.payload.input) evaluates to:

{
      "text": "get weather for jfk"
}

If instead, I use msg~>|payload|{"originalInput":"junk"}| then it gets updated

It would be useful if the following JavaScript string operators were exposed to JSONata:

• encodeURIComponent()
• encodeURI()
• decodeURI()
• decodeURIComponent()

This functionality would be useful in order to write transformations which produce or consume URLs/URIs. Would adding such functionality be in the spirit of the library's intended purposes?

Hello Andrew.

I open a new issue regarding mixins.

Currently with a few lines of code, I can extend jsonata to fetch missing data from a database with orm relations. Other data sources could also be possible.

I would like a standardized way of these extensions, typically by the way of mixin.
Example.
Instanciation:

            "orm": function(input, expr){
                const Model = require('objection').Model;
                // If input is a model class instance and has orm relation
                if(input instanceof Model && input.constructor.$$relations[expr.value]){
                    //load relation from db
                    return input.$relatedQuery(expr.value);
                }
            }            else{
                for(var mixinName in mixin){
                    mixin[mixinName](input,expr);
                }
            }
        }
        expression.evaluate(data, bindings, callback, mixins);

In jsonata, method evaluateName, if nothing found:

                for(var mixinName in mixins){
                    mixins[mixinName](input,expr);
                }
            }

What do you think?

Cheers
Martin