nanoporetech / data-rambler Goto Github PK

Path expressions can include segments that sort the incoming data. We support parsing these but do not evaluate them at this time. They are slightly complicated in that they support sorting by multiple criteria, so this will require a slightly more complex implementation and careful consideration of JSONata behaviour.

It's not clear at this time if sorting occurs over an entire match of a path expression or just on the current branch ( like filters ).

Path expressions use an automatic packing/unpacking algorithm in several places. It works on 2 equivalency rules, where any place one would be accepted the other must also. Also when resolving a value the latter is the preferred output.

[ value ] == value
[] == undefined

This means in some scenarios you will get a value or undefined instead of an array when the input data changes, making the expressions somewhat unpredictable.

I propose we modify this behaviour slightly to make it more predictable. We unfortunately cannot do this everywhere as it breaks some nice to haves.

• path expressions will always resolve to an array.
• field expressions will normally resolve to an array but may also resolve to the value of the only element or undefined.
• wildcard expressions will normally resolve to an array but may also resolve to the value of the only element or undefined.

let output = ({ a: 1 }).a 

We could support regex literals in the scanning stage, this would allow us to process strings in much more powerful ways and would be consistent with JS behaviour. Supporting these in the scanner would be tricky, but not impossible.

https://github.com/mozilla/sweet.js/wiki/design

Using the JSONata playground example (see below) I get different output when using JSONata and Data Rambler

With the expression Account.Order.[OrderID, [Product.'Product Name']]

JSONata gives:

[
  [
    "order103",
    [
      "Bowler Hat",
      "Trilby hat"
    ]
  ],
  [
    "order104",
    [
      "Bowler Hat",
      "Cloak"
    ]
  ]
]

However Data Rambler gives flattening the result:

[
  "order103",
  "Bowler Hat",
  "Trilby hat",
  "order104",
  "Bowler Hat",
  "Cloak"
]

The expected output from the expression is that of the JSONata output above

Input source:

{
  "Account": {
    "Account Name": "Firefly",
    "Order": [
      {
        "OrderID": "order103",
        "Product": [
          {
            "Product Name": "Bowler Hat",
            "ProductID": 858383,
            "SKU": "0406654608",
            "Description": {
              "Colour": "Purple",
              "Width": 300,
              "Height": 200,
              "Depth": 210,
              "Weight": 0.75
            },
            "Price": 34.45,
            "Quantity": 2
          },
          {
            "Product Name": "Trilby hat",
            "ProductID": 858236,
            "SKU": "0406634348",
            "Description": {
              "Colour": "Orange",
              "Width": 300,
              "Height": 200,
              "Depth": 210,
              "Weight": 0.6
            },
            "Price": 21.67,
            "Quantity": 1
          }
        ]
      },
      {
        "OrderID": "order104",
        "Product": [
          {
            "Product Name": "Bowler Hat",
            "ProductID": 858383,
            "SKU": "040657863",
            "Description": {
              "Colour": "Purple",
              "Width": 300,
              "Height": 200,
              "Depth": 210,
              "Weight": 0.75
            },
            "Price": 34.45,
            "Quantity": 4
          },
          {
            "ProductID": 345664,
            "SKU": "0406654603",
            "Product Name": "Cloak",
            "Description": {
              "Colour": "Black",
              "Width": 30,
              "Height": 20,
              "Depth": 210,
              "Weight": 2
            },
            "Price": 107.99,
            "Quantity": 1
          }
        ]
      }
    ]
  }
}

Our current path expression evaluator is based on a breadth first process that processes data for each path segment in sequence. This is a relatively optimal way to evaluate content but it comes with some tradeoffs in compatibility.

The original algorithm in the JSONata spec is based on a recursive depth first system. This is well suited for certain concepts like binding a variable for a particular branch of the tree. To work around this in our algorithm we had to add a special recurse step for this variable binding, but this doesn't work perfectly.

Resolving these edge cases is becoming increasingly complex and fragile, we've reached the point where the easiest way forward would be to replace our existing algorithm with a depth first system.

The following functions from JSONata are missing.

• substring
• sort
• shuffle
• assert
• error
• lookup
• foldLeft
• filter
• map
• formatBase
• formatNumber
• base64encode
• base64decode
• substringAfter
• substringBefore
• replace
• match
• contains
• split
• type
• power
• not

In some cases we have provided syntactic capability that replaces them, in others it's not clear that they really need to be part of the stdlib. But there are still a number which still need to be implemented.