Proposal: Function(s) for casting string-encoded binary data to numeric values about jsonata HOT 10 OPEN

@bartbutenaers careful not to feature creep here. The specific requirement is, given a string of hex encoded binary data, be able to parse out different numeric types from it. This is not about converting something that is already a number type to a different type of representation.

There is certainly scope for that discussion, but I don't that's what this issue from @tomklapiscak is focussed on.

Also worth getting @dceejay in the loop for his input.

from jsonata.

It would be interesting to get opinions and feedback from the Node-RED community because of the relevance to IoT (parsing sensor data). fyi @knolleary @shrickus @bartbutenaers

from jsonata.

Makes sense to me.

I'm personally not a fan of the dozen functions they have on the Buffer API, rather than more succinct, but overloaded, functions. But I can't point at another 'standard' API as an alternative - and there's no point in reinventing yet another api.

from jsonata.

Wow, that is a lot of functions -- due to all the permutations of options (endianness, base, byte count). So I must agree with Nick on this... the JSONata function space is nice and compact, and adding all these named functions is not ideal.

But it occurs to me that we already have the node-red-node-binary logic that does something similar, it encodes/decodes binary buffers using the packet pattern syntax. I have not thought through how well it would cover all those buffer functions that were requested, but I think it may allow us to have just two new functions: something like:

• $msgPack(object, pattern) -- convert a number/string/object to a buffer, and
• $msgUnpack(buffer, pattern) -- parse a binary buffer into a number/string/object

From the packet readme, the javascript example syntax for decoding IP headers could be adapted to look something like this:

$msgUnpack(payload, {
    "header": { "format": "b8", "parts": { "version": "b4", "headerLength": "b4" } },
    "typeOfService": "b8",
    "length": "b16",
    "identification": "b16",
    "optional": { "format": "b16", "parts": { "flags": "b3", "fragmentOffset": "b13" } },
    "timeToLive": "b8",
    "protocol": "b8",
    "checksum": "b16",
    "sourceAddress": "b32",
    "destinationAddress": "b32"
})

In typical JSONata style, the datatype of the 2nd arg (pattern) would determine the returned datatype -- so the pattern could be a simple string like "b64f" (to return a big-endian IEEE 754 floating point number), an array of formats like [ "b8", "b8" ] to return an array of 8-bit integers, or an object (like the example above) to return an object with the named properties (which can be objects containing other properties).

I'm not sure how the "base" would be handled, but I wonder if converting to another base should really be its own function anyway. Then the (e.g.) array of ints returned by the unpack could be chained to a $base(input, base) function, some thing like $msgUnpack(payload.rgb, [ "b8", "b8", "b8" ])~>$base(16) to get an array of hex color values.

from jsonata.

Hey Andrew,

I'm afraid that 'covering all permutations', will not be an easy task, as the others already pointed out.
For example there was a new question on the Node-Red forum today:

A user wants to convert the number in the payload:

• 0 => 'OFF'
• -1 => 'ON'
• number > 0 => x hr, y min, z secs

Indeed, a number can also represent a boolean, a switch value, a time interval ...
Bart

from jsonata.

sorry - only just seen this... Personally I like the pack / unpack as @shrickus mentions. It uses the "Packet" language - https://github.com/bigeasy/packet , but allow unpacking of a complete packet into a set of objects in one go using a template. As usual it's yet another thing to get you head around, but is quite powerful, as it also allows the reverse as well.

from jsonata.

Thank you all for your feedback and suggestions. It seems @shrickus's proposal is popular, and I certainly appreciate the importance of keeping the JSONata function space compact.

As @shrickus mentions, applying it to meet the requirements of my original use case probably needs a separate function for converting the (hexademical) UTF input string into a binary buffer.

$fromBase(string, base)
Converts a UTF string representation of data in some base (e.g. base16 = hex = chars 0-F) into a binary buffer.

The output of this forms the input to $msgUnpack. The pattern string skips the first 16 bits (i.e. the first 4 hex chars) and pulls the rest out as a 32-bit big-endian float.

$msgUnpack( $fromBase("540a41b1999a", 16), "x16, b32f" )

from jsonata.

Great discussion, and I like @shrickus's proposal. It also has the benefit that it could be developed incrementally, e.g. a simple string pattern for the first release (which would meet @tomklapiscak's original requirement), then array/object patterns in subsequent releases.

The bit I have a problem with is the idea of a buffer as the first argument. I assume this refers to node.js Buffer objects or ES6 TypedArrays. In either case, these are not representable in JSON and so would not be an expected input to JSONata. So I would like to amend this proposal to replace the buffer argument with a string of hex octets (e.g. "540a41b1999a").

Would this still be useful?

from jsonata.

Well, yes, I'm kinda stuck in the javascript mindset right now -- but I think I used the word "buffer" as the first arg to unpack to show it was different than the "object" passed to the pack function. Practically, that will be some representation of binary data, so maybe "data" is a better arg name. But you raise a good question... what datatype(s) would make sense to be allowed? Obviously they would need to be represented in some json-compatible way.

I don't like the idea of a string of octets, but mostly because I've not had a need to use them so far. I'm sure lots of systems would generate this kind of data, so it does make sense. The other problem I have with it is that it looks so similar to data that has been encoded using something like base64. I realize that is a different function, but you know someone will try to get the data directly out of an encoded basic-auth header field or run-length-encoded image string... and you don't want to have to handle any decoding/decompression/decryption processing within this unpack function.

My use cases would probably see arrays of integers as the most common input (like from Modbus read registers and such). Fortunately, the unpack function could handle both types of input just by detecting the input datatype. But what elements are allowed inside the array? I would think both of these should be valid:

[  72, 101, 108, 108, 111,  44,  32,  87, 111, 114, 108, 100,  33 ]
[ 'H', 'e', 'l', 'l', 'o', ',', ' ', 'W', 'o', 'r', 'l', 'd', '!' ]

Each has its own restrictions: the ints of the first array must be in the range 0-255, and the chars in the second must be 8-bit ascii.

from jsonata.

Not sure where this stands, but ...

So I would like to amend this proposal to replace the buffer argument with a string of hex octets (e.g. "540a41b1999a")

data that has been encoded using something like base64.

hex octets and base64 are both common ways of representing binary data in text/string format.
In the general case, there is no way to distinguish between the two, e.g. "0000".
Additional functions for converting between base64 and hex octets would do the trick.

from jsonata.