qntfy / kazaam Goto Github PK

• string paths are handled improperly
• v3 and v5 isn't handled properly
• failover to default values doesn't work
• spec should be updated for nameSpace to not be camelcased. it's one word

Add a transform that allows a user to create a UUID based on the presence of fields. If those fields are not there it can then generate a new UUID

Under the "Usage" section of the readme, the reader is instructed to cd into the Kazaam directory after running go get:
cd $GOPATH/src/gopkg.in/qntfy.kazaam.v3/kazaam

But this is the wrong path. The correct command with the correct path is:
cd $GOPATH/src/gopkg.in/qntfy/kazaam.v3/kazaam

Whichever transformation spec example I am trying with the command line tool, I get this error.

Write up a Readme to describe the project, its uses, status, and some examples.

im not really sure why the simplejson type is not being interpreted correctly

In README.md the json message under "executed on a JSON message with format" needs a "," .

It shows:

{
  "doc": {
    "uid": 12345,
    "guid": ["guid0", "guid2", "guid4"]
    "guidObjects": [{"id": "guid0"}, {"id": "guid2"}, {"id": "guid4"}]
  },
  "top-level-key": null
}

when it shall show:

{
  "doc": {
    "uid": 12345,
    "guid": ["guid0", "guid2", "guid4"],
    "guidObjects": [{"id": "guid0"}, {"id": "guid2"}, {"id": "guid4"}]
  },
  "top-level-key": null
}

Currently, while processing an over directive, the code alters the structure by introducing extra nesting, which negatively affects subsequent operations in multi-operation transformations.

For example:

kSpec := `[{
	"operation": "shift",
	"over": "a",
	"spec": {"bar": "foo"}
}]`
jsonIn := `{"a":[{"foo": 0}, {"foo": 1}, {"foo": 2}]}`

produces

{"a":[{"bar":0},{"bar":1},{"bar":2}]}

However,

kSpec := `[{
	"operation": "shift",
	"over": "a",
	"spec": {"bar": "foo" }
}, {
	"operation": "shift",
	"over": "a",
	"spec": {"buz": "bar" }
}]`
jsonIn := `{"a":[{"foo": 0}, {"foo": 1}, {"foo": 2}]}`

results in

 {"a":null}

If you use $unix and your value on the source json is formatted as integer (without quotes) the transform will throw an error.

It doesn't seem like Kazaam supports some of the wildcard functionality that Jolt offers in terms of the shift operation...

As an example, here is a Jolt shift spec:

  {
    "operation": "shift",
    "spec": {
      "job": {
        "task": {
          "*": {
            "query": {
              "*": {
                "name": "tasks[&3].queries[&1].query",
                "enabled": "tasks[&3].queries[&1].query_enabled",
                "@(2,name)": "tasks[&3].queries[&1].task_name",
                "@(4,name)": "tasks[&3].queries[&1].job_name"
              }
            }
          }
        }
      }
    }
 }

Specifically:
"&" - references key values captured by a walked wildcard "*" path to get a value
Example:

{
    "foo" : {
        "bar": {
            "baz":  // &0 = baz, &1 = bar, &2 = foo
        }
    }
}

"@" - reference another key by name in a walked wildcard "*" path
Example: "@(3,title)"
"3" means go up the tree 3 levels and then lookup the key "title" and use the value at that key

Any plans to support these?

Should use glide to manage dependencies in case there are any breaking changes under the hood.

It would be nice to be able to specify a string to be concatenated to the contents of a given field. This could potentially be extended to regex more generally.

Hi, thx for the great work. I'm struggling to define a spec to transform the first element (labels) of an array in an object.

from:
{ "metadata": [ { "label": "Amount", "labels": [ { "en": "Amount", "ptBR": "Quantidade" } ] }, { "label": "Value", "labels": [ { "en": "Value", "ptBR": "Valor" } ] } ] }

into

{ "metadata": [ { "label": "Amount", "labels": { "en": "Amount", "ptBR": "Quantidade" } }, { "label": "Value", "labels": { "en": "Value", "ptBR": "Valor" } } ] }

Can someone please guide me on how to achieve this? Thx!

Required fields like operation should be required and if not present an error should be thrown.

If you have an object like this:

{ 
"myrootkey":
   "subkey1": "field1",
   "subkey2": "field2"
}

How do you get the "myrootkey" literal value?

Add Travis-CI continuous integration and integrate build results into README.

Add an optional flag to shift which allows the shifting of elements while not changing any of the existing elements.

Would be useful to be able to delete keys in place as a transform, does not currently exist.

Using: go version go1.11.1 linux/amd64 in a known-working environment.

On the page/section: https://godoc.org/gopkg.in/qntfy/kazaam.v3#Config.RegisterTransform

The example block shown, when copied and used verbatim, produces:

cannot use func literal (type func(*"gopkg.in/qntfy/kazaam.v3/transform".Config, []byte) ([]byte, error)) as type kazaam.TransformFunc in argument to kc.RegisterTransform

While jsonpath is largely supported in specifying a source path, we cannot currently use it for specifying a target path.

Must be able to specify to set a value at:
meta.username[0].results[0], for instance.

Tests often fail due to the random order of JSON keys. This may be due to golang's randomized map iteration. All of the tests should be reviewed to ensurer they account for this fact and that go test doesn't fail at random.

Specks to transform array to array. array length will be dynamic.

eg:

{
"prices": [
{
"name": "Journey Mileage",
"type": "BASE",
"tax": "V20",
"amount": 30,
"rate": 2.5,
"unit": "miles",
"charge": 75
}
]
}

output:
{
"charges": [
{
"amount": "2095",
"currency": "EUR",
"type": "price_base"
}
]
}

First of all, I would like to thank everyone for contributing to this project, it's really great and works very well. 👍

Feature :
Currently, we can access the first element of an array of JSON objects with indexing.
The first element of an array: array[0]
Nth element of an array: array[n]
Can we please add a feature to do reverse indexing of the array? This would be very useful when one would be aware of the length of an array.
Or access the last element of an array with index -1.

I would like to take up this task if it can be accepted.

Hello all. I love your library, but I needed the ability to convert values as well as having a bit of conditional logic to decide whether values would be mapped based on the values of other values in the JSON document.

I put together a feature change that supports inline JSON Path Expressions for conditional branching and conversion of values, and would be curious if this would be something welcomed into the project?

Use case: let's say I had a JSON document with the following money value:
{ "annual_revenue": "$1,000,000" } and I wanted the output to be: {"agr":"1000000.00"} if the field existed, or {"agr":"0.00"} if it did not.

The above example could be achieved with:

{
  "operation": "shift",
  "converters": {
    "regex": {
      "remove_dollar_sign": {
        "match": "\\$?\\s*(.*)",
        "replace": "$1"
      },
      "remove_comma": {
        "match": ",",
        "replace": ""
      }
    }
  },
  "spec": {
    "agr": "annual_revenue ? \"0.00\" | regex remove_dollar_sign | regex remove_comma | ston | format %.2f"
  }
}

I've updated the documentation with syntax examples for all of the Converters that I added, and have the feature built out with tests in this same commit:

https://github.com/mbordner/kazaam/blob/feature/value-conditional-and-converter-expressions/README.md#json-path-parameters

https://github.com/mbordner/kazaam/blob/feature/value-conditional-and-converter-expressions/README.md#converter-specification-support

Another example: Say, I want to change the state name from Ohio to OH in the following document: {"state":"Ohio"} to produce: {"state":"OH"}

I could achieve this with:

{
  "operation": "shift",
  "converters": {
    "mapped": {
      "states": {
        "Ohio": "OH",
        "Texas": "TX"
      }
    }
  },
  "spec": {
    "state": "state | mapped states"
  }
}

or with

{
  "operation": "shift",
  "spec": {
    "state": "state | upper | substr 0 2"
  }
}

When path validation is required the array wildcarding breaks.

Many IoT systems produce JSON strings for MQTT brokers. And many of them will populate a tinmestamp as millis from epoche. It would be nice to have another $unix macro for supporting milli since epoche. (Maybe as $unixext )

It would be great if a timestamp field could be specified and a transformation applied to the string. For example you could transform a Unix timestamp to standard GMT. Probably could use the built-in golang date formatting as a template.

Add support in Coalesce() to take an optional ignore array parameter. If a found key is set to a value in ignore, Coalesce() will skip it as though it doesn't exist. Useful for ignoring things like {"key": ""}

does kazaam alow type convertion?
eg: input 👍 {
"price": "35.94"
}

output:
{
"p":35.94
}

Ensure that json that is kazaam'd doesn't modify how long integers are handled.

See: http://stackoverflow.com/questions/22343083/json-marshaling-with-long-numbers-in-golang-gives-floating-point-number
https://go-review.googlesource.com/#/c/30371/

JSON decoding may need to change to something along these lines:

        var src map[string]interface{}
        decoder := json.NewDecoder(bytes.NewReader(payload))
        decoder.UseNumber()
        err := decoder.Decode(&src)

The readme stated for the timestamp the following syntax:

{
  "operation": "timestamp",
  "timestamp[0]": {
    "inputFormat": "Mon Jan _2 15:04:05 -0700 2006",
    "outputFormat": "2006-01-02T15:04:05-0700"
  },
  "nowTimestamp": {
    "inputFormat": "$now",
    "outputFormat": "2006-01-02T15:04:05-0700"
  },
  "epochTimestamp": {
    "inputFormat": "2006-01-02T15:04:05-0700",
    "outputFormat": "$unix"
  }
}

but it should be

{
  "operation": "timestamp",
  "spec": {
    "timestamp[0]": {
      "inputFormat": "Mon Jan _2 15:04:05 -0700 2006",
      "outputFormat": "2006-01-02T15:04:05-0700"
    },
    "nowTimestamp": {
      "inputFormat": "$now",
      "outputFormat": "2006-01-02T15:04:05-0700"
    },
    "epochTimestamp": {
      "inputFormat": "2006-01-02T15:04:05-0700",
      "outputFormat": "$unix"
    }
  }
}

Add functionality to require source fields to be present in the input JSON. Otherwise throw an error.

Hi,

I am wondering if there's a way to create an array from existing keys using existing transform functions only ? something like

converting the following JSON

{ 
   "key1": "field1",
   "key2": "field2",
   "key3": "field3"
}

to something like:

{ 
   "arr": ["field1", "field2", "field3"]
}

Another followup would be, Is it possible to create an array at top level ? for example, converting the above JSON to something like:

["field1", "field2", "field3"]

Thanks

go get github.com/qntfy/kazaam/kazaam
# github.com/qntfy/kazaam/transform
../../../qntfy/kazaam/transform/uuid.go:33:11: assignment mismatch: 2 variables but 1 values

Due to change :

satori/go.uuid@0ef6afb

Be able to add a current timestamp to a message rather than just format an existing timestamp.

Great work here!
Is there a way to build strings (ala go fmt.Sprintf or other interpolation) in the shift operation? Something simpler that concat.

I'm looking for a mix of "value" fields (that preserve the selected values' type) and "interpolated string fields" (that are always strings and may be simple static strings).

Like this perhaps:

Transform (note the '):

{
  "outputField": [ "'some-prefix-", "input.value" ]
}

Input:

{
  "input": { "value": 47 }
}

Output:

{
  "outputField": "some-prefix-47"
}

A binary was checked into /kazaam. Remove it.

Id like to delete a key that might occur at multiple places in a json file, for which the paths are not known. Can I do this with kazaam? In other words id like wildcards in paths.

Supporting JSONPath-like syntax would be a great feature, or even something like https://github.com/tidwall/gjson.

Would you welcome a PR to this effect?

when executing in go version go1.9 linux/amd64
go get gopkg.in/qntfy/kazaam.v3
below output in command line

> # github.com/qntfy/kazaam/transform
> go-workspace/src/github.com/qntfy/kazaam/transform/uuid.go:33:18: multiple-value uuid.NewV4() in single-value context

For testing purposes, it would be nice to have a command-line tool for specifying a spec and running on some data.

'Extract' should allow specifying a json path and returning it as the top-level object.

The transformer name default to me implies that the key is set to the value only if no value is already set. Instead, as far as I can tell, right now it simply means "always set this value, no matter what".

I'm writing my own transformer to override this behaviour, but I figured I'd report this issue, in case you agree with me that the current behaviour is unexpected, based on the name.

Hi,

in the master branch of github.com/satori/go.uuid, uuid.NewV4() returns an error. In the pinned version v1.2.0, that function does not return an error. But since go get always pulls the master branch, github.com/qntfy/kazaam is not go get-able at the moment.

Here's the error:
../github.com/qntfy/kazaam/transform/uuid.go:33:18: multiple-value uuid.NewV4() in single-value contex

This problem was introduced with PR #76 .

As described in #77
spec:

{
  "operation": "shift",
  "spec": {
    "arr[+]": "key1",
    "arr[+]": "key2",
    "arr[+]": "key3"
  }
}

Input:

{ 
   "key1": "field1",
   "key2": "field2",
   "key3": "field3"
}

Expected output: {"arr":["field1", "field2", "field3"]}
Actual output: {"arr":["field3"]}

Hey qntfy Crew! Does your library have any support for JSONPath expression filtering?
For example:
Input

{
  "firstName": "John",
  "lastName" : "doe",
  "age"      : 26,
  "address"  : {
    "streetAddress": "naist street",
    "city"         : "Nara",
    "postalCode"   : "630-0192"
  },
  "phoneNumbers": [
    {
      "type"  : "iPhone",
      "number": "0123-4567-8888"
    },
    {
      "type"  : "home",
      "number": "0123-4567-8910"
    }
  ]
}

Path
phoneNumbers[?(@.type == "iPhone")].number
Result

[
  "0123-4567-8888"
]

Looking through the code briefly I don't see anything that would support filtering, so I think the answer is probably no. If the library doesn't support this syntax would this be a contribution that would be welcome? Thanks!

Hello I have source JSON with dots in keys like that:

{
   "dotted.key": "value"
}

Is there any way to access it ?

if you use $unix as an output format on a timestamp command, the value will be quoted. JSON integer values should not have any quotes.

Hi, I am straggling to define a spec to transform a json with an array inside an array. Here is the original json:
{
"contents": [
{
"id": 1,
"name": "aaa1",
"members": [
{
"member_id": 11,
"member_name": "bbb1"
},
{
"member_id": 12,
"member_name": "ccc1"
}
]
},
{
"id": 2,
"name": "aaa2",
"members": [
{
"member_id": 21,
"member_name": "bbb2"
},
{
"member_id": 22,
"member_name": "ccc2"
}
]
}
],
"meta": {
"meta1": "mmm1",
"meta2": "mmm2"
}
}

And here is the expected json:
{
"contents": [
{
"c_id": 1,
"c_name": "aaa1",
"members": [
{
"mem_id": 11,
"mem_name": "bbb1"
},
{
"mem_id": 12,
"mem_name": "ccc1"
}
]
},
{
"c_id": 2,
"c_name": "aaa2",
"members": [
{
"mem_id": 21,
"mem_name": "bbb2"
},
{
"mem_id": 22,
"mem_name": "ccc2"
}
]
}
],
"meta": {
"c_meta1": "mmm1",
"c_meta2": "mmm2"
}
}

Can any one help me with the spec? Thanks!

Support conversion of timestamp to and from unix epoch time as int64 string

spec: {
  "ts1": {
    "inputFormat": "$unix",
    "outputFormat": "2006-01-02T15:04:05-07:00"
  },
  "ts2": {
    "inputFormat": "2006-01-02T15:04:05-07:00",
    "outputFormat": "$unix"
  }
}