paesslerag / jsonpath Goto Github PK

When path doesn't begin with a $ Get doesn't return an error when the field doesn't exist, Get does return an error when path begins with $. See example below or use playground

package main

import (
	"encoding/json"
	"fmt"
	"github.com/PaesslerAG/jsonpath"
)

func main() {
	v := interface{}(nil)
	json.Unmarshal([]byte(`{
	"welcome":{
			"123456":["Good Morning", "Hello World!"]
		}
	}`), &v)

	if unknownWithoutPrefix, err := jsonpath.Get("nope", v); err != nil {
		fmt.Printf("Error: %s\n", err)
	} else if unknownWithoutPrefix == nil {
		fmt.Println("no value for missing field and no error")
	}

	if knownWithoutPrefix, err := jsonpath.Get("welcome", v); err == nil {
		fmt.Printf("welcome: %v\n", knownWithoutPrefix)
	}

	if unknownWithPrefix, err := jsonpath.Get("$.nope", v); err != nil {
		fmt.Printf("Error: %s\n", err)
	} else {
		fmt.Println(unknownWithPrefix)
	}
}

JSON:

{ "v3":[1,2,3,4,5,6,7]}

Path:
$.v3[2:3,5]
Expected:
[2,5]
Actual:
Error

Hi,

How would one go about to get support for the [@.length - n] syntax illustrated @ https://goessner.net/articles/JsonPath/

I know that [start:stop] using negative indices is an ok workaround if we have control over the input. Given that gval and jsonpath is meant for end-users to input business logic it would be nice to expose full jsonpath engine and not jsonpath minus some features.

	v := interface{}(nil)
	json.Unmarshal([]byte(`{
		"welcome":{
				"message":["Good Morning", "Hello World!"]
			}
		}`), &v)
	welcome, err := jsonpath.Get("$.welcome.message[(@.length -1)]", v)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	fmt.Println(welcome)

https://go.dev/play/p/tDVmCrg6_OJ

=>

parsing error: $.welcome.message[(@.length -1)]	:1:29 - 1:30 unexpected "-" while scanning parentheses expected ")"

PS. Awesome work with both gval and jsonpath; Amazing libraries!

I'm new to golang, so my question probably more stems from a lack of knowledge, but I'm getting there.

I have a small JSON sample that looks like the following:

{
    "FullOTAConfig": {
        "Licensing": {
            "isLicensed": true,
            "canTerminate": false
        },
        "BirdGlobal": {},
        "Tenants": {
            "Tenant1_Config": null,
            "Tenant2_Config": null,
            "Tenant3_Config": null
        }
    }
}

I am able to retrieve the JSON that I'm interested at path $.FullOTAConfig.Licensing via:

  // JSON map interface
  jsonInput map[string]interface{}
  
  // Load some JSON
  // ... 
  // ...
  
  // Get JSON at the requested path
  subJson, err := jsonpath.Get(jsonPath, jsonInput)
  if err != nil {
    fmt.Println(err)
    os.Exit(1)
  }
  
  // How do I update the JSON at path 'jsonPath'?
  // ... ???
  // jsonInput[?] = new_json_object

but what I would like to do is once the subobject is returned, I want to update the 'jsonInput' var but I don't know how I can use the jsonpath to then update the JSON in the map based on the path.

I think I should get the full paths and values when running this expression {#: $.a.b.c..[?(@.e == 1)]} on this json:

{
  "a": {
       "x": {
           "d":1
       },
       "b": {
           "c": {
               "da" : {
                   "d" : 2
                },
 "db" : {
                   "d" : 1
                },
            }
        }
  }
}

when running this I get this path $["db"] while I should get the full path which is $["a"]["b"]["c"]["db"]

I can run this expression {#: $..[?(@.e == 1)]} but I don't want to search the all tree and get also the $["a"]["x"] path, just what under a.b.c hierarchy
Is there a fix for that?

`v := interface{}(nil)

json.Unmarshal([]byte(`{
	"welcome":{
			"123456":["Good Morning", "Hello World!"]
		}
	}`), &v)

welcome, err := jsonpath.Get("$.welcome.123456[1]", v)`

A very interesting parser. I'm trying to use it to parse log files that contain objects at the top level where the object structure being logged is different from line to line. The log file is not one valid JSON object, but rather a series of valid JSON objects. I have logic that parses one object byte slice at a time, and unmarshals to map[string]interface{}.

But I haven't found a jsonpath syntax for testing the root object for the property of having a particular key. I believe I've looked through all the unit tests to better understand the syntax and saw no examples where the root was an object and the test was for the existence of a key. But there also wasn't anything in the documentation (that I found) to explain why this shouldn't be possible.

I could get things working if I read the entire file into memory and created an array out of them ([]interface{}) or if I wrapped each object in a single entry array but now I'm curious whether this should even be possible given the description of a jsonpath.

I've tried a number of things, most nonsensical anyway, but here are a few attempts with their corresponding errors.

package play_test

import (
	"encoding/json"
	"fmt"
	"strings"
	"testing"

	"github.com/PaesslerAG/jsonpath"
)

func TestPlay(t *testing.T) {
	for idx, c := range []struct{ expr, data, expect string }{
		{
			// This works. No surprise.
			expr:   `$.ip`,
			data:   `{"ip": "8.8.8.8", "action": "dns"}`,
			expect: `result: 8.8.8.8`,
		},
		{
			// This one doesn't. It is not legal to request a key that does not exist.
			expr:   `$.ip`,
			data:   `{"action": "dns"}`,
			expect: `error : unknown key ip`,
		},
		{
			// Same error.
			expr:   `($.ip)`,
			data:   `{"action": "dns"}`,
			expect: `error : unknown key ip`,
		},
		{
			// Other illegal syntax.
			expr: `?($.ip != "")`,
			data: `{"action": "dns"}`,
			expect: `error : parsing error: ?($.ip != "")	:1:1 - 1:2 unexpected "?" while scanning extensions`,
		},
		{
			// So how to test whether a key exists?
			// Here the syntax is legal but the result is an empty array.
			expr:   `$[?($.ip)]`,
			data:   `{"ip": "8.8.8.8, "action": "dns"}`,
			expect: `result: []`,
		},
		{
			// This doesn't parse correctly.
			expr: `("ip" in $)`,
			data: `{"ip": "8.8.8.8, "action": "dns"}`,
			expect: `error : parsing error: ("ip" in $)	:1:7 - 1:9 unexpected Ident while scanning parentheses expected ")"`,
		},
		{
			// This doesn't parse correctly either.
			expr: `$[?("ip" in $)]`,
			data: `{"ip": "8.8.8.8, "action": "dns"}`,
			expect: `error : parsing error: $[?("ip" in $)]	:1:10 - 1:12 unexpected Ident while scanning parentheses expected ")"`,
		},
	} {
		v := interface{}(nil)
		json.Unmarshal([]byte(c.data), &v)
		result, err := jsonpath.Get(c.expr, v)
		var buf strings.Builder
		if err == nil {
			fmt.Fprint(&buf, "result: ", result)
		} else {
			fmt.Fprint(&buf, "error : ", err)
		}
		got := buf.String()
		if c.expect != got {
			t.Errorf("idx %d failed comparison", idx)
			t.Log("expect:", c.expect)
			t.Log("got   :", got)
		}
	}

}

One official example of jsonpath uses jsonpath.PlaceholderExtension, but when we ran it and got an error "undefined: jsonpath.PlaceholderExtension", how to fix it?

Hi,

trying out the Get example from the Godocs:

v := interface{}(nil)

json.Unmarshal([]byte(`{
		"welcome":{
				"message":["Good Morning", "Hello World!"]
			}
		}`), &v)

welcome, err := jsonpath.Get("$welcome.message[1]", v)
if err != nil {
    fmt.Println(err)
    os.Exit(1)
}

fmt.Println(welcome)

// Output
// Hello World!

Produces the following error:

parsing error: $welcome.message[1]      :1:2 - 1:9 unexpected Ident while scanning operator

Am I missing something?

/Edit: I'm jusing Go 1.11 btw.
go version go1.11.2 darwin/amd64

Thanks

When accessing arrays like this:

arr["0"]

... I expect jsonpath to return an error. Array-indices are always integer and never strings. Instead, I get the first element. Is this by design? If so, I find it counter-intuitive.
Also, in combination to issue 2: #2, the following panics:

arr["-1"]

To handle my dependencies I'm using a different repo than my main code and using git submodules to handle my dependencies I'm trying to fetch both json path and gval repositories into my project but getting the following error:
cannot use evaluable (type “github.com/PaesslerAG/jsonpath/vendor/github.com/PaesslerAG/gval”.Evaluable) as type “github.com/PaesslerAG/gval”.Evaluable in field value

The only solution I found is deleting the vendor directory on the json path repo is there any other way around it?

Hey,

Is there a reason why GVal defaults to returning nil if the expression references an undefined data key, but JsonPath errors out?

https://github.com/PaesslerAG/gval/blob/cd026a3dee26df39acddc2292229512c4fa68756/evaluable.go#L133
vs

As an example,

	g := gval.NewLanguage(
		gval.Full(),
		jsonpath.Language())
	value, err := g.Evaluate(`$.data`, map[string]any{"data": "test"})
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(value)

Results in test; however, if the expression is $.invalidData then we error out:
can not evaluate $.invalidData: unknown key invalidData.

BUT if we just use standard Gval expressions, i.e:

	value, err := g.Evaluate(`invalidData`, map[string]any{"data": "test"})

we get nil as a return value.

The only reason I can think of is to have a distinction between null values and undefined values in JSON (null -> nil, undefined -> error), but we don't always want to error out if the key is undefined, it'd be nice if we could configure this default behavior. I might be missing something here so please let me know if this is the case!

Given the following json and jsonpath expression:

{
  "server": [
    {
      "key": "op_host",
      "props": {
          "isEncrypt": true
      }
    }
  ]
}

$.server[?(@.props.isEncrypt == true)].key

the following code can't achieve the desired result, but jsonpath.com can.

package main

import (
	"encoding/json"
	"fmt"
	"os"

	"github.com/PaesslerAG/jsonpath"
)

func main() {
	v := interface{}(nil)

	json.Unmarshal([]byte(`{
  "server": [
    {
      "key": "op_host",
      "props": {
          "isEncrypt": true
      }
    }
  ]
}`), &v)

	welcome, err := jsonpath.Get(`$.server[?(@.props.isEncrypt == true)].key`, &v)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	fmt.Println(welcome)
}

The JSONPath standard supports single-quotes for bracket notation (see here) but they fail with a syntax error:

m := map[string]interface{}{
  "field": map[string]interface{}{
    "sub-field": "hello",
  },
}
value, err := jsonpath.Get("$.field['sub-field']", m) 
// err is incorrectly set to `could not parse string: invalid syntax`

// the following case works correctly:
value, err := jsonpath.Get("$.field[\"sub-field\"]", m) 
// err is nil, value is "hello"

Instead I'd expect this to work correctly.

Currently, when trying to parse strings using single quotes ', the tool will return a syntax error. This is not according to the JSONPath standard (see here) and also not always what a user might expect, as expressed in #31, #23 and #19.

This seems to be because the tool uses Gval, which supports especially Go-like arbitrary expressions using go's text/scanner package.

Strings in Go are portrayed by double-quotes "..." and as
raw-strings `...` . Single-quoted-strings '...' are not known in Go. Instead, they are used to indicate single characters and runes. This means, when the tool encounters single-quotes strings, it will try to parse it as a single character and fail, throwing an error.

Since text/scanner is working correctly within Go context, it's unlikely that they will amend their library to allow single-quoted strings. As such, it's probably easier to modify the tools Gval implementation to allow its scanner to identify and parse single-quotes strings.

Addtionally, the strconv.Unquote functionality used by this tool also is based on Go, and as such will not accept single-quoted strings. As such, the parseString-method could be adjusted as well, so that it transforms single-quoted strings into double-quoted strings, allowing them to be parsed.

If these changes are made, jsonpath is able to handle single-quotes (although e.g. unit tests would've to be adjusted, since they already assume failure at single quotes). These changes could be made optional as to not disrupt potential workflows that already depend on jsonpath not evaluating single quoted strings.

Would this, in your opinion, be a sensible way to solve the issue?

Hello,
I use jsonpath in one of my projects. It works perfectly for most of my use-cases.
Nonetheless, I need to do something like this:

$..[? @.example ]

So, basically I want to get (recursively) all nodes that have a property named example.
Is that query possible in jsonpath?
I've tried a couple of similar queries and they didn't work for me - no results found.

I don't know if that's a feature, a bug or something that requires hacking GVal expressions?
Could you help me/point in a good direction?
Thanks in advance!

When using this library I noticed that single quotes do not work within a JSONPath, specifically I was evaluating a path like $.data.attributes[?(@.name=='title')].value which fails but changing to double quotes around title like $.data.attributes[?(@.name=="title")].value works fine. I'm not sure if the JSONPath spec addresses quoting at all but the evaluator at http://jsonpath.com/ handles the single quotes fine.

Hi.
Is there a way to construct a single jsonpath expression that has multiple condition parts.
If I have this JSON: {"a":1,"b":[{"x":2,"y":3},{"x":3,"y":2}]}
And want to check that (a==1) AND (b[0].x==2 OR b[1].y==2) returns true, is there a single jsonpath expression that evaluate that (Instead of making multiple expressions and run each of them sequentially with AND/OR by code) ?
Thanks.

Hi,
sorry for opening another issue so soon after the first one. I took a few examples off https://goessner.net/articles/JsonPath/index.html#e3 but can't get them to work.

Example code:

	v := interface{}(nil)

	json.Unmarshal([]byte(`{ "store": {
		"book": [ 
		  { "category": "reference",
			"author": "Nigel Rees",
			"title": "Sayings of the Century",
			"price": 8.95
		  },
		  { "category": "fiction",
			"author": "Evelyn Waugh",
			"title": "Sword of Honour",
			"price": 12.99
		  },
		  { "category": "fiction",
			"author": "Herman Melville",
			"title": "Moby Dick",
			"isbn": "0-553-21311-3",
			"price": 8.99
		  },
		  { "category": "fiction",
			"author": "J. R. R. Tolkien",
			"title": "The Lord of the Rings",
			"isbn": "0-395-19395-8",
			"price": 22.99
		  }
		],
		"bicycle": {
		  "color": "red",
		  "price": 19.95
		}
	  }
	}`), &v)

	result, err := jsonpath.Get("$..book[?(@.isbn)]", v)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	fmt.Println(result)

Will print an empty result.

Other results using the example expressions:

• $..book[-1:] works
• $..book[(@.length-1)] leads to parsing error: $..book[(@.length-1)] :1:18 - 1:19 unexpected "-" while scanning parentheses expected ")"
• $..book[?(@.price<10)] leads to parsing error: $..book[?(@.price<10)] :1:18 - 1:19 unexpected "<" while scanning parentheses expected ")"

#1 suggests that expressions containing [?(@.foo)] should work, but I can't reproduce that.

Does your syntax differ from https://goessner.net/articles/JsonPath/ ?

If I look at
{ "a": { "b": { "c": 1, "d": 2 } } }

and running this expression '$..c' I will get the value 1.
Is there a way to get c full path as well (like '$a.b.c') ?

I am trying to apply a filter and then pick the first element from the resulting set and just trying things wildly :)

JSONPath: $.items[?(@.available && @.id>1),0].id

parsing error: $.items[?(@.available && @.id>1),0].id	:1:33 - 1:34 mixed 63 and 44 in JSON bracket

I think the error wording is not helpful in this case, as I had to look into the ascii table to see that 64 is ? and 44 is ,, meaning that I mixed up the comma operator and filter thingy.

I found an issue with the return value of a query returning an array.
If I query for the array element itself it returns the array instead of an array that contains the value (which is also an array).

Example (based on https://jsonpath.com/)
JSON:

{
  "firstName": "John",
  "lastName" : "doe",
  "age"      : 26,
  "address"  : {
    "streetAddress": "naist street",
    "city"         : "Nara",
    "postalCode"   : "630-0192"
  },
  "phoneNumbers": [
    {
      "type"  : "iPhone",
      "number": "0123-4567-8888"
    },
    {
      "type"  : "home",
      "number": "0123-4567-8910"
    }
  ]
}

Query: $.phoneNumbers[0:1]
returns (correctly):

[
  {
    "type": "iPhone",
    "number": "0123-4567-8888"
  }
]

Query: $.phoneNumbers
returns:

  [
    {
      "type": "iPhone",
      "number": "0123-4567-8888"
    },
    {
      "type": "home",
      "number": "0123-4567-8910"
    }
  ]

instead of (see the outer array):

[
  [
    {
      "type": "iPhone",
      "number": "0123-4567-8888"
    },
    {
      "type": "home",
      "number": "0123-4567-8910"
    }
  ]
]

The worst part is that I have no way of knowing if I am expecting a single value or multiple values just by querying the returned type and without parsing the query itself.

Thanks.

When accessing an array like this:

arr[-1]

... I expect jsonpath to return an error. Instead it panics (range error).
In evaluable.go (line 105), only the array length (upper limit) is checked, but not if the index is negative.

As mentioned by @generikvault in #10 (comment), there is a "hack" for getting the path to the value(s) returned by jsonpath.Get(). This works great for wildcard paths but for absolute paths, the onus is on the jsonpath consumer to be able to accurately parse the provided JSONPath to be able to append to the provided path. Here is an example of some JSONPath values and the returned path using the aforementioned link:

JSONPath: $
  $
JSONPath: $.info
  $
JSONPath: $.paths[*]["get","put","post","delete","options","head","patch","trace"]
  $["/people/{personId}"]["delete"]
  $["/people"]["get"]
  $["/people"]["post"]
  $["/people/{personId}"]["get"]
  $["/people/{personId}"]["put"]
JSONPath: $.paths..parameters[*]
  $["/people"]["get"]["0"]
  $["/people"]["get"]["1"]
  $["/people/{personId}"]["0"]
JSONPath: $.components.parameters[*]
JSONPath: $.paths
  $
JSONPath: $.paths["/people"]
  $
JSONPath: $.paths["/people"].get.parameters[?(@.name == "pageSize")]
  $["0"]
JSONPath: $.paths["/people"].get.parameters[?(@.name == "pageToken")]
  $["1"]
JSONPath: $.paths["/people"].get.responses["200"].content["application/json"].schema.properties
  $

As you can tell, the path provided by the placeholder # hack is far from the actual path of the resolved value provided by jsonpath.Get(). Is there any possibility of extending the placeholder # hack to be more accurate for non-wildcard JSONPath values, or could there be an API exposed (or hack) that would make it where we can parse the provided JSONPath (consistently) so that we can add the proper suffix to the value provided by the placeholder # hack?

The following queries provide results that do not match those of other implementations of JSONPath
(compare https://cburgmer.github.io/json-path-comparison/):

• $.2
  Input:

  {"a": "first", "2": "second", "b": "third"}
  

  Expected output:

  ["second"]
  

  Error:

  parsing error: $.2	:1:2 - 1:4 unexpected Float while scanning operator
  

• $[-1]
  Input:

  ["first", "second", "third"]
  

  Expected output:

  ["third"]
  

  Error:

  index -1 out of bounds
  

• $[?(@.key=='value')]
  Input:

  [{"key": "some"}, {"key": "value"}]
  

  Expected output:

  [{"key": "value"}]
  

  Error:

  parsing error: $[?(@.key=='value')]	:1:12 - 1:19 could not parse string: invalid syntax
  

• $[?(@.key)]
  Input:

  [{"some": "some value"}, {"key": "value"}]
  

  Expected output:

  [{"key": "value"}]
  

  Actual output:

  []
  

• $['key']
  Input:

  {"key": "value"}
  

  Expected output:

  ["value"]
  

  Error:

  parsing error: $['key']	:1:3 - 1:8 could not parse string: invalid syntax
  

• $['one','three'].key
  Input:

  {"one": {"key": "value"}, "two": {"k": "v"}, "three": {"some": "more", "key": "other value"}}
  

  Expected output:

  ["value", "other value"]
  

  Error:

  parsing error: $['one','three'].key	:1:3 - 1:8 could not parse string: invalid syntax
  

• $['two.some']
  Input:

  {"one": {"key": "value"}, "two": {"some": "more", "key": "other value"}, "two.some": "42"}
  

  Expected output:

  ["42"]
  

  Error:

  parsing error: $['two.some']	:1:3 - 1:13 could not parse string: invalid syntax
  

• $['special:"chars']
  Input:

  {"special:\"chars": "value"}
  

  Expected output:

  ["value"]
  

  Error:

  parsing error: $['special:"chars']	:1:3 - 1:19 could not parse string: invalid syntax
  

For reference, the output was generated by the program in https://github.com/cburgmer/json-path-comparison/tree/master/implementations/Golang_github.com-PaesslerAG-jsonpath.

@cburgmer's JSONPath comparison project is currently discussing some issues relating to a proposed implementation of JSONPath known as "Proposal A". May I encourage you to get involved if you are interested in influencing the direction or, indeed, if you want to make your own proposals.

Similarly, please join us in slack (invitation) for informal discussions on the comparison project and potential JSONPath standardisation.

Hi, thanks for this great library.

I was hoping to do something like this.

$.items.length or $.items.length(). I get the following errors.

could not select value, invalid key: expected number but got length (string)

Is there another way to get the length?

Thanks

First of all, thank you for for this library!

Would you consider adding support for json.Number? More specifically, json.Decoder has the UseNumber() method which can be used to decode number values to json.Number instead of float64. Here is some sample code:

package main

import (
	"context"
	"encoding/json"
	"strings"

	"github.com/PaesslerAG/gval"
	"github.com/PaesslerAG/jsonpath"
	"github.com/generikvault/gvalstrings"
)

func main() {
	jsonPathLanguage := gval.Full(jsonpath.Language(), gvalstrings.SingleQuoted())

	eval, err := jsonPathLanguage.NewEvaluable(`$.body[?(@.type==1)]`)
	if err != nil {
		panic(err)
	}

	parsed := map[string]any{}
	d := json.NewDecoder(strings.NewReader(`{"body":[{"type":1,"id":"foo"},{"type":2,"id":"bar"}]}`))
	d.UseNumber()
	if err := d.Decode(&parsed); err != nil {
		panic(err)
	}

	x, err := eval(context.Background(), parsed)
	if err != nil {
		panic(err)
	}

	v, err := json.Marshal(x)
	if err != nil {
		panic(err)
	}

	println(string(v))
}

If you comment out d.UseNumber(), then the output will be [{"id":"foo","type":1}] instead of [].

Note: If I change $.body[?(@.type==1)] to $.body[?(@.type>1)], then the query does return [{"id":"bar","type":2}] when using d.UseNumber().

I haven't looked at the code in detail yet, but I can try to dig into it if you'd be interested in a PR.

IETF RFC 9535 defines a proposed standard for JSONPath. Other resources:

• A partial Compliance Test Suite.
• Behaviours of various implementations can be checked in the comparison project. Please note that the JSONPath Reference Implementation referenced in the comparison project is stalled/archived and incomplete.
• At the time of writing, online sandboxes for RFC 9535 are available in C# and in Rust.
• For more background and comments on the status of the CTS and RI, see this this post (also here in the IETF blog).
• Public discussion occurs in the IETF JSONPath Working Group's mailing list (archive).

(See here for updates to this list of resources.)

Valid approaches for this project are to (a) implement the RFC in whole or in part, (b) document its behaviour relative to the RFC, and (c) do nothing.

Hello,

I have this struct with the following JSON annotations:

MyFancyStruct struct {
	SomeNestedProperty []map[string]interface{} `json:"someNestedProperty,omitempty,nocopy"`
}

And this JSONPath:

$.someNestedProperty[*][*].some_other_prop

If I do the following:

JSONPathResults, err := jsonpath.Get("$.someNestedProperty[*][*].some_other_prop", myFancyStructInstance)

It won't find anything for obvious reasons, because it's searching for someNestedProperty on the struct with the first letter of the property uppercase.

🤔 Is it possible to enforce honoring JSON annotations? Please excuse my ignorance for not diving right into the code.

Hi, thank you for the library.

I am trying out to upgrade version from v0.1.0 to 0.1.1

Execution of go get -u github.com/PaesslerAG/jsonpath do nothing.

Manual change of the version in go.mod file to v0.1.1 and run project produces the error:

go: finding github.com/PaesslerAG/jsonpath v0.1.1
go: github.com/PaesslerAG/[email protected]: unknown revision v0.1.1
go: error loading module requirements

Usage of version string '0.1.1' without leading 'v' is forbidden in go.mod.

Am I missing something?

go version go1.11.5 linux/amd64

Thanks