Simplest explanation is via a code example comparing fastjson to the stdlib encoding/json:

package main

import (
	"encoding/hex"
	"encoding/json"
	"fmt"
	"github.com/valyala/fastjson"
)

func main() {
	raw := []byte(`"\ud83e\udd2d"`)

	var outDefault string
	err := json.Unmarshal(raw, &outDefault)
	if err != nil {
		panic(err)
	}

	var p fastjson.Parser
	v, err := p.ParseBytes(raw)
	if err != nil {
		panic(err)
	}

	outFastBytes, err := v.StringBytes()
	if err != nil {
		panic(err)
	}

	outFast := string(outFastBytes)

	fmt.Println(outDefault)
	fmt.Println(outFast)

	fmt.Println(hex.Dump([]byte(outDefault)))
	fmt.Println(hex.Dump(outFastBytes))
}

Which when run outputs:

🤭
��
00000000  f0 9f a4 ad                                       |....|

00000000  ef bf bd ef bf bd                                 |......|

The encoding/json output appears to be correct. It looks like the code difference is in UTF-16 surrogate pairs. The relevant snippit of code in encoding/json that handles this case is here: https://github.com/golang/go/blob/release-branch.go1.11/src/encoding/json/decode.go#L1286

I want to define a Parser only one time, and use it to parse different string several times. But it seems that the variables only are assigned with the latest parsing result, even I use different variable. My demo code as follows:

package main

import "fmt"
import "github.com/valyala/fastjson"

func main() {

        var p fastjson.Parser

        s0 := `{"hello": "world"}`
        s1 := `{"hello": "dlrow"}`

        v, _ := p.Parse(s0)
        v1, _ := p.Parse(s1)

        fmt.Printf("%v\n", v.Get("hello"))
        fmt.Printf("%v\n", v1.Get("hello"))
}

Hi. I like the looks of this library and I'm trying to work out from the API and examples how I can Parse an arbitrary blob of JSON into a map[string]interface{}. Is this possible?

For the development I am working on, I have to Parse JSON where the keys are unknown to the parsing code. The map is then given to a function pointer (essentially) which does know the keys.

If it is possible, I do not want to have to pass objects or structures from fastjson in order to keep dependencies at a minimum.

Thanks!

I can't find any API that convert Value to a map, just like gson does:

m, ok := gjson.Parse(json).Value().(map[string]interface{})

Currently, Parser.Parse and Parser.ParseBytes are making a copy of the argument and then modify it.
I have a case where I already pass a copy which can be modified directly. It would be nice to have an API for that.

Is there a way to fetch whatever the value is as an interface{} ("any") type? Essentially, something like: func (v *Value) GetInterface(keys ..string) interface{}.

My use case is pulling out individual values that will simply be serialized again as JSON later so I don't have any need to know what the underlying type it is.

I can implement it outside as a switch statement on .Type() (recursing into Arrays and Objects), but thought it might be useful to include as part of the library so that others don't need to de the same (and perhaps the library could do it more efficiently?).

If it's desirable, I can try to integrate whatever I come up with into the library as a PR.

The strconv.ParseInt in Golang std lib

• has syntax check and return an error if s is empty or contains invalid digits.
• has range check and return the the closest number among the range if overflow.

// The errors that ParseInt returns have concrete type *NumError
// and include err.Num = s. If s is empty or contains invalid
// digits, err.Err = ErrSyntax and the returned value is 0;
// if the value corresponding to s cannot be represented by a
// signed integer of the given size, err.Err = ErrRange and the
// returned value is the maximum magnitude integer of the
// appropriate bitSize and sign.
func ParseInt(s string, base int, bitSize int) (i int64, err error) {

I need these but fastjson.ParseInt64BestEffort behaves differently.

Hi!
I'm coping with OOMs in wal-g/wal-g#738
And looking for good JSON lib so that we could avoid materializing JSON in memory. But the code is shared among many different types of JSON, so I do not want to use precompiled JSON encoder and want reflective one.
I see that fastjson is mostly about parsing. But maybe you have some ideas where to look at?

I was just wondering why the type returned for boolean is eithe true or false. It's not really in line with the data types for JSON, is it? Why doesn't it return boolean in that case?

This library looks great, and I love the simple API. I considered using this + fasttemplate to transform JSON messages.

fasttemplate supports passing a byte slice as the value to be used for a template tag, but I don't see any way to get the byte slice of a value using this library. I mostly don't care about its type, I just want the underlying value at a given path, and be able to either 1) transform those bytes myself, or 2) pass those bytes as-is along to fasttemplate to put in a new JSON structure.

Right now I'm using jsonparser to do exactly this, by using jsonparser.EachKey, which returns the value as bytes, but it has some extra overhead compared to fastjson.

Would adding a Value.Bytes() function be something to consider here? Or is there another way you'd propose tackling this problem without sacrificing performance?

edit just to make sure: I'm aware of fastjson.GetBytes, but I need to fetch 40+ values in the JSON payload, so this is significantly slower than Parser.ParseByte combined with something like Value.GetBytes

Is this the correct way to loop over a json array?
Or is there a more efficient way?

	values, err := json.Array()
	if err != nil {
		fmt.Println(err)
	}
	for i := 1; i < len(values); i++ {
		fmt.Println(values[i].GetInt("id"))
	}

Commit 619fb6c adds a limit of 300 to the depth of nested objects to the parser.

Would it be possible to make that limit configurable? I need to parse a JSON object that has more (around 2000).
Making MaxDepth a var would suffice.

I found that floats in exponential notation with a point after the last digit of the coefficient do not parse correctly, meanwhile strconv.ParseFloat(s, 64) do the job.

Passed:

// Fractional + exponent part
f("1e4", 1.e4)
f("-1E-10", -1.e-10)

Failed:

// Fractional + exponent part
f("1.e4", 1.e4)
f("-1.E-10", -1.e-10)

ParseFloat test case

n, _ := strconv.ParseFloat("11.e4", 64)
if n != 11.e4 {
	t.Fatalf("Unexpected result from \"11.e4\" got: %v, want: %v", n, 11.e4)
}
n, _ = strconv.ParseFloat("-11.E-10", 64)
if n != -11.e-10 {
	t.Fatalf("Unexpected result from \"-11.E-10\" got: %v, want: %v", n, -11.E-10)
}

@valyala hi, How can i parse a json string which looks like that? what is the fastest way to parse it? I data is uint64
id is string
debug is uint64
temp is uint64

s := "{\"data\":\"sampleData\", \"id\":\"data_12344\", \"debug\":10000, \"temp\":123213}"

example code:

jsonObj:=fastjson.MustParse(`{"key":"value"}`)
for key,value := range jsonObj.toMap() {
// .....
}

When calling Float64(), if the json value is 0.0 it fails to parse. When the value is 0 however it works.

It looks like the issue is this line here:

That line checks for the literal value of "0", but not for "0.0". Semantically 0.00 should also behave the same. Perhaps fastfloat.ParseBestEffort should return an explicit error that can be checked rather than relying on the sentinel value of 0?

@valyala what do you think about adding more complex validation primitives like expected object keys, expected key value types ? It would be also great to be able to add some custom registrable validation func's - min/max for numbers, minLength / maxLength / regex for strings etc.

Basic stuff for a fail-fast strategy.

func TestMarshalTo(t *testing.T) {
	var p fastjson.Parser
	v, err := p.Parse(`{
			"str": "bar",
			"int": 123,
			"float": 1.23,
			"bool": true,
			"arr": [1, "foo", {}]
	}`)
	if err != nil {
		log.Fatal(err)
	}

	out := make([]byte, 0, 50000)
	log.Println(string(v.MarshalTo(out)))
	log.Println(string(out))
}

2019/01/24 12:15:45: {"str":"bar","int":123,"float":1.23,"bool":true,"arr":[1,"foo",{}]}
2019/01/24 12:15:45:

You are returning a newly appended slice instead of attempting to append to the passed in slice. Passed in slice needs to be *[]byte for it to work as I think you have intended.

Or you can just copy() instead of append as append will cause allocations if the slice isn't large enough.

This code:

jp := new(fastjson.Parser)
parsed, _ := jp.Parse(`{"a": 8838840643388017390}`)
fmt.Printf("AS INT = %d, AS FLOAT = %f\n", parsed.GetInt("a"), parsed.GetFloat64("a"))

produces this output:

AS INT = 8838840643388017664, AS FLOAT = 8838840643388017664.000000

Which is clearly wrong.

Could you please include jsoniter in your benchmark?

Some parts of the new Arena API that I find hard to use:

• Object only provides O(n) implementation of Get/Set, which is useful in most cases, but it also makes it impossible to create very large objects. For faster Set it might make sense to add another method called Append which skips checking for equal keys and append to kvs directly. For Get I see two options: create a map[string]*Value on first Get and use the map on subsequent Get calls; or add a ToMap convenience function that converts the Object to a map[string]*Value for faster gets.
• The only way to create an Array is to call Arena.NewArray(), and call SetArrayItem for each index. This requires the user to maintain array length by himself, and not be able to use the more idiomatic append(). The API also prevents shrinking arrays or taking a slice from another []*Value directly. Consider adding a []*Value parameter to NewArray (breaking API change), or adding a Replace method that replaces everything in the array.
• Commit c188f76 might cause memory leaks. For example, when a very large array is reused as an object, the array is kept in memory. In fact, this was the behavior before the commit as well. Consider emptying out everything, including slice pointers.

BTW, might not be related to this, but I don't think optimizations like fb293df is doing anything since there is only one memory read anyway, and CPU is clever enough to execute them in parallel.

The addition of Value.MarshalTo to fastjson opens the door for supporting transformation.

What do you think about supporting the ability to alter Object field names or even removing them or replacing their values?

With given test

func TestFastjsonBug(t *testing.T) {
	str := "192.168.10.1"
	v, err := fastjson.Parse(str)
	require.NoError(t, err) // EXPECT error here. 

	vStr := v.MarshalTo(nil)
	err = fastjson.ValidateBytes(vStr)
	require.NoError(t, err) // FAIL: unexpected tail: ".10.1"
}

Expect to get a error during parse.
But actually parse gives Value with an invalid json representation

This bit of invalid json causes Validate to hang:

https://github.com/nst/JSONTestSuite/blob/master/test_parsing/n_structure_100000_opening_arrays.json

Hi,

i have the following, how can i add a key/value to the json?

// mapping.data is []*fastjson.Value

for _, val := range mapping.data {
   settings := getSettings(val.Get("Type"))

   val.Set("ID", fastjson.MustParse(settings.ID))
}

i currently get: panic: cannot parse JSON: cannot parse empty string; unparsed tail: ""

Hello,

fastJSON is used by an ArarangoDB client driver ArangoDB-NET.
An issue has been opened there yojimbo87/ArangoDB-NET#53 because apparently there is a blocking problem in fastJSON serialization.

Here is the problem i paste here :

Saved to restsharp, i retrieve some json data from a rest api source.
I do my stuff with the related objects then i want to store into arango.
For that i follow the doc https://github.com/yojimbo87/dictator#convert-strongly-typed-object-to-document.

And it works for very little objects as mostly i got this error :
"Serializer encountered maximum depth of 20"
I don't know what is this maximum depth but i imagined 20 is not big either so i changed this setting;

    ASettings.JsonParameters.SerializerMaxDepth = 255;

Apparently it's not enough as i get "Serializer encountered maximum depth of 255".

I looked at some objects, some were not so little so i loop them to do the same thing on each of their element instead of the whole structure but still the same error.

This drives me to think that "Convert strongly typed object to document" is OK but with very little object then...

Hi,
Thanks for a great library. It seems like i've found an edge case where unescapeStringBestEffort(), which is called from .Type() on a value ends up changing the value of the field. Meaning that simply calling .Type() results in a change of the data which is unexpected.

So it's two issues really. One is that unescapeStringBestEffort() have issues unescaping some strings. I've included an example json below. In my testing it seems to be some emojis followed by a line break.

The second and bigger issue is that, unescapeStringBestEffort() is called when checking type of a value. Resulting in simply checking the Type changes the data which seems like unwanted behaviour as it brings about quite unexpected issues.

Steps to reproduce:

Json affected:
{"caption":"󠁧󠁢󠁥󠁮󠁧󠁿🏴󠁧󠁢\n"}

Code:
var p fastjson.Parser strBytes := []byte({"caption":"󠁧󠁢󠁥󠁮󠁧󠁿🏴󠁧󠁢\n"}`)

parser, _ := p.ParseBytes(strBytes)
ob, _ := parser.Object()

// Marshals correctly: {"caption":"󠁧󠁢󠁥󠁮󠁧󠁿🏴󠁧󠁢\n"}
fmt.Println(string(ob.MarshalTo(nil)))

ob.Visit(func(key []byte, v *fastjson.Value) {
v.Type()
})

// Marshals incorrectly:{"caption":"\U000e0067\U000e0062\U000e0065\U000e006e\U000e0067\U000e007f🏴\U000e0067\U000e0062\n"}

fmt.Println(string(ob.MarshalTo(nil)))
`

This error wont happen if you comment out the .Type() in the visit loop or take out line 660 and 661 in parser.go.

I'm trying to figure out what the idiomatic way of migrating from using json.NewDecoder(<bytes>).Decode(<interface>).

I'm not sure what the correct way of using this library like I would by adding UnmarshalJSON to each struct that I have specific instructions for.

Is there a fully fledged example of doing deep json unmarshalling?

Version:
4864f0c

BugStrace:

panic: runtime error: index out of range [recovered]
 
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7a4933]

goroutine 1182 [running]:
MYGOPATH/src/my_package.Write.func1(0xc000723400, 0xc01968fed0)
	MYGOPATH/src/my_package.go:106 +0xa3
panic(0xc2f480, 0x15a78d0)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/valyala/fastjson.(*Object).unescapeKeys(0xc000124000)
	MYGOPATH/src/github.com/valyala/fastjson/parser.go:488 +0xec
github.com/valyala/fastjson.(*Object).Get(0xc000124000, 0xc00072e5c0, 0x5, 0xc019d86020)
	MYGOPATH/src/github.com/valyala/fastjson/parser.go:515 +0x3e
github.com/valyala/fastjson.(*Value).Get(0xc000124000, 0xc01968fbb0, 0x1, 0x1, 0x10)
	MYGOPATH/src/github.com/valyala/fastjson/parser.go:686 +0x84
github.com/valyala/fastjson.(*Value).GetStringBytes(0xc000124000, 0xc01968fbb0, 0x1, 0x1, 0x41fd08, 0x10, 0xbe4980)
	MYGOPATH/src/github.com/valyala/fastjson/parser.go:816 +0x4d

var unmarshaller fastjson.Parser
jsonString := `[{"update_date":null,"reading_time_minutes":{"bytes.decimal":"\u0014"},"ID":42,"title":"lorem ipsum","body":"lorem ipsum etc...","content_type":null,"post_date":1239321600}]`
value, err := unmarshaller.Parse(jsonString)
if err != nil {
	panic(err)
}
fmt.Println(value.String())
values, err := value.Array()
if err != nil {
	panic(err)
}
fmt.Println(values[0].String())

prints the following:

[{"title":"lorem ipsum","body":"lorem ipsum etc...","content_type":null,"post_date":1239321600,"update_date":null,"reading_time_minutes":{"bytes.decimal":"\u0014"},"ID":42}]
{"updat":"ate":null,"","ding":"me_minutes":{"byte","ecimal":"\u0":null,"D":42,"ti":42,"sum","body"":null,"ipsum etc...","conte":"lorem ipsum","32":1239321600}

the Array function seems to be fishy.

When Fastjson parses timestamp wrapped with text format, for example, "1570636800000", it doesn't parse this timestamp because the timestamp has quote. If we use replaceAll to remove quote then become 1570636800000, Fastjson is able to parse it.
This issue annoys to RequestBody Object to receive JSON from others.
Can it be fixed?

I'm working with an endpoint that is subject to dynamic updates in key and value pairs. What is the most efficient way to parse a nested json object in a json array without knowing the exact key, but knowing potential value characters? I had in mind to iterate over values and simply use switch / if clauses to match contained characters by key value until reaching the nested object. Is that the ideal solution?

 package main

 import (

    "log"
    "fmt"
    "github.com/valyala/fastjson"

 )

 func main() {
    var p fastjson.Parser
    v, err := p.Parse(`{
            "str": "bar",
            "int": 123,
            "float": 1.23,
            "bool": true,
            "arr": [1, "foo", {}],
            "a": { "c":1, "d":"foo", "e":"e" }
         }`)
         if err != nil {
                 log.Fatal(err)
         }
 //      fmt.Printf("foo=%s\n", v.GetStringBytes("str"))
 //      fmt.Printf("int=%d\n", v.GetInt("int"))
 //      fmt.Printf("float=%f\n", v.GetFloat64("float"))
 //      fmt.Printf("bool=%v\n", v.GetBool("bool"))
 //      fmt.Printf("arr.1=%s\n", v.GetStringBytes("a", "c"))
 //        fmt.Printf("arr.1=%s\n", v.GetStringBytes("a", 0))
        fmt.Printf("arr.1=%s\n", v.GetStringBytes("a"))
}

How can I get "a" value only to have :
{ "c":1, "d":"foo", "e":"e" }

This is the only feature missing from fastjson I need.

The following program:

package main
import (
	"fmt"
	"github.com/valyala/fastjson"
)
func main() {
	b := make([]byte, 16384)
	var parser fastjson.Parser
	_, err := parser.ParseBytes(b)
	fmt.Println(err)
}

prints:

cannot parse JSON: cannot parse number: unexpected char: "\x00"; unparsed tail: "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 [...65472 more bytes...]

All the unparsed data is echoed in the error message. This is a problem for debugging and logging.

By contrast, the following program:

package main
import (
	"encoding/json"
	"fmt"
)
func main() {
	b := make([]byte, 16384)
	err := json.Unmarshal(b, &struct{}{})
	fmt.Println(err)
}

only prints:

invalid character '\x00' looking for beginning of value

Hello,

Why don't having a Uint64 convert function? There are any problem to implement it? I see that gjson implemented it

Thanks.

HI,

fastjson seems very good library which I would love to have in my project. In one of my use cases, I want to convert a value struct to my defined struct. Is there any method available in fastjson for that?

one issue , like this :

package main

import (
	"fmt"

	"github.com/valyala/fastjson"
)

func main() {
	postByte := []byte(`{"name":"string","next":{"key":"value"}}`)
	var p fastjson.Parser
	v, _ := p.ParseBytes(postByte)

	name := v.Get("name").String()
	name_should_be := "string"
	fmt.Println(name, name_should_be)
	value := v.Get("next").Get("key").String()
	value_should_be := "value"
	fmt.Println(value, value_should_be)
}

go run main.go

"string" string
"value" value

it's wrong result of fastjson in .String() func .

The JSON could contains >200MB and memory usage was 3GB+, how do you handle large JSON efficiently and access value on 2nd level depth?

f, err := ioutil.ReadFile("1.json")
        if err != nil {
                panic(fmt.Errorf("cannot read: %s", err))
        }

        var p fastjson.Parser
        v, err := p.Parse(string(f))
        if err != nil {
                fmt.Println(err)
        }

readme tells golang1.12 used, so i am not clear

Could we get a function to just return the string type instead of a byte slice?

I often just need the string and noticed GetStringByte is actually converting the underlying string to a byte.

func (v *Value) GetString(keys ...string) string {
  v = v.Get(keys...)
  if v == nil || v.Type() != TypeString {
    return ""
  }
  return v.s
}

Some float64 numbers are parsed differently than when using encoding/json or easyjson.

Here is an example:

package main

import (
	"encoding/json"
	"fmt"

	"github.com/valyala/fastjson"
)

type T struct {
	F float64
}

func main() {
	input := []byte(`{"F": 0.3}`)
	t := &T{}
	json.Unmarshal(input, t)
	fmt.Println("encoding/json:", t.F)

	t = &T{}
	var p fastjson.Parser
	v, _ := p.ParseBytes(input)
	fv := v.Get("F")
	f, _ := fv.Float64()
	fmt.Println("fastjson:     ", f)
}

Output:

encoding/json: 0.3
fastjson:      0.30000000000000004

I know fastjson uses a fast custom implementation of the float64 parser; however, I was just wondering if there is any possibility of getting this in line with how strconv.ParseFloat works.

Go version 1.11 includes cl47152 and cl98440. The changes when compared to Go version 1.10 are significant.

benchmark                                          old ns/op     new ns/op     delta
BenchmarkParse/small/stdjson-map-12                1344          1248          -7.14%
BenchmarkParse/small/stdjson-map-12                1336          1249          -6.51%
BenchmarkParse/small/stdjson-map-12                1327          1255          -5.43%
BenchmarkParse/small/stdjson-map-12                1334          1268          -4.95%
BenchmarkParse/small/stdjson-map-12                1387          1329          -4.18%
BenchmarkParse/small/stdjson-struct-12             1028          637           -38.04%
BenchmarkParse/small/stdjson-struct-12             1040          632           -39.23%
BenchmarkParse/small/stdjson-struct-12             973           626           -35.66%
BenchmarkParse/small/stdjson-struct-12             1032          645           -37.50%
BenchmarkParse/small/stdjson-struct-12             1020          646           -36.67%
BenchmarkParse/small/stdjson-empty-struct-12       874           440           -49.66%
BenchmarkParse/small/stdjson-empty-struct-12       867           438           -49.48%
BenchmarkParse/small/stdjson-empty-struct-12       872           420           -51.83%
BenchmarkParse/small/stdjson-empty-struct-12       874           439           -49.77%
BenchmarkParse/small/stdjson-empty-struct-12       885           434           -50.96%

benchmark                                          old allocs     new allocs     delta
BenchmarkParse/small/stdjson-map-12                55             51             -7.27
%
BenchmarkParse/small/stdjson-map-12                55             51             -7.27
%
BenchmarkParse/small/stdjson-map-12                55             51             -7.27
%
BenchmarkParse/small/stdjson-map-12                55             51             -7.27
%
BenchmarkParse/small/stdjson-map-12                55             51             -7.27%
BenchmarkParse/small/stdjson-struct-12             40             4              -90.00%
BenchmarkParse/small/stdjson-struct-12             40             4              -90.00%
BenchmarkParse/small/stdjson-struct-12             40             4              -90.00%
BenchmarkParse/small/stdjson-struct-12             40             4              -90.00%
BenchmarkParse/small/stdjson-struct-12             40             4              -90.00%
BenchmarkParse/small/stdjson-empty-struct-12       50             2              -96.00%
BenchmarkParse/small/stdjson-empty-struct-12       50             2              -96.00%
BenchmarkParse/small/stdjson-empty-struct-12       50             2              -96.00%
BenchmarkParse/small/stdjson-empty-struct-12       50             2              -96.00%
BenchmarkParse/small/stdjson-empty-struct-12       50             2              -96.00%

Hey! Thanks for such a great JSON framework. I am happy to use!

We could provide a string field in *Value struct or a function that called like: parentKey or GetParentKey() string

Why we should need like these field or function is that actually is we can not get parent key's value from in *Value context.

func get(v *fastjson.Value) {
 var key string = v.GetParentKey()
}

We must use Visit function in order to get key's value, which is not useful due to we must scan all of JSON's fields.

func get(v *fastjson.Value) {
 v.Visit(func(key []byte, v *fastjson.Value) {
  var key string = string(key)
 })
}

A few GetParentKey() examples:

{
  "a": "foo",
  "b": "bar",
  "c": {
    "d": "baz",
    "e": 0
  }
}

• If Value is at the top level (which means there is no parent key), it returns nil or "".
• If Value is at inside c key, it returns "c"

Thanks!

It is a bit scary to use parsing package that does not contain fuzzing tests.

json-iterator also claims to be faster and is a complete drop in replacement for encoding/json . https://github.com/json-iterator/go. Is it faster than this package?

Hello,

I have been using fastjson for some time now and it really is very excellent.

Now I have hit a very specific issue that I am struggling to resolve and I would be most grateful for any helpful suggestions.

I have a .json file structured in this way (inserting this as a quote as code formatting was messed up):

{
"id": 1,
"display": 1,
"sync_state": 0,
"name": "Test",
"type": "Type",
"version": "v1",
"social_fields":[
{"type": "social",
"label": "Phone",
"field_value":"123456789"
},
{"type": "social",
"label": "Facebook",
"field_value":"www.facebook.com"
},
{"field_type": "social",
"label": "Twitter",
"ffield_value":"www.twitter.com"
},
{"field_type": "social",
"label": "Pinterest",
"field_value":"www.pinterest.com"
}
],
"col1_heading": "Column Heading",
"column_1_fields":[
{"field_type": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
},
{"field_name": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
}
],
"col2_heading": "Column Heading",
"column_2_fields":[
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
},
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
}
],
"col3_heading": "Column Heading",
"column_3_fields":[
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
},
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
}
]
}

This .json file is manipulated via golang based on a form submission with field names identical to the ones you see here.

So, here is the function which ingests this file and updates it (error handling and print statements omitted for brevity):

func processJSON(form map[string][]string, jsonContent *fastjson.Value) (*fastjson.Value, error) {
// first, delete the sections of json I want to update (it is easier to just delete the whole section, then start again, I think)
jsonContent.Del("social_fields")
jsonContent.Del("column_1_fields")
jsonContent.Del("column_2_fields")
jsonContent.Del("column_3_fields")

//second, make four arenas (as it seems arenas share memory somehow, so reusing the same arena won't work in the scenario below)
var arena1 fastjson.Arena
var arena2 fastjson.Arena
var arena3 fastjson.Arena
var arena4 fastjson.Arena

//third, make four separate fastjson values, so that we can call arena.Reset() after each for/range loop, while preserving what was stored in the arrays
var fjvalue1 *fastjson.Value
var fjvalue2 *fastjson.Value
var fjvalue3 *fastjson.Value
var fjvalue4 *fastjson.Value

//two counters to be iterated, due to the difference in field structure for fields in column_2
j := 0
i := 0

//four arrays, one for each field type, all using unique arenas due to previously mentioned memory allocation
social_field_array := arena1.NewArray()
column_1_field_array := arena2.NewArray()
column_2_field_array := arena3.NewArray()
column_3_field_array := arena4.NewArray()

//iterate over all fields submitted from the form (the form contains a number of inputs all named 'field_type', to make it possible to iterate over them all in one go

for _, field_type := range form["field_type"] {
if field_type == "social" {
// create a unique fastjson.Parser to avoid memory allocation issues
var p fastjson.Parser
v, err := p.Parse({ "field_type":" + field_type + ", "label":" + jsonEscape(form["field_label"][i]) + ", "find_us_field_value":" + jsonEscape(form["field_value"][i]) + " })
if err != nil {
//handle error
}
// add the parsed json to the array we created earlier, using the index i
social_field_array.SetArrayItem(i, v)
i += 1
}
}
//once the loop is done, copy this value into a new variable and reset the arena to avoid memory allocation issues
fjvalue1 = social_field_array
arena1.Reset()
for _, field_type := range form["field_type"] {
if field_type == "column_1" {
// here is another new parser again
var q fastjson.Parser
v, err := q.Parse({"field_type":" + field_type + ", "placeholder":"First Column Info", "field_value":" + jsonEscape(form["field_value"][i]) + "})
if err != nil {
//handle error
}
i += 1
}
}
// same as above
fjvalue2 = column_1_field_array
arena2.Reset()
for _, field_type := range form["field_type"] {
if field_type == "column_2" {
var r fastjson.Parser
v, err := r.Parse({ "field_type":" + field_type + ", "placeholder_1":"Second Column Info:", "field_value_1":" + jsonEscape(form["field_value_1"][j]) + ", "placeholder_2":"Second Column Info:", "field_value_2":" + jsonEscape(form["field_value_2"][j]) + " })
if err != nil {
//handle error
}
j += 1
}
}
fjvalue3 = column_2_field_array
arena3.Reset()

//technically there is another loop in here, but I am omitting this for brevity, as it does the same thing again

}
fjvalue4 = column_3_field_array
arena4.Reset()
jsonContent.Set("social_fields", fjvalue1)
jsonContent.Set("column_1_fields", fjvalue2)
jsonContent.Set("column_2_fields", fjvalue3)
jsonContent.Set("column_3_fields", fjvalue4)
return jsonContent, nil
}

Now, this approach generates a .json file that looks a bit like this:

{
"id": 1,
"display": 1,
"sync_state": 0,
"name": "Test",
"type": "Type",
"version": "v1",
"social_fields":[
{"type": "social",
"label": "Phone",
"field_value":"123456789"
},
{"type": "social",
"label": "Facebook",
"field_value":"www.facebook.com"
},
{"field_type": "social",
"label": "Twitter",
"ffield_value":"www.twitter.com"
},
{"field_type": "social",
"label": "Pinterest",
"field_value":"www.pinterest.com"
}
],
"col1_heading": "Column Heading",
"column_1_fields":[null, null, null, null,
{"field_type": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
},
{"field_name": "column_1",
"placeholder": "First Column Info",
"field_value":"First Column Info"
}
],
"col2_heading": "Column Heading",
"column_2_fields":[
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
},
{"field_type": "column_2",
"description_1": "Second Column Info:",
"find_us_field_value_1":"Second Column Info",
"description_2": "Second Column Info:",
"find_us_field_value_2":"Second Column Info"
}
],
"col3_heading": "Column Heading",
"column_3_fields":[ null, null, null, null, null, null, null,
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
},
{"field_type": "column_3",
"placeholder": "Third Column Info",
"field_value":"Third Column Info"
}
]
}

It would appear that no matter what I do, due to the looping I am doing, there are always these "null" fields inserted (see just above this "column_3_fields"), which, for the first column data coincidentally always seems to be just as many as there were social fields submitted.

It is worth noting that you are not looking at the first iteration of this issue. I originally had all of this packed into a single for loop, with a single arena, and arena.reset calls, trying to reuse the same parser and arena for efficiency. I experimented with different ways of trying to 'convince' fastjson to not create these additional "null" values, but by now I am sadly out of ideas.

I have used the methodology of deleting a whole section and then just adding it back in by parsing pre-written json strings in golang and appending them to a new array created from an arena via value.SetArrayItem(), which is then re-appended to the .json file via value.Set("arrayname", value) before, but it was always fine, as it only ever concerned just one array and not multiple arrays within the same .json file.

It is obvious that the above cannot be the correct approach, but I am illustrating it here to show you where I am at with my mental model of things.

You got any tips for me? :)

Hey,
Not sure this is a bug or by design, I didn't find anything on the docs mention that and I think it made my app OOM

This is the implementation of arena pool:

// Get returns an Arena from ap.
//
// The Arena must be Put to ap after use.
func (ap *ArenaPool) Get() *Arena {
	v := ap.pool.Get()
	if v == nil {
		return &Arena{}
	}
	return v.(*Arena)
}

// Put returns a to ap.
//
// a and objects created by a cannot be used after a is put into ap.
func (ap *ArenaPool) Put(a *Arena) {
	ap.pool.Put(a)
}

However (AFAIK), pools usually should call reset on returned object
so shouldn't Put implementation should be:

a.Reset()
ap.pool.Put(a)

I can call .Reset myself but since the pool is not doing that, I'm afraid that I might be missing something?

Would really appreciate if you have time to look,
Thanks!

I'm impressed with the performance of Validate, it comes close to my own special validating implementation (note: mine was slower until very recently)! However, it is overly permissive. Attached is a patch taken from test cases from my own package. Feel free to keep all or none of it or pick cases to keep as appropriate (I did see TestValidate in handy_test.go).

--- FAIL: TestValid (0.00s)
	parser_test.go:172: #12: "\"\\uz\"" got valid? true, exp? false
	parser_test.go:172: #15: " \"f\x00o\"" got valid? true, exp? false
	parser_test.go:172: #17: " \"\\uazaa\" " got valid? true, exp? false
	parser_test.go:172: #31: "-" got valid? true, exp? false
	parser_test.go:172: #32: "1e" got valid? true, exp? false
	parser_test.go:172: #33: "1e+" got valid? true, exp? false
	parser_test.go:172: #34: " 03e+1 " got valid? true, exp? false
	parser_test.go:172: #35: " 1e.1 " got valid? true, exp? false
	parser_test.go:172: #36: " 00 " got valid? true, exp? false
	parser_test.go:172: #37: "1.e3" got valid? true, exp? false
	parser_test.go:172: #38: "01e+6" got valid? true, exp? false
	parser_test.go:172: #39: "-0.01e+0.6" got valid? true, exp? false

diff --git a/parser_test.go b/parser_test.go
index f8ef01f..331107f 100644
--- a/parser_test.go
+++ b/parser_test.go
@@ -1,6 +1,7 @@
 package fastjson
 
 import (
+	"encoding/json"
 	"fmt"
 	"strings"
 	"testing"
@@ -76,6 +77,103 @@ func TestUnescapeStringBestEffort(t *testing.T) {
 	})
 }
 
+func TestValid(t *testing.T) {
+	var tests = []string{
+		"",
+		"   ",
+		" z",
+		" 1  1",
+		" 1  {}",
+		" 1  []",
+		" 1  true",
+		" 1  null",
+		" 1  \"n\"",
+
+		// string
+		`"foo"`,
+		"\"\xe2\x80\xa8\xe2\x80\xa9\"", // line-sep and paragraph-sep
+		` "\uaaaa" `,
+		`"\uz"`,
+		` "\`,
+		` "\z`,
+		" \"f\x00o\"",
+		` "foo`,
+		` "\uazaa" `,
+
+		// number
+		"1",
+		"  0 ",
+		" 0e1 ",
+		" 0e+0 ",
+		" -0e+0 ",
+		"-0",
+		"1e6",
+		"1e+6",
+		"-1e+6",
+		"-0e+6",
+		" -103e+1 ",
+		"-0.01e+006",
+		"-z",
+		"-",
+		"1e",
+		"1e+",
+		" 03e+1 ",
+		" 1e.1 ",
+		" 00 ",
+		"1.e3",
+		"01e+6",
+		"-0.01e+0.6",
+
+		// object
+		"{}",
+		`{"foo": 3}`,
+		` {}    `,
+		strings.Repeat(`{"f":`, 1000) + "{}" + strings.Repeat("}", 1000),
+		`{"foo": [{"":3, "4": "3"}, 4, {}], "t_wo": 1}`,
+		` {"foo": 2,"fudge}`,
+		`{{"foo": }}`,
+		`{{"foo": [{"":3, 4: "3"}, 4, "5": {4}]}, "t_wo": 1}`,
+		"{",
+		`{"foo"`,
+		`{"foo",f}`,
+		`{"foo",`,
+		`{"foo"f`,
+		"{}}",
+
+		// array
+		`[]`,
+		`[ 1, {}]`,
+		strings.Repeat("[", 1000) + strings.Repeat("]", 1000),
+		`[1, 2, 3, 4, {}]`,
+		`[`,
+		`[1,`,
+		`[1a`,
+		`[]]`,
+
+		// boolean
+		"true",
+		"   true ",
+		"false",
+		"  true f",
+		"fals",
+		"falsee",
+
+		// null
+		"null ",
+		" null ",
+		" nulll ",
+	}
+	for i, test := range tests {
+		in := []byte(test)
+		got := ValidateBytes(in) == nil
+		exp := json.Valid(in)
+
+		if got != exp {
+			t.Errorf("#%d: %q got valid? %v, exp? %v", i, in, got, exp)
+		}
+	}
+}
+
 func testUnescapeStringBestEffort(t *testing.T, s, expectedS string) {
 	t.Helper()