Not 100% compatibility with encoding/json about sonic HOT 10 CLOSED

func TestPrefilledArr(t *testing.T) {
	a := &[...]int{1, 2}
	b := &[...]int{1, 2}
	err1 := json.Unmarshal([]byte(`[3]`), a)
	err2 := Unmarshal([]byte(`[3]`), b)
	require.Equal(t, err1, err2)
	require.Equal(t, a, b)
}

=== RUN   TestPrefilledArr
    fuzz_test.go:109: 
                Error Trace:    fuzz_test.go:109
                Error:          Not equal: 
                                expected: &[2]int{3, 0}
                                actual  : &[2]int{3, 2}
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -2,3 +2,3 @@
                                  (int) 3,
                                - (int) 0
                                + (int) 2
                                 })
                Test:           TestPrefilledArr
--- FAIL: TestPrefilledArr (0.00s)
FAIL
exit status 1

from sonic.

func TestPrefilledArr(t *testing.T) {
	a := &[...]int{1, 2}
	b := &[...]int{1, 2}
	err1 := json.Unmarshal([]byte(`[3]`), a)
	err2 := Unmarshal([]byte(`[3]`), b)
	require.Equal(t, err1, err2)
	require.Equal(t, a, b)
}

=== RUN   TestPrefilledArr
    fuzz_test.go:109: 
                Error Trace:    fuzz_test.go:109
                Error:          Not equal: 
                                expected: &[2]int{3, 0}
                                actual  : &[2]int{3, 2}
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -2,3 +2,3 @@
                                  (int) 3,
                                - (int) 0
                                + (int) 2
                                 })
                Test:           TestPrefilledArr
--- FAIL: TestPrefilledArr (0.00s)
FAIL
exit status 1

@wangkechun Please move this into a separate issue. Thanks.

from sonic.

func TestCompatibility(t *testing.T) {
	inputs := []interface{}{`"<&>"`, `"\"<&>\""`, "\b", float64(-0), float32(-0), map[string]int{"3": 3, "2": 2, "1": 1}}
	for _, input := range inputs {
		t.Run(fmt.Sprintf("case %v", input), func(t *testing.T) {
			buf1, err1 := json.Marshal(input)
			buf2, err2 := Marshal(input)
			require.Nil(t, err1)
			require.Nil(t, err2)
			require.Equal(t, string(buf1), string(buf2))
		})
	}
}

--- FAIL: TestCompatibility (0.00s)
    --- FAIL: TestCompatibility/case_"<&>" (0.00s)
        fuzz_test.go:112: 
                Error Trace:    fuzz_test.go:112
                Error:          Not equal: 
                                expected: "\"\\\"\\u003c\\u0026\\u003e\\\"\""
                                actual  : "\"\\\"<&>\\\"\""
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -"\"\u003c\u0026\u003e\""
                                +"\"<&>\""
                Test:           TestCompatibility/case_"<&>"
    --- FAIL: TestCompatibility/case_"\"<&>\"" (0.00s)
        fuzz_test.go:112: 
                Error Trace:    fuzz_test.go:112
                Error:          Not equal: 
                                expected: "\"\\\"\\\\\\\"\\u003c\\u0026\\u003e\\\\\\\"\\\"\""
                                actual  : "\"\\\"\\\\\\\"<&>\\\\\\\"\\\"\""
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -"\"\\\"\u003c\u0026\u003e\\\"\""
                                +"\"\\\"<&>\\\"\""
                Test:           TestCompatibility/case_"\"<&>\""
    --- FAIL: TestCompatibility/case_\b (0.00s)
        fuzz_test.go:112: 
                Error Trace:    fuzz_test.go:112
                Error:          Not equal: 
                                expected: "\"\\u0008\""
                                actual  : "\"\\b\""
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -"\u0008"
                                +"\b"
                Test:           TestCompatibility/case_\b
    --- FAIL: TestCompatibility/case_map[1:1_2:2_3:3] (0.00s)
        fuzz_test.go:112: 
                Error Trace:    fuzz_test.go:112
                Error:          Not equal: 
                                expected: "{\"1\":1,\"2\":2,\"3\":3}"
                                actual  : "{\"2\":2,\"1\":1,\"3\":3}"
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -{"1":1,"2":2,"3":3}
                                +{"2":2,"1":1,"3":3}
                Test:           TestCompatibility/case_map[1:1_2:2_3:3]
FAIL
exit status 1
FAIL    github.com/bytedance/sonic      0.144s

This IS compatible with stdlib. Compatibility does not mean it should operates exactly like stdlib, minor diffs does not affect compatibility.

The \b escape

According to RFC8259, \b is a valid escape sequence, and the stdlib does not explain why it don't escape to \b. So I consider it's a bug in the stdlib.

HTMLEscape

Should we support this? @PureWhiteWu

The key order of map

maps in Go are intrinsically unordered, the Go Spec also says you should NOT rely on the traversal order. And for performance reason, the key is not ordered by design.

from sonic.

Do you use HTMLEscape in your production code? @wangkechun

from sonic.

Do you use HTMLEscape in your production code? @wangkechun

Never, but I want it to be configurable, like https://pkg.go.dev/github.com/json-iterator/go#pkg-variables

from sonic.

func TestPrefilledArr(t *testing.T) {
	a := &[...]int{1, 2}
	b := &[...]int{1, 2}
	err1 := json.Unmarshal([]byte(`[3]`), a)
	err2 := Unmarshal([]byte(`[3]`), b)
	require.Equal(t, err1, err2)
	require.Equal(t, a, b)
}

=== RUN   TestPrefilledArr
    fuzz_test.go:109: 
                Error Trace:    fuzz_test.go:109
                Error:          Not equal: 
                                expected: &[2]int{3, 0}
                                actual  : &[2]int{3, 2}
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -2,3 +2,3 @@
                                  (int) 3,
                                - (int) 0
                                + (int) 2
                                 })
                Test:           TestPrefilledArr
--- FAIL: TestPrefilledArr (0.00s)
FAIL
exit status 1

@wangkechun Please move this into a separate issue. Thanks.

done #7

from sonic.

The key order of map

maps in Go are intrinsically unordered, the Go Spec also says you should NOT rely on the traversal order. And for performance reason, the key is not ordered by design.

For unit test, reproducible serialization results are very important.

from sonic.

Do you use HTMLEscape in your production code? @wangkechun

Never, but I want it to be configurable, like https://pkg.go.dev/github.com/json-iterator/go#pkg-variables

I think we can support it if there's real needs for it, but in our investigation, this function is rarely used.
Maybe you can submit another issue if you really need it in your production code.

from sonic.

The key order of map

maps in Go are intrinsically unordered, the Go Spec also says you should NOT rely on the traversal order. And for performance reason, the key is not ordered by design.

For unit test, reproducible serialization results are very important.

I think you should not rely on the undefined behaviour in your unit tests. Instead, you can test your code by marshal and unmarshal it and compare to the original value.
So I think this is not really an issue.

from sonic.

Closed since there's no more replies.
If you still have questions, feel free to comment or open another issue.

from sonic.