Comments (1)
zhoujiexiong
commented on July 4, 2024
Hi @Jamel-jun ,
Maybe there is a minor problem of lua-protobuf while converting integer from a series of string patterns.
I have proposed a PR(starwing/lua-protobuf#269) to fix it.
BTW & FYI
It seems that it's the ability of lua-protobuf to support INT string patterns like "#123666666", "#0x+123abc", "+123", "-#123666666", ...
As a contrast, https://pkg.go.dev/mod/google.golang.org/protobuf/encoding/protojson only support a few types, e.g. "123", "-123".
I suggest that we should
- use the subset of the supported patterns while composing the HTTP request body with field[s] of type INT64.
- And for the HTTP response body with field[s] of type INT64, using the option
int64_as_stringcan avoid loss of accuracy, but the client site should notice the '#' prefix.
See also:
https://github.com/zhoujiexiong/lua-protobuf?tab=readme-ov-file#options
Note: The string returned by int64_as_string or int64_as_hexstring will prefix a '#' character. Because Lua may convert between string with number, prefix a '#' makes Lua return the string as-is.
all routines in all module accepts '#' prefix string/hex string as arguments regardless of the option setting.
Some test snippets and outputs
Simulate The Flow of grpc-transcode Plugin
local pb = require "pb"
local protoc = require "protoc"
local cjson = require("cjson.safe")
pb.option "int64_as_string"
--pb.option "int64_as_hexstring"
assert(protoc:load [[
syntax = "proto3";
message Money {
string currency = 1;
repeated int64 values = 2;
}]])
local str_fmt = string.format
local function display_banner(msg)
print(str_fmt([[
--------------------------------------------------------------------------------
%s
--------------------------------------------------------------------------------]], msg))
end
local req_body = [[
{
"currency": "MYR",
"values": [1, 2, -3,
"#123", "0xabF", "#-0x123abcdef", "-#0x123abcdef", "#0x123abc",
"922337203685480", "9.2233720368548e+14", 9.2233720368548e+14]
}]]
display_banner("Simulate The Flow of `grpc-transcode` Plugin")
display_banner("step 1: get http req. body of JSON format")
print(req_body)
display_banner("step 2: unmarshal/deserilize it to Lua object")
local req_body_tbl = cjson.decode(req_body)
if req_body_tbl then
print(require "serpent".block(req_body_tbl))
end
display_banner("step 3: marshal/serilize the Lua object to PB wireformat then send to upstream")
local bytes = assert(pb.encode("Money", req_body_tbl))
--print(pb.tohex(bytes))
print(ngx.encode_base64(bytes))
display_banner("step 4: recv. PB wireformat from upstream then unmarshal/deserilize it to Lua object")
local data2 = assert(pb.decode("Money", bytes))
print(require "serpent".block(data2))
display_banner("step 5: marshal/serilize the Lua object to JSON wireformat then respond to client")
print(cjson.encode(data2))
OUTPUT
--------------------------------------------------------------------------------
Simulate The Flow of `grpc-transcode` Plugin
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
step 1: get http req. body of JSON format
--------------------------------------------------------------------------------
{
"currency": "MYR",
"values": [1, 2, -3,
"#123", "0xabF", "#-0x123abcdef", "-#0x123abcdef", "#0x123abc",
"922337203685480", "9.2233720368548e+14", 9.2233720368548e+14]
}
--------------------------------------------------------------------------------
step 2: unmarshal/deserilize it to Lua object
--------------------------------------------------------------------------------
{
currency = "MYR",
values = {
1,
2,
-3,
"#123",
"0xabF",
"#-0x123abcdef",
"-#0x123abcdef",
"#0x123abc",
"922337203685480",
"9.2233720368548e+14",
922337203685480
} --[[table: 0x08eda328]]
} --[[table: 0x08eda280]]
--------------------------------------------------------------------------------
step 3: marshal/serilize the Lua object to PB wireformat then send to upstream
--------------------------------------------------------------------------------
CgNNWVISPgEC/f//////////AXu/FZHk0OLt/////wGR5NDi7f////8BvPVI6JCO68Xb0QHokI7rxdvRAeiQjuvF29EB
--------------------------------------------------------------------------------
step 4: recv. PB wireformat from upstream then unmarshal/deserilize it to Lua object
--------------------------------------------------------------------------------
{
currency = "MYR",
values = {
1,
2,
-3,
123,
2751,
"#-4893429231",
"#-4893429231",
1194684,
"#922337203685480",
"#922337203685480",
"#922337203685480"
} --[[table: 0x08eebde0]]
} --[[table: 0x08eebd60]]
--------------------------------------------------------------------------------
step 5: marshal/serilize the Lua object to JSON wireformat then respond to client
--------------------------------------------------------------------------------
{"currency":"MYR","values":[1,2,-3,123,2751,"#-4893429231","#-4893429231",1194684,"#922337203685480","#922337203685480","#922337203685480"]}
Test/compare the ability of "google.golang.org/protobuf/encoding/protojson"
package foo
import (
"encoding/base64"
"testing"
v1 "github.com/foo/bar/api/baz/service/v1"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
)
// message MoneyWiredFromLuaProtobuf {
// string currency = 1;
// repeated int64 values = 2;
// }
// --------------------------------------------------------------------------------
// step 1: get http req. body of JSON format
// --------------------------------------------------------------------------------
// {
// "currency": "MYR",
// "values": [1, 2, -3,
// "#123", "0xabF", "#-0x123abcdef", "-#0x123abcdef", "#0x123abc",
// "922337203685480", "9.2233720368548e+14", 9.2233720368548e+14]
// }
func Test_GolangProtojsonUnmarshalWiredFromLuaProtobuf(t *testing.T) {
money := &v1.MoneyWiredFromLuaProtobuf{}
wiredFromLuaProtobufB64Encoded := "Ej4BAv3//////////wF7vxWR5NDi7f////8BkeTQ4u3/////Abz1SOiQjuvF29EB6JCO68Xb0QHokI7rxdvRAQoDTVlS"
data, err := base64.StdEncoding.DecodeString(wiredFromLuaProtobufB64Encoded)
require.Nil(t, err)
err = proto.Unmarshal(data, money)
require.Nil(t, err)
var expected = int64(922337203685480)
require.Equal(t, expected, money.Values[8])
require.Equal(t, expected, money.Values[9])
require.Equal(t, expected, money.Values[10])
indented, err := protojson.MarshalOptions{Multiline: true, Indent: "\t"}.Marshal(money)
require.Nil(t, err)
t.Log("\n" + string(indented))
}
// message Money {
// string currency = 1;
// int64 value = 2;
// }
func Test_GolangProtojsonUnmarshal(t *testing.T) {
money := &v1.Money{}
cases := []struct {
Name string
ReqBody string
RequireNil bool
}{
{
"hexdecimal string without 0x[X] prefix",
`{"currency": "MYR", "value": "abcdef"}`, false,
},
{
"hexdecimal string with 0x[X] prefix",
`{"currency": "MYR", "value": "0xabcef"}`, false,
},
{
"hexdecimal string with 0x[X] prefix",
`{"currency": "MYR", "value": "#123"}`, false,
},
{
"decimal string",
`{"currency": "MYR", "value": "123"}`, true,
},
{
"decimal integer",
`{"currency": "MYR", "value": 123}`, true,
},
{
"decimal(negative) integer",
`{"currency": "MYR", "value": -123}`, true,
},
{
"decimal(negative) string",
`{"currency": "MYR", "value": "-123"}`, true,
},
{
"decimal(positive) integer",
`{"currency": "MYR", "value": +123}`, false,
},
{
"decimal(positive) string",
`{"currency": "MYR", "value": "+123"}`, false,
},
{
"scientific notation",
`{"currency": "MYR", "value": 9.2233720368548e+14}`, true,
},
{
"scientific notation string",
`{"currency": "MYR", "value": "9.2233720368548e+14"}`, true,
},
}
var err error
for _, c := range cases {
name := c.Name
if !c.RequireNil {
name = "[Unsupported] " + name
}
t.Run(name, func(t *testing.T) {
err = protojson.Unmarshal([]byte(c.ReqBody), money)
if c.RequireNil {
require.Nil(t, err)
} else {
require.NotNil(t, err)
}
})
}
}
// Local Variables:
// go-test-args: "-v -count=1"
// End:OUTPUT
go test -v -count=1 -run='Test_GolangProtojsonUnmarshalWiredFromLuaProtobuf|Test_GolangProtojsonUnmarshal' .
=== RUN Test_GolangProtojsonUnmarshalWiredFromLuaProtobuf
{
"currency": "MYR",
"values": [
"1",
"2",
"-3",
"123",
"2751",
"-4893429231",
"-4893429231",
"1194684",
"922337203685480",
"922337203685480",
"922337203685480"
]
}
--- PASS: Test_GolangProtojsonUnmarshalWiredFromLuaProtobuf (0.00s)
=== RUN Test_GolangProtojsonUnmarshal
=== RUN Test_GolangProtojsonUnmarshal/[Unsupported]_hexdecimal_string_without_0x[X]_prefix
=== RUN Test_GolangProtojsonUnmarshal/[Unsupported]_hexdecimal_string_with_0x[X]_prefix
=== RUN Test_GolangProtojsonUnmarshal/[Unsupported]_hexdecimal_string_with_0x[X]_prefix#01
=== RUN Test_GolangProtojsonUnmarshal/decimal_string
=== RUN Test_GolangProtojsonUnmarshal/decimal_integer
=== RUN Test_GolangProtojsonUnmarshal/decimal(negative)_integer
=== RUN Test_GolangProtojsonUnmarshal/decimal(negative)_string
=== RUN Test_GolangProtojsonUnmarshal/[Unsupported]_decimal(positive)_integer
=== RUN Test_GolangProtojsonUnmarshal/[Unsupported]_decimal(positive)_string
=== RUN Test_GolangProtojsonUnmarshal/scientific_notation
=== RUN Test_GolangProtojsonUnmarshal/scientific_notation_string
--- PASS: Test_GolangProtojsonUnmarshal (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/[Unsupported]_hexdecimal_string_without_0x[X]_prefix (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/[Unsupported]_hexdecimal_string_with_0x[X]_prefix (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/[Unsupported]_hexdecimal_string_with_0x[X]_prefix#01 (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/decimal_string (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/decimal_integer (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/decimal(negative)_integer (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/decimal(negative)_string (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/[Unsupported]_decimal(positive)_integer (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/[Unsupported]_decimal(positive)_string (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/scientific_notation (0.00s)
--- PASS: Test_GolangProtojsonUnmarshal/scientific_notation_string (0.00s)
PASS
ok
Go-Test finished at Wed Jun 19 18:51:11
from apisix.
Related Issues (20)
- help request: Apisix ETCD going into Crash loop back off HOT 10
- feat: support environment variables in config-default.yaml HOT 3
- help request: What is the best way to dynamically route a cluster by token in customer plugin? HOT 3
- help request: proxy rewrite rule does not work HOT 2
- why the promethues metrics apisix_shared_dict_free_space_bytes{name="worker-events"} 0 HOT 2
- feat: As a user, I want to expand the weight in kubernetes service discovery
- bug: The command "apisix reload" sometimes causes the plugin's sock file to disappear HOT 2
- help request: want to know how to support sslv3 in apisix HOT 2
- help request: How to expose custom Prometheus metric with custom plugin
- The client secret for oidc plugin cannot be configured as environment variable
- bug: When I turn on kafka-logger and body-transformer at the same time, the response body conversion of body-transformer plugin will fail. HOT 2
- help request: Request for Assistance: Loki Logging Not Receiving Logs from Specific APIsix Route
- help request: 尝试代理 hdfs 的 webui，但是会返回 302 重定向，应该如何配置呢？或者是否有示例 HOT 1
- apisix will retry the post method? HOT 6
- bug: forward-auth does not return upstream headers to client HOT 1
- bug: http2 requests are not supported without content-length header HOT 2
- How to forward to upstream after successful authentication using the forward-auth plugin HOT 1
- help request: missing user key in JWT token HOT 1
- feat: support env var for secret config of plugins like kafka-logger, elasticsearch-logger HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from apisix.