uber / zanzibar Goto Github PK

View Code? Open in Web Editor NEW

452.0 1.8K 66.0 32.51 MB

A build system & configuration system to generate versioned API gateways.

License: MIT License

Shell 1.07% Makefile 0.76% Go 96.82% Lua 0.95% JavaScript 0.41%

zanzibar's People

Contributors

Stargazers

Watchers

zanzibar's Issues

easy json generate only works for http endpoint

https://github.com/uber/zanzibar/blob/master/codegen/runner/pre-steps.sh#L92
hardcoded check to bypass non http endpoint, we might want this to be configured from a list of endpoints that want to use easy-json

Add instrumentation for HTTP Client

Add support for tally based, request/response instrumentation
Ensure that the instrumentation support is similar to the http server

We need to add support for metrics to be emitted for request/response count, time latency, error codes, etc.

Handle thrift errors in http client/endpoint

I would like to see an example of a thrift error being returned/handled by the http client. We need this so we can implement a conforming implementation with tchannel. How we choose to define these interfaces should be consistent where possible.

Thrift integration needs better support

Currently we have two processes that independently handle thrift, and it isn't clear to me how to interact with the thrift spec and the related generated structs. We probably need to look at generating thrift structs as part of the build step, and also proving the compiled specs in an accessible way, otherwise modules that require access to thrift specs are going to be reading from disk and manually compiling the specs on the fly.

Check imports in generated code before gofmt

I have a suspicion that code generation followed by gofmt leads to implicit inclusion of packages, and this behavior is flappy. We should write a check to compare imports before and after formatting to enforce that code generation includes all required packages.

flappy test : `make fast-bench` fails on travis sometimes

Not sure why, trying to reproduce locally.

provide basic traffic control features

on top of my head that is necessary for a production service

[] circuit breaking
[] load shedding

Call staticConfig.Destroy after initialization

https://github.com/uber/zanzibar/blob/master/runtime/static_config.go#L324

static config should be destroyed after all modules have been initialized, explicitly. This is to avoid any runtime access to MustGet.

HTTP client generation should support multiple thrift services

Current HTTP client generation assumes its thrift has only one service defined, which is not the case in reality. TChannel client addresses the problem by introducing a method mapping exposedMethods in the client-config.json file, e.g.

{
	"name": "baz",
	"type": "tchannel",
	"config": {
		"thriftFile": "clients/baz/baz.thrift",
		"thriftFileSha": "{{placeholder}}",
		"exposedMethods": {
			"Call": "SimpleService::Call",
			 ...
			"Echo": "SecondService::Echo"
		}
	}
}

We should have parity on the HTTP client side, despite it is going to be a breaking change.

Pack the zanzibar-defaults config into the binary

The generate main.go files depend on "zanzibar-defaults.json". The main.go is generated with a hard coded relative file path. All files except the binary and the config directory will be removed, so this doesn't work. Either expect the defaults to be in the config directory itself, or pack the defaults into the binary.

Upgrade httprouter package

This repo depends on julienschmidt/httprouter who made a breaking change in 2015 by changing its API from http.HandleFunc to http.Handler. We are still using a prior version which means attempting to update past this change on httprouter triggers a build failure.

Typo in repo description

The current description says

A build system & configuration system to generate versionsed API gateways.

It should say "versioned".

Move example-gateway to a standalone example

example-gateway is interleaved with Zanzibar and does not work as a standalone example. It should be a full-fledged standalone example.

Move `backend/repository` out of zanzibar

backend/repository, as an application relies on Zanzibar, does not really need to be part of Zanzibar.

Add goleak

https://github.com/uber-go/goleak

Change logging levels for zap child logger

Sometimes we want to pass a child logger with different logger levels to sub systems.

We can do this with something like :

childLogger := logger.With(
   zap.String("library", "{{.LibraryName}}"),
).WithOptions(zap.WrapCore(func(c zapcore.Core) zapcore.Core {
    copy := c.With()
    copy.LevelEnabler = zap.NewAtomicLevel(zapcore.WarnLevel)
    return copy
})

We would need to test this but it should allow us to have a child logger with a "library=X" field and with a different minimum logging level.

cc @srivastavankit @ChuntaoLu

Consider using jennifer code gen when templating becomes too complex.

There is an open source package jennifer ( https://github.com/davelondon/jennifer )

Which can be an alternative code gen library if text/template approach hits a complexity bottleneck.

The jennifer code gen approach works on AST so it will generate correct whitespace and imports and such.

This can be evaluated once our if / else logic in the template becomes too messy.

Add kafka logging

Possibly use filebeat to send logs to kafka.

Tracking HTTP+JSON thrift compatibility work

Thrift semantics TODO:

Logger.Warn() about unexpected fields in JSON payload

Modify easyjson to track keys in objects that are skipped.

Required field semantics

Add support for marking a field required. Need to change thriftrw
to generate the required tag. Verify that it works as expected.

Recursive converter functions

Currently our server / client struct conversions are not
recursive and only convert a subset of the fields.
These functions need to be generated in a recursive fashion.

Do not parse JSON body on GET / DELETE method

Do not parse the body.

Check Content-Type header before parsing

Currently we naively parse json, need to verify content-type
header before attempting to parse.

Add support for parsing params to server

Currently on the client supports params, need to add support
to the server for this.

Add support for "-" for non-body fields

Some fields come from other parts of the HTTP request, aka
headers, params, query, etc.

Need to ensure generated code has "-" annotation to not parse
said field from the body.

This will require a thriftrw change, potentially. Will probably
need to add an annotation conversion logic to customize the
semantics of our annotations and how thriftrw parse them.

Add support for exceptions with the same status code

Currently HTTP clients do not support exceptions with duplicate
status codes.

Remove req.def = boxed annotation

This annotation is deprecated and should not be used, instead
we should use many top level optional & required fields.

Ensure status, method & path are mandatory annotations

Verify our code generation error messages when these are missing.

Add support for http.ref = headers

We need to be able to read header keys into body fields
This needs to happen for both client & server.

Add support for http.ref = query

We need to be able to read/write fields from query parameters
in both the client and the server.

Add support for http.ref = body

We need to be able to change the JSON body name that this field
is written or read from. This is generally for camelCase vs
snake case.

Add support for headerGroups

When validating required headers some clients have many, many
required headers ( more then 10 ). To keep the code clean and DRY
we want to be able to define a struct with a set of required
headers and annotate a thrift method with "headerGroups=structName".

This would mean that this thrift method has a set of required
headers defined in the zanzibar.http.headers annotation as well
as the zanzibar.headerGroups annotation.

Remove zanzibar.meta annotation

This is a dead annotation with no meaning

Remove zanzibar.handler annotation

This is a dead annotation with no meaning

Add support for `zanzibar.validation.type` coercions

This means forking easyjson and adding coercion options to it. Need to implement
all of the coercion semantics for bool, string, i32 & double.

Customize the i64 encoder/decoder

We need to add detailed support for encoding i64 into different types, aka date, long, buffer.
This requires a change in easyjson

The date change is slightly involved.

ensure we serialize thrift enums properly

Thrift enums are integers in golang and need to be serialized to strings on the wire.
This will require customization in thriftrw-go

ensure we serialize thrift binary properly

Thrift binary is []byte in golang and need to be serialized to array of numbers on the wire.
[]byte currently serializes to base64 encoded string.

This will require customization in easyjson or thriftrw-go.

pinned linter to a specific version in vendor/

pin honnef.co/go/tools/cmd/staticcheck to 8ed405e85c65fb38745a8eafe01ee9590523f172
in #397

lastest master of this linter is grumpy about
https://golang.org/pkg/fmt/#Fprintln returned error is not checked in generated easy_json go files

Another thought, ideally we don't need pin linter, in linter we trust if we commit to a linter our code repo should evolve style with it, otherwise what could happen is 1 year from now no one bothers to bump this and we missed some important linting rules provided, I'm open to either way

Unable to use list<X> in a HTTP query parameter results in panic

When decoding URI parameters for HTTP requests, zanzibar lacks an encoder and decoder for list Thrift types, e.g. list<string>. Currently this causes a panic due to a failing type switch when trying to create the encoder or decoder.

A standard encoding of composite types is undefined. RFC 3986 § 3.4 merely says:

[...] query components are often used to carry identifying information in the form of "key=value" pairs

but defines no convention for delimiters or encodings of complex types like maps or lists, leaving it up to the particular HTTP daemon to decide.

Golang net standard library will encode a list e.g. foo=[a, b] as foo=a&foo=b:

	v := url.Values{}
	v.Set("name", "Ava")
	v.Add("friend", "Jess")
	v.Add("friend", "Sarah")
	v.Add("friend", "Zoe")
	// v.Encode() == "name=Ava&friend=Jess&friend=Sarah&friend=Zoe"

Decoding is done by using the map access since url.Values is typedef of a map[string][]string.

Zanzibar has support for struct types in URL parameters by using a dotted notation: a.b=foo would map b as a string property on struct a. This creates ambiguity if we add support for list types; a.b.c=foo&a.b.c=bar can be decoded to either of these structs:

As c is list of string:

struct A {
    1: required B b
}

struct B {
    1: required list<string> c
}

Or as b is list of struct:

struct A {
    1: required list<B> b
}

struct B {
    1: required string c
}

To avoid this ambiguity, we should restrict lists to scalar types.

Random hang of test suite

Sometimes the ./test/ package itself will hang in a test case

Here is a dump of sigabort to see why the tests / goroutines are hanging








{"level":"info","ts":1505849366.9648693,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4765,"zone":"unknown","realHTTPAddr":"10.32.106.72:34521","realTChannelAddr":"10.32.106.72:40161","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:38385","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:40114","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"error","ts":1505849367.029912,"msg":"Failed to Bootstrap in TestStartGateway()","error":"error listening on port: listen tcp 10.32.106.72:34521: bind: address already in use"}
{"level":"error","ts":1505849367.0298743,"msg":"Error listening on port","hostname":"raynos-ThinkPad-T440p","pid":4776,"zone":"unknown","error":"listen tcp 10.32.106.72:34521: bind: address already in use"}
{"level":"info","ts":1505849367.0834262,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4784,"zone":"unknown","realHTTPAddr":"10.32.106.72:36721","realTChannelAddr":"10.32.106.72:37823","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:33303","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:49837","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"info","ts":1505849367.0845563,"msg":"Finished an incoming server HTTP request","hostname":"raynos-ThinkPad-T440p","pid":4784,"zone":"unknown","endpointID":"health","handlerID":"health","method":"GET","remoteAddr":"10.32.106.72:33878","pathname":"/health","host":"10.32.106.72:36721","timestamp-started":1505849367.0845108,"timestamp-finished":1505849367.0845234,"statusCode":200,"Request Body":"","Response Body":"{\"ok\":true,\"message\":\"Healthy, from example-gateway\"}\n","Request-Header-User-Agent":"Go-http-client/1.1","Request-Header-Accept-Encoding":"gzip","Response-Header-Content-Type":"application/json"}
{"level":"info","ts":1505849367.126711,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4793,"zone":"unknown","realHTTPAddr":"10.32.106.72:35879","realTChannelAddr":"10.32.106.72:35419","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:40493","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:51699","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":false,"metrics.runtime.enableGCMetrics":false,"metrics.runtime.enableMemMetrics":false,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
{"level":"info","ts":1505849367.1276844,"msg":"Finished an incoming server HTTP request","hostname":"raynos-ThinkPad-T440p","pid":4793,"zone":"unknown","endpointID":"health","handlerID":"health","method":"GET","remoteAddr":"10.32.106.72:37640","pathname":"/health","host":"10.32.106.72:35879","timestamp-started":1505849367.127651,"timestamp-finished":1505849367.1276588,"statusCode":200,"Request Body":"","Response Body":"{\"ok\":true,\"message\":\"Healthy, from example-gateway\"}\n","Request-Header-User-Agent":"Go-http-client/1.1","Request-Header-Accept-Encoding":"gzip","Response-Header-Content-Type":"application/json"}
{"level":"info","ts":1505849367.1945999,"msg":"Started ExampleGateway","hostname":"raynos-ThinkPad-T440p","pid":4802,"zone":"unknown","realHTTPAddr":"10.32.106.72:39537","realTChannelAddr":"10.32.106.72:44285","config":{"clients.bar.ip":"127.0.0.1","clients.bar.port":4001,"clients.bar.timeout":10000,"clients.baz.ip":"127.0.0.1","clients.baz.port":4002,"clients.baz.serviceName":"Qux","clients.baz.timeout":1000,"clients.baz.timeoutPerAttempt":1000,"clients.contacts.ip":"127.0.0.1","clients.contacts.port":4000,"clients.contacts.timeout":10000,"clients.google-now.ip":"127.0.0.1","clients.google-now.port":14120,"clients.google-now.timeout":10000,"clients.googleNowTChannel.connectionType":"p2p","clients.googleNowTChannel.hostList":["127.0.0.1:4002"],"clients.googleNowTChannel.timeout":10000,"datacenter":"unknown","env":"test","http.port":0,"jaeger.disabled":false,"jaeger.reporter.flush.milliseconds":0,"jaeger.reporter.hostport":"127.0.0.1:57802","jaeger.sampler.param":1,"jaeger.sampler.type":"const","logger.fileName":"/var/log/example-gateway/example-gateway.log","logger.output":"stdout","metrics.flushInterval":10,"metrics.m3.hostPort":"127.0.0.1:49706","metrics.m3.maxPacketSizeBytes":1440,"metrics.m3.maxQueueSize":10000,"metrics.runtime.collectInterval":10,"metrics.runtime.enableCPUMetrics":true,"metrics.runtime.enableGCMetrics":true,"metrics.runtime.enableMemMetrics":true,"metrics.serviceName":"test-gateway","metrics.type":"m3","service.env.config":{},"serviceName":"example-gateway","tchannel.port":0,"tchannel.processName":"test-gateway","tchannel.serviceName":"test-gateway","useDatacenter":false}}
fatal error: concurrent map iteration and map write

goroutine 50 [running]:
runtime.throw(0xba8457, 0x26)
	/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc42004d988 sp=0xc42004d968
runtime.mapiternext(0xc420443f80)
	/usr/local/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc42004da38 sp=0xc42004d988
reflect.mapiternext(0xc420443f80)
	/usr/local/go/src/runtime/hashmap.go:1156 +0x2b fp=0xc42004da50 sp=0xc42004da38
reflect.Value.MapKeys(0xac5220, 0xc420332b10, 0x15, 0xc42004dbe9, 0x98, 0xc42004dc38)
	/usr/local/go/src/reflect/value.go:1112 +0x1bc fp=0xc42004db00 sp=0xc42004da50
github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert.includeElement(0xac5220, 0xc420332b10, 0xa8aac0, 0xc420390b10, 0xc420040000)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert/assertions.go:617 +0x2db fp=0xc42004dbc8 sp=0xc42004db00
github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert.Contains(0xf3cdc0, 0xc420338340, 0xac5220, 0xc420332b10, 0xa8aac0, 0xc420390b10, 0xc42004dd98, 0x2, 0x2, 0x1)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/stretchr/testify/assert/assertions.go:645 +0x59 fp=0xc42004dc48 sp=0xc42004dbc8
github.com/uber/zanzibar/test_test.TestRuntimeMetrics(0xc420338340)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/health_test.go:223 +0x731 fp=0xc42004dfa8 sp=0xc42004dc48
testing.tRunner(0xc420338340, 0xbb66a8)
	/usr/local/go/src/testing/testing.go:657 +0x96 fp=0xc42004dfd0 sp=0xc42004dfa8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42004dfd8 sp=0xc42004dfd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:697 +0x2ca

goroutine 1 [chan receive]:
testing.(*T).Run(0xc420071520, 0xb98dc4, 0x12, 0xbb66a8, 0xc4202d5d01)
	/usr/local/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc420071520)
	/usr/local/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc420071520, 0xc4202d5de0)
	/usr/local/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc420299d40, 0xf723e0, 0x4, 0x4, 0xaf4c20)
	/usr/local/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc420049f20, 0xc4202d5f20)
	/usr/local/go/src/testing/testing.go:822 +0xfc
main.main()
	github.com/uber/zanzibar/test/_test/_testmain.go:50 +0xf7

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 37 [IO wait]:
net.runtime_pollWait(0x7f7d951b1948, 0x72, 0xd)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420446298, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420446298, 0xc42048c028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc420446230, 0xc42048c028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4203ca0a8, 0xc42048c028, 0xfde8, 0xfde8, 0xc42048bfc8, 0x411598, 0xc42048bfc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc420394480, 0xc42048c028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc420388300, 0xc4204332d0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 52 [IO wait]:
net.runtime_pollWait(0x7f7d951b1708, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42042c458, 0x72, 0xf386e0, 0xc420421800)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42042c458, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42042c3f0, 0x0, 0xf3ca80, 0xc420421800)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc4204c8088, 0xc420373a40, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4204c8088, 0xc420373a10, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4204e00b0, 0xf43180, 0xc4204c8088, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4204cca20)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 8 [IO wait]:
net.runtime_pollWait(0x7f7d951b1e88, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202daa78, 0x72, 0xf386e0, 0xc42033c000)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202daa78, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202daa10, 0x0, 0xf3ca80, 0xc42033c000)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e148, 0xc4203320c0, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e148, 0xc420332090, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001ec60, 0xf43180, 0xc42000e148, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc42006fce0)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 9 [IO wait]:
net.runtime_pollWait(0x7f7d951b1f48, 0x72, 0x5)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202daa08, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202daa08, 0xc420312028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202da9a0, 0xc420312028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e138, 0xc420312028, 0xfde8, 0xfde8, 0xc420311fc8, 0x411598, 0xc420311fc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2480, 0xc420312028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc420019d80, 0xc4202aed80)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 36 [IO wait]:
net.runtime_pollWait(0x7f7d951b1888, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420446308, 0x72, 0xf386e0, 0xc42033c0c0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420446308, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4204462a0, 0x0, 0xf3ca80, 0xc42033c0c0)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc4203ca0b8, 0xc420332330, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4203ca0b8, 0xc420332300, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc4204480b0, 0xf43180, 0xc4203ca0b8, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc420442180)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 53 [IO wait]:
net.runtime_pollWait(0x7f7d951b17c8, 0x72, 0xf)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42042c3e8, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42042c3e8, 0xc42051e028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42042c380, 0xc42051e028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc4204c8078, 0xc42051e028, 0xfde8, 0xfde8, 0x411f6d, 0xc4204d0500, 0xc42051dfc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc42033a630, 0xc42051e028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fe380, 0xc420337a40)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 12 [IO wait]:
net.runtime_pollWait(0x7f7d951b1c48, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db1e8, 0x72, 0xf386e0, 0xc420396000)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db1e8, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202db180, 0x0, 0xf3ca80, 0xc420396000)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e270, 0xc42038c0c0, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e270, 0xc42038c090, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001efd0, 0xf43180, 0xc42000e270, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4203441e0)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 13 [IO wait]:
net.runtime_pollWait(0x7f7d951b1d08, 0x72, 0x9)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db178, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db178, 0xc4203b8028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202db110, 0xc4203b8028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e260, 0xc4203b8028, 0xfde8, 0xfde8, 0xc4203b7fc8, 0x411598, 0xc4203b7fc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2870, 0xc4203b8028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fc180, 0xc420360110)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 51 [runnable]:
bytes.(*Buffer).WriteByte(0xc420110f50, 0xc42037592b, 0x1c, 0x1c)
	/usr/local/go/src/bytes/buffer.go:236 +0x82
github.com/uber/zanzibar/vendor/github.com/uber-go/tally.KeyForPrefixedStringMap(0xc420375900, 0x1c, 0xc420038ca0, 0xc420038dd8, 0xc420038ca0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/key_gen.go:68 +0x267
github.com/uber/zanzibar/test/lib/test_m3_server.(*FakeM3Service).EmitMetricBatch(0xc420300550, 0xc4203755e0, 0x0, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_m3_server/test_m3_server.go:159 +0x30b
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift.(*m3ProcessorEmitMetricBatch).Process(0xc420337960, 0xc400000006, 0xf4bd60, 0xc42044a1e0, 0xf4bd60, 0xc42044a1e0, 0xfa00, 0x525, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift/m3.go:163 +0xeb
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift.(*M3Processor).Process(0xc42033d7c0, 0xf4bd60, 0xc42044a1e0, 0xf4bd60, 0xc42044a1e0, 0x0, 0x0, 0xc4203755c0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/thrift/m3.go:137 +0x306
github.com/uber/zanzibar/test/lib/test_m3_server.(*FakeM3Server).Serve(0xc4203005a0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_m3_server/test_m3_server.go:95 +0x269
created by github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).setupMetrics
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway.go:127 +0xc5

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f7d951b1ac8, 0x72, 0xf3eb00)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db728, 0x72, 0xf386e0, 0xc42033c020)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db728, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4202db6c0, 0x0, 0xf3ca80, 0xc42033c020)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e390, 0xc4203321b0, 0xabc420, 0xf6ade0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e390, 0xc420332180, 0xabc420, 0xf6ade0, 0xb23780)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc42001f340, 0xf43180, 0xc42000e390, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
net/http/httptest.(*Server).goServe.func1(0xc4203444e0)
	/usr/local/go/src/net/http/httptest/server.go:235 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:236 +0x5c

goroutine 18 [IO wait]:
net.runtime_pollWait(0x7f7d951b1b88, 0x72, 0x8)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4202db6b8, 0x72, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4202db6b8, 0xc4203fe028, 0xfde8)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4202db650, 0xc4203fe028, 0xfde8, 0xfde8, 0x0, 0xf3eb00, 0xf386e0)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e380, 0xc4203fe028, 0xfde8, 0xfde8, 0xc4203fdfc8, 0x411598, 0xc4203fdfc8)
	/usr/local/go/src/net/net.go:181 +0x70
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*TUDPTransport).Read(0xc4200b2d80, 0xc4203fe028, 0xfde8, 0xfde8, 0x0, 0xfde8, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/udp_transport.go:94 +0x174
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.(*MockAgent).serve(0xc4202fc680, 0xc4203619c0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:109 +0x3b7
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils.StartMockAgent
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/testutils/mock_agent.go:61 +0x1dc

goroutine 23 [select]:
net/http.(*persistConn).readLoop(0xc42045aea0)
	/usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 21 [select]:
net/http.(*persistConn).readLoop(0xc420424000)
	/usr/local/go/src/net/http/transport.go:1599 +0x9ec
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 22 [select]:
net/http.(*persistConn).writeLoop(0xc420424000)
	/usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 24 [select]:
net/http.(*persistConn).writeLoop(0xc42045aea0)
	/usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 43 [syscall]:
syscall.Syscall(0x0, 0x11, 0xc4204e6000, 0x1000, 0x20, 0x20, 0xaf6700)
	/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x11, 0xc4204e6000, 0x1000, 0x1000, 0x7f7d951f5960, 0x0, 0xc420027cf0)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55
syscall.Read(0x11, 0xc4204e6000, 0x1000, 0x1000, 0xaf6700, 0x736901, 0xc42000c560)
	/usr/local/go/src/syscall/syscall_unix.go:162 +0x49
os.(*File).read(0xc42000e5d0, 0xc4204e6000, 0x1000, 0x1000, 0xc, 0xc, 0x0)
	/usr/local/go/src/os/file_unix.go:165 +0x4f
os.(*File).Read(0xc42000e5d0, 0xc4204e6000, 0x1000, 0x1000, 0x0, 0x0, 0x8)
	/usr/local/go/src/os/file.go:101 +0x76
bufio.(*Reader).fill(0xc420442060)
	/usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).ReadSlice(0xc420442060, 0x73c70a, 0xc4203ca008, 0xf4bd60, 0xc420094000, 0x0, 0xb331c0)
	/usr/local/go/src/bufio/bufio.go:338 +0xbb
bufio.(*Reader).ReadBytes(0xc420442060, 0xf4bd0a, 0xc420094000, 0xf4bd60, 0xc420094000, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:416 +0x66
bufio.(*Reader).ReadString(0xc420442060, 0xc42001010a, 0xc4200140e0, 0xc420027fc0, 0xc420094000, 0xc42052e000)
	/usr/local/go/src/bufio/bufio.go:456 +0x38
github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).copyToStdout(0xc4204ec000, 0xc420442060)
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway_process.go:221 +0x34
created by github.com/uber/zanzibar/test/lib/test_gateway.(*ChildProcessGateway).createAndSpawnChild
	/home/raynos/gocode/src/github.com/uber/zanzibar/test/lib/test_gateway/test_gateway_process.go:125 +0x61f
SIGABRT: abort
PC=0x45c2d3 m=2 sigcode=0

goroutine 0 [idle]:
runtime.futex(0xf23f58, 0x0, 0x7f111938adc8, 0x0, 0x7f1100000000, 0x45bf66, 0x3c, 0x0, 0x7f111938ae10, 0x40f720, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:426 +0x23
runtime.futexsleep(0xf23f58, 0x0, 0xdf8475800)
	/usr/local/go/src/runtime/os_linux.go:62 +0xd7
runtime.notetsleep_internal(0xf23f58, 0xdf8475800, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:174 +0xd0
runtime.notetsleep(0xf23f58, 0xdf8475800, 0x104363f71d3d01)
	/usr/local/go/src/runtime/lock_futex.go:194 +0x56
runtime.sysmon()
	/usr/local/go/src/runtime/proc.go:3805 +0x135
runtime.mstart1()
	/usr/local/go/src/runtime/proc.go:1179 +0x11e
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1149 +0x64

goroutine 1 [chan receive, 49 minutes]:
testing.(*T).Run(0xc420063380, 0xb5b1dc, 0x10, 0xb78860, 0xc4202bdcf0)
	/usr/local/go/src/testing/testing.go:698 +0x2f4
testing.runTests.func1(0xc420063380)
	/usr/local/go/src/testing/testing.go:882 +0x67
testing.tRunner(0xc420063380, 0xc4202bddb0)
	/usr/local/go/src/testing/testing.go:657 +0x96
testing.runTests(0xc420287b20, 0xf15000, 0x1, 0x1, 0xf24600)
	/usr/local/go/src/testing/testing.go:888 +0x2c1
testing.(*M).Run(0xc42004ff20, 0x1e)
	/usr/local/go/src/testing/testing.go:822 +0xfc
command-line-arguments.TestMain(0xc4202bdf20)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:44 +0x58
main.main()
	command-line-arguments/_test/_testmain.go:40 +0xf7

goroutine 17 [syscall, 49 minutes, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 6 [syscall, 49 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 7 [select, 49 minutes, locked to thread]:
runtime.gopark(0xb7a250, 0x0, 0xb532ec, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc42002cf50, 0x0, 0x18)
	/usr/local/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc42002cf50)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 8 [chan receive, 49 minutes]:
command-line-arguments.listenOnSignals.func1(0xc420061740)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:57 +0x53
created by command-line-arguments.listenOnSignals
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:62 +0xbc

goroutine 9 [semacquire, 49 minutes]:
sync.runtime_Semacquire(0xc4202c2a6c)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4202c2a60)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/uber/zanzibar/runtime.(*Gateway).Wait(0xc420089400)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:263 +0x2f
command-line-arguments.logAndWait(0xc420089400)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main.go:102 +0x2f0
command-line-arguments.TestStartGateway(0xc420063450)
	/home/raynos/gocode/src/github.com/uber/zanzibar/examples/example-gateway/build/services/example-gateway/main/main_test.go:92 +0x461
testing.tRunner(0xc420063450, 0xb78860)
	/usr/local/go/src/testing/testing.go:657 +0x96
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:697 +0x2ca

goroutine 10 [chan receive]:
github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3.(*reporter).process(0xc4202cac80)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/reporter.go:483 +0xe3
created by github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3.NewReporter
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/m3/reporter.go:241 +0x811

goroutine 11 [select]:
github.com/uber/zanzibar/vendor/github.com/uber-go/tally.(*scope).reportLoop(0xc420089540, 0x989680)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/scope.go:241 +0x158
created by github.com/uber/zanzibar/vendor/github.com/uber-go/tally.newRootScope
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber-go/tally/scope.go:180 +0x6b2

goroutine 12 [runnable]:
github.com/uber/zanzibar/runtime.(*runtimeCollector).Start.func1(0xc42001d680)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/runtime_metrics.go:184 +0x13f
created by github.com/uber/zanzibar/runtime.(*runtimeCollector).Start
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/runtime_metrics.go:192 +0x84

goroutine 13 [select]:
github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc4203cb440)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/reporter.go:231 +0x324
created by github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go.NewRemoteReporter
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/jaeger-client-go/reporter.go:201 +0x288

goroutine 20 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4e00, 0x72, 0xee8660)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498a78, 0x72, 0xee2660, 0xc420716d60)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498a78, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420498a10, 0x0, 0xee6720, 0xc420716d60)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e078, 0xc42070d1a0, 0xc42003ad90, 0x5dbb5d)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e078, 0x6f1463, 0xc42003ada8, 0xc42003ada0)
	/usr/local/go/src/net/tcpsock.go:215 +0x49
github.com/uber/zanzibar/runtime.tcpKeepAliveListener.Accept(0xc42000e078, 0xc42070d170, 0xa86f20, 0xf13fa0, 0xaeb820)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:107 +0x2f
net/http.(*Server).Serve(0xc4204b0000, 0xeece20, 0xc42000e078, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
github.com/uber/zanzibar/runtime.(*HTTPServer).JustServe(0xc420474500, 0xc4202c2a60)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:74 +0x78
created by github.com/uber/zanzibar/runtime.(*Gateway).Bootstrap
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:164 +0x5cc

goroutine 21 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4ec0, 0x72, 0xee8660)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498a08, 0x72, 0xee2660, 0xc420716d80)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498a08, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4204989a0, 0x0, 0xee6720, 0xc420716d80)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e070, 0xc42070d290, 0xc42003ed90, 0x5dbb5d)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e070, 0x6f1463, 0xc42003eda8, 0xc42003eda0)
	/usr/local/go/src/net/tcpsock.go:215 +0x49
github.com/uber/zanzibar/runtime.tcpKeepAliveListener.Accept(0xc42000e070, 0xc42070d260, 0xa86f20, 0xf13fa0, 0xaeb820)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:107 +0x2f
net/http.(*Server).Serve(0xc4204b00b0, 0xeece20, 0xc42000e070, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
github.com/uber/zanzibar/runtime.(*HTTPServer).JustServe(0xc420474550, 0xc4202c2a60)
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/http_server.go:74 +0x78
created by github.com/uber/zanzibar/runtime.(*Gateway).Bootstrap
	/home/raynos/gocode/src/github.com/uber/zanzibar/runtime/gateway.go:168 +0xce9

goroutine 22 [IO wait, 49 minutes]:
net.runtime_pollWait(0x7f1119af4d40, 0x72, 0xee8660)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420498ae8, 0x72, 0xee2660, 0xc420716760)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420498ae8, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc420498a80, 0x0, 0xee6720, 0xc420716760)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e080, 0xc400000008, 0xc420480000, 0x0)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e080, 0xb796e8, 0xc420716560, 0x1, 0x0)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/tnet.(*listener).Accept(0xc420716560, 0x0, 0x0, 0x0, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/tnet/listener.go:80 +0x8f
github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go.(*Channel).serve(0xc4204b6000)
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/channel.go:390 +0x56
created by github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go.(*Channel).Serve
	/home/raynos/gocode/src/github.com/uber/zanzibar/vendor/github.com/uber/tchannel-go/channel.go:269 +0x4b3

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x45c2d3
rdx    0x0
rdi    0xf23f58
rsi    0x0
rbp    0x7f111938add8
rsp    0x7f111938ad90
r8     0x0
r9     0x0
r10    0x7f111938adc8
r11    0x246
r12    0x0
r13    0x7ffee299717f
r14    0x7f111938b9c0
r15    0x7ffee2997200
rip    0x45c2d3
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
*** Test killed: ran too long (10m0s).
FAIL	github.com/uber/zanzibar/test	2973.475s
ok  	github.com/uber/zanzibar/test/clients/bar	0.945s
ok  	github.com/uber/zanzibar/test/clients/baz	0.839s
ok  	github.com/uber/zanzibar/test/config	0.005s
ok  	github.com/uber/zanzibar/test/endpoints/bar	1.215s
ok  	github.com/uber/zanzibar/test/endpoints/baz	1.354s
ok  	github.com/uber/zanzibar/test/endpoints/contacts	0.054s
ok  	github.com/uber/zanzibar/test/endpoints/googlenow	0.390s
ok  	github.com/uber/zanzibar/test/endpoints/tchannel/baz	0.044s
?   	github.com/uber/zanzibar/test/lib	[no test files]
?   	github.com/uber/zanzibar/test/lib/bench_gateway	[no test files]
?   	github.com/uber/zanzibar/test/lib/test_backend	[no test files]
?   	github.com/uber/zanzibar/test/lib/test_gateway	[no test files]
?   	github.com/uber/zanzibar/test/lib/test_m3_server	[no test files]
?   	github.com/uber/zanzibar/test/lib/util	[no test files]

define behavior for unset transform

when we do transform:

foo.a.b -> bar.c.d
in foo: {a: field, b: field} -> bar {c: field, d: field}

it's possible that some field don't have assignment from proxy / overridden input, the behavior for such cases are not deterministic

Overlap between ExportType and CustomClientType for custom client

For custom clients, the CustomClientType and ExportType in client spec struct seem to mean the same thing, the client's type when it's referred in other packages. Having two different fields for the same thing is bad, and the worse part is the discrepancy between how the field is set: CustomClientType is read from client config while ExportType is always set as {Name}Client.

Zanzibar client generation should respect client config

Currently the generated client code sometimes uses the thrift file as source of truth and sometimes the client-config.json.

This leads to broken generated code if the client-config.json and the thrift file do not use the exact same string literals.

Zanzibar http+client improvements

Add support for query params
Add support for multiple thrift exceptions with the same status code.
Add support for a zanzibar.HTTPError instead of returning "Unexpected status code"

improve header transform and header propagate

currently zanzibar operates headers in following 3 ways:

1: in endpoint thrift schema configure zanzibar.http.reqHeaders = "header1,header2"
this schema annotation will invoke checkHeaders when endpoint handling an incoming request, and consider all specified endpoint mandatory required, if any of the headers not presented, it will fail the incoming request

2: in endpoint config json configure header propagate, which introduced in #346, that allows headers to be used as request field to downstream client call

3: in endpoint config json configure {req|res}HeaderMap, to extract headers from incoming endpoint request, and (if exist) propagate to client request headers with mapped naming

a couple of problems:

1: no single source of truth for headers
2: no schema for header map and necessary required / optional annotation
3: loose check around headers

going to address this with a header schema dot thrift files that
1: imported from include of endpoint schema
2: annotated in endpoint annotations
3: used as source of truth for headers check / validation

Detect goroutine leaks in integration tests

https://github.com/uber-go/goleak

Resque alike workerpool with redis(?) to queue up repo update jobs instead of lock

currently updating files in repo cannot scale with multiple users with high update traffic, we'd like to implement a workerpool and redis to back jobs exec, this should support us with the growth of use in next 6 months

Method request/response type does not cover thrift basic and container types

The SetResponseType and SetRequestType methods on MethodSpec assumes the Go type is the same as the thrift type, which is only correct for struct type and a couple of basic thrift types. The example thrifts don't cover all thrift types, a thorough thrift with all thrift types is needed to justify the implementation.

typedef in ThriftSpec not handled correctly in zanzibar

typedef Thrift aliases are treated as custom types in the compiled.Field golang structs instead of as at the underlying type. This breaks code generation.

A common pattern is alias UUID as strings and returning lists of UUIDs and maps keyed by UUIDs. Code generation in zanzibar breaks because it does not understand the UUID fields are string types.

Logo design

We should get a zanzibar logo designed to start building a brand/identity around the project.

Don't rely on dev branches

Hi,

Please stop relying on development branches of ThriftRW Go. If it's not on
master, you shouldn't be relying on it. If you need a feature from a
development branch, please work with us to get it merged into master.

Context: thriftrw/thriftrw-go#305 (comment)

CC @breerly @blampe @Raynos @ChuntaoLu

Make generate does not remove obselete files generated by easyjson

Reproduce steps:

add a new method to a thrift file;
make generate and commit changes;
removes the method in step 1 from the thrift file;
make generate and the esayjson files are still present.

Pin linters in glide

https://github.com/uber/zanzibar/blob/master/Makefile#L78

[proposal] Custom client methods for non-conforming HTTP endpoints

It's inevitable that some downstream services will not implement conforming HTTP+JSON or tchannel+thrift endpoints and we will require some way to talk to them.

The current prospect is that people will write full custom clients by hand, which will side-step the instrumentation we have clients and bloat the dependency tree.

After some discussion offline, we're proposing the ability to decorate a client method with a custom annotation so that type adapters for endpoints can be implemented by hand. The idea is that instead of calling into the thrift-generated (de)serialization code, we allow a pattern for custom type serializers.

This would suggest that clients are structured with pluggable type coercion/serialization code, which allows us to support arbitrary encodings for migration purposes, and offers a "need it tomorrow" solution for integrating with existing services, before they can be migrated to a standard thrift interface.

Serve /debug endpoints with a different internal http server

Currently the /debug endpoints are served by the same server as the one serving business endpoints. If Zanzibar server is exposed directly to public, then these endpoints will be a security concern. In that sense, it's safer to serve /debug endpoints with a different unexposed internal sever.

Generalize endpoint workflow constructor parameter

Currently the workflow constructor has two parameters: ClientDependencies and logger, they should be generalized to be just one dependency parameter.

Test transformRequest middleware

{
  "log": "2017/11/22 01:39:46 http: panic serving 10.69.9.27:39308: interface conversion: interface {} is []interface {}, not []map[string]interface {}\n",
  "stream": "stderr",
  "time": "2017-11-22T01:39:46.668796196Z"
}
{
  "log": "goroutine 288294 [running]:\n",
  "stream": "stderr",
  "time": "2017-11-22T01:39:46.668848547Z"
}
{
  "log": "net/http.(*conn).serve.func1(0xc421698320)\n",
  "stream": "stderr",
  "time": "2017-11-22T01:39:46.668860063Z"
}
{
  "log": "\t/usr/lib/go-1.9/src/net/http/server.go:1697 +0xd0\n",
  "stream": "stderr",
  "time": "2017-11-22T01:39:46.668867196Z"
}
{
  "log": "panic(0xa77e80, 0xc420557200)\n",
  "stream": "stderr",
  "time": "2017-11-22T01:39:46.668873826Z"
}

There will probably be a panic with interface conversion.

Make example-gateway production.json the source of truth

Currently, running make test-update over-writes the examples/example-gateway/config/production.json config file with values from backend/repository/data/client/baz_client_update.json.

Please make examples/example-gateway/config/production.json the one source of truth for config settings.

HTTP+JSON proxy endpoint : add support for non-struct return type

Currently the proxy code generation assumes either void or struct return type.

There is a string-return-type branch demonstrating this issue

If we use a string return type we get a panic like:

raynos at raynos-ThinkPad-T440p  
~/gocode/src/github.com/uber/zanzibar on string-return-type*
$ make generate
Compiled thriftrw : +1
Generating Go code from Thrift files
Generated structs : +2
Compiled easyjson : +2
Generating JSON Marshal/Unmarshal
Generated structs : +21
Generating module system components:
Generating     http   client bar                  in build/clients/bar              1/12
Generating tchannel   client baz                  in build/clients/baz              2/12
Generating     http   client contacts             in build/clients/contacts         3/12
Generating tchannel   client corge                in build/clients/corge            4/12
Generating     http   client google-now           in build/clients/google-now       5/12
Generating   custom   client quux                 in build/clients/quux             6/12
Generating     http endpoint bar                  in build/endpoints/bar            7/12
panic: interface conversion: compile.TypeSpec is *compile.StringSpec, not *compile.StructSpec

goroutine 1 [running]:
github.com/uber/zanzibar/codegen.(*MethodSpec).setTypeConverters(0xc420485600, 0xc42046aaf0, 0xc42041c0f0, 0x0, 0x0, 0xc4202bc1b0, 0x0, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/codegen/method.go:790 +0x2fd
github.com/uber/zanzibar/codegen.(*ModuleSpec).SetDownstream(0xc4202bbf80, 0xc4205d1d30, 0x4, 0xc4205d1d36, 0xa, 0xc420350b40, 0xc4205d1e10, 0xa, 0x0, 0x0, ...)
	/home/raynos/gocode/src/github.com/uber/zanzibar/codegen/service.go:226 +0x725
github.com/uber/zanzibar/codegen.(*EndpointSpec).SetDownstream(0xc420435800, 0xc42031f1b8, 0x1, 0x1, 0xc4202bc1b0, 0x0, 0x0)
	/home/raynos/gocode/src/github.com/uber/zanzibar/codegen/gateway.go:820 +0x1d3
github.com/uber/zanzibar/codegen.(*EndpointGenerator).Generate(0xc42038f700, 0xc4203cf950, 0xc4203de87c, 0x4, 0xc4203bc1a8)
	/home/raynos/gocode/src/github.com/uber/zanzibar/codegen/module_system.go:605 +0x57b
github.com/uber/zanzibar/codegen.(*ModuleSystem).GenerateBuild(0xc42027fb00, 0xc420019100, 0x31, 0xc42001aa50, 0x49, 0xc42001ac30, 0x4f, 0x0, 0xc420019140, 0x40)
	/home/raynos/gocode/src/github.com/uber/zanzibar/codegen/module.go:911 +0x706
main.main()
	/home/raynos/gocode/src/github.com/uber/zanzibar/codegen/runner/runner.go:107 +0x926
exit status 2
Makefile:84: recipe for target 'generate' failed
make: *** [generate] Error 1

It looks like there is a TODO in the code for this edge case ( https://github.com/uber/zanzibar/blob/string-return-type/codegen/method.go#L782 ).

A mismatch between client & endpoint thrift exceptions generates invalid code

It should return a human readable validation error instead of generating invalid go code and failing a linter check.

Add support for Thrift typedefs

Currently typedefs cause type errors when mapped to fields with the same underlying type or used as keys in maps or elements in lists.

Test hangs when a stdout print is added to a client initializer

Reproduce steps:

add fmt.Println("test") in here.
run test bar_bar_method_argnotstruct_test.go.
test process hangs

I suspect it has something to do with how the child process test gateway works.

`go test -bench` has some kind of socket/fd leak

When running make bench for a non-trivial number of seconds sometimes we get a socket/fd leak.

This means the first 10,000 iterations of the benchmark are fast and then it gets really slow.

I looked into this and figured out how to use dlv but didnt root cause it.
I suspect our usage of keep alive http is broken ...

This only happens in make bench and not benchmarks/runner/runner.

I suspect the http client in test/lib/bench_gateway is somehow broken.

go test -c -v -run _NONE_ -bench . -benchmem -cpuprofile=cpu.prof -benchtime 6s -cpu 2 ./test/endpoints/google_now/google_now_test.go ;
dlv exec ./google_now.test -- -test.v -test.run _NONE_ -test.bench . -test.benchmem -test.cpuprofile=cpu.prof -test.benchtime 6s -test.cpu 2

correct moduleSpec SetDownstream signature

moduleSpec setDownstream

the fact that endpointSpec passes so many its internal properties to ModuleSpec for setdownstream seems a failure of isolation between classModuleSpec and class EndpointSpec

if ModuleSpec depend so much on EndpointSpec to do set downstream job (and seems like its only job), we'd rather just pass EndpointSpec as its anchor param

again a proper fix will require some serious look at these 2 classes function / responsibility

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.