aporeto-inc / trireme-lib Goto Github PK

View Code? Open in Web Editor NEW

298.0 298.0 51.0 37.43 MB

Simple, scalable and secure application segmentation

Home Page: https://trireme.io

License: Apache License 2.0

Go 99.23% C 0.06% Shell 0.68% Makefile 0.03%

apereto

trireme-lib's People

Stargazers

Watchers

trireme-lib's Issues

pod monitor

Hello, I saw your txt file under monitor/internal/kubernetes saying that the kubetnetes monitor has been deprecated, replacing it with the pod monitor, then I tried using the pod monitor and found the error, because The monitor configuration is much less. You didn't provide some default configuration of the pod monitor, but wrote a few interfaces. Is it necessary for us to implement it ourselves? Will you provide it in the future?

Logs: Received application ACK packet in the wrong state! 2

Looking at the logs, I can find a lot of those:

{"level":"debug","ts":1497986446.561756,"caller":"enforcer/datapath_tcp.go:192","msg":"Dropping packet ","flow":"10.8.1.19:10.8.1.18:80:59642","Flags":".AP...","error":"Received application ACK packet in the wrong state! 2"}

Is this project supported for use on Openshift Origin?

This project has show the most potential for network isolation for microservices we are deploying to Openshift. It was featured on the openshift blog, however it was demo'd under kubernetes. We are trying to deploy this to the latest release of Openshift Origin, currently v.1.5.0-alpha. We were able to create the PSK secret, and the DaemonSet deploys, but the pods never fully spin up.

oc v1.4.1+3f9807a
kubernetes v1.4.0+776c994
features: Basic-Auth

openshift v1.5.0-alpha.2+e4b43ee
kubernetes v1.5.2+43a9be4

Failed validation Error validating pod trireme-5vfgm.demo from api, ignoring: spec.containers[0].env[4].valueFrom.fieldRef.fieldPath: Unsupported value: "spec.host": supported values: metadata.name, metadata.namespace, status.podIP

It seems that the Openshift API does not accept the same values that k8s does. Is there any evidence that this project is supported to run on Openshift, or are there future plans to test / develop for use on the Openshift platform? We would absolutely use it.

encrypt policy rule

When I configure the first policy p1, the action is allow. The second policy p2, the action is allow and encrypted, then the question is, why is there this comment when searching for the policy, don’t overwrite allow? If so, then the policy id of the encrypted policy p2 will not be matched , But the policy id of the first p1

[REFACTORING] linter is currently deactivated for some unit tests

Right now, the gometalinter is deactivated for some unittests.

The method deactivated are marked with the following comments :

// TODO: remove nolint
// nolint
func (p *Packet) AddTCPLayer(srcPort layers.TCPPort, dstPort layers.TCPPort) error {

Deactivated methods are in the packages : packetgen and cgnetcls.
Reactivated the test will probably needs some refactoring

Metadata extractor for SELinux Labels

Requested behavior

The Trireme example can associate with the Trireme identity the SELInux labels of the
process that is being executed. This will help us to define SELinux based network
policies that span hosts, similar to what SELinux was trying to achieve with much
more complex implementations.

Expected behavior

When a process is instantiated through Trireme, the SELinux labels associated with
the process are captured and become part of the identity.

I can then use these SELinux labels to define a policy. For example if a file
has the labels: system_u:object_r:user_a these labels can become part of an authorization
policy.

Index out of range on delete namespace in Trireme

Got this error on a remote with Trireme when an ongoing delete was happening for that container:
(Triggered with Trireme-Kubernetes)

[0fd74fad21ab]:panic: runtime error: index out of range
[0fd74fad21ab]:
[0fd74fad21ab]:goroutine 23 [running]:
[0fd74fad21ab]:github.com/aporeto-inc/trireme-kubernetes/vendor/github.com/aporeto-inc/trireme/enforcer.(*Datapath).startApplicationInterceptor.func1(0xc4201b06e0, 0xc420531000, 0x10, 0x10, 0xc420c049ec, 0xc4216a0000)
[0fd74fad21ab]:	/home/travis/gopath/src/github.com/aporeto-inc/trireme-kubernetes/vendor/github.com/aporeto-inc/trireme/enforcer/nfq_linux.go:89 +0xaf
[0fd74fad21ab]:created by github.com/aporeto-inc/trireme-kubernetes/vendor/github.com/aporeto-inc/trireme/enforcer.(*Datapath).startApplicationInterceptor
[0fd74fad21ab]:	/home/travis/gopath/src/github.com/aporeto-inc/trireme-kubernetes/vendor/github.com/aporeto-inc/trireme/enforcer/nfq_linux.go:93 +0x424

ContextStore Upgrade

Actual behavior

Description of the actual behavior.

Expected behavior

Description of the expected behavior.

Steps to reproduce

Description of the various steps required to reproduce the error.

Solution proposal

Description of what you thingk would need to be done.
The context store today will not be able to handle format changes between restarts or upgrade or trireme.
If we add fields to the context store and restart we will not be able to recover PU's.
Need to add this capability to the contextstore

Installation type

console.aporeto.com
on-prem
dev

Version: ?
Customer: ?
ETA: ?

Kubernetes: unable to find pu id associated given context id

Actual behavior

Description of the actual behavior.
Deployed Trireme using the "Getting Started" guide in: https://github.com/aporeto-inc/trireme-kubernetes and seeing a ton of errors"

[4388f9e84a6a]:ERROR	destNFLogsHandler: create flow record	{"error": "nflog: unable to find pu id associated given context id: 4388f9e84a6a"}

Expected behavior

Description of the expected behavior.

Steps to reproduce

Description of the various steps required to reproduce the error.
Deploy using https://github.com/aporeto-inc/trireme-kubernetes

Solution proposal

Description of what you thingk would need to be done.

Installation type

console.aporeto.com
on-prem
dev

Version: ?
Customer: ?
ETA: ?

metadataExtractor for each Monitor type should be packaged with the monitor type

Having all the extractors on a completely separate package is weird to me and doesn't seen particulartly a good pattern as it distribute almost redundant code accross the monitor and extractor inplementation all over the place (such as helper functions).

I would suggest to move the extractors for each type in the same package as the monitor implementation itself

unable to fetch the new repository where in the prismacloud

Actual behavior

go mod tidy comand with update the depends

Expected behavior

update the trireme-lib deps

Steps to reproduce

git clone https://github.com/aporeto-inc/trireme-lib.git
go mod tidy

Solution proposal

share the base libary

pu runtime

Hello, may I ask: how to get the pid field in the puruntime structure returned by the metadata extractor in the pod monitor?

BUG: Need to retry RPC call when resource temporarily unavailable

During a soaking run, I ran into the following error:

ERROR	Failed to execute command run: Cannot connect to policy process dial unix /var/run/trireme.sock: connect: resource temporarily unavailable	{"error": "Cannot connect to policy process dial unix /var/run/trireme.sock: connect: resource temporarily unavailable"}

Talking with @amitlimaye, the root cause was narrowed down to the following line in cmd/systemdutil/systemdutil.go:

client, err := net.Dial("unix", rpcmonitor.DefaultRPCAddress)

We need to retry here if the error is due to resource temporarily unavailable, what ever error code that resolves to.

unable to compile the code on rhel7/go1.4.2

make build

CGO_ENABLED=1 go build -a -installsuffix cgo
/local/golang/src/github.com/docker/docker/client/hijack.go:15:2: no buildable Go source files in /local/golang/src/github.co
make: *** [build] Error 1

code seems not to compile either on 1.5 + due to circular import issues..

Validate length of the ContextID// PUID in Trireme

As a general rule, the ContextID is coming straight from the Monitor//Extractor.

The Supervisor and Enforcers are crashing out random errors if the Length of the ContextID is "Too Long".

Keeping this as a placeHolder to refactor the way we handle the ContextID.

About using your simple example trireme-example controller to instantiate the problem. . .

Controller instantiation in trireme-example/triremecli/cli.go: line 102
// Initialize the controllers
ctrl := controller.New("ExampleNode", controllerOptions...)

But the new lib library new is not the same as the latest lib library config structure field.

before：

now：

Why is this field necessary? I think controller.New() --> newTrireme(c) --> t.newEnforcers() -> enforcer.New() --> nfqdatapath.New(***) --> d.SetTargetNetworks(cfg) --> cfg.TCPTargetNetworks This throws a pannic. Because cfg is nil. This is when the config structure is instantiated, no assignment is made.

data race happening randomly

[46c8cb939ace]:time="2017-04-01T00:27:27Z" level=info msg="Supervise status remote_enforcer " error="exit status 2: iptables v1.6.0: host/network `' not found
[46c8cb939ace]:Try `iptables -h' or 'iptables --help' for more information.
[46c8cb939ace]:" method=Supervise package="remote_enforcer" 
[46c8cb939ace]:==================
[46c8cb939ace]:WARNING: DATA RACE
[46c8cb939ace]:Read at 0x00c4202a2910 by goroutine 30:
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).processApplicationPacketsFromNFQ()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:442 +0x28a
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).StartApplicationInterceptor.func1()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:366 +0xcd
[46c8cb939ace]:
[46c8cb939ace]:Previous write at 0x00c4202a2910 by goroutine 26:
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).processApplicationPacketsFromNFQ()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:442 +0x2a4
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).StartApplicationInterceptor.func1()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:366 +0xcd
[46c8cb939ace]:
[46c8cb939ace]:Goroutine 30 (running) created at:
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).StartApplicationInterceptor()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:369 +0x29c
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).Start()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:299 +0x2b2
[46c8cb939ace]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).InitEnforcer()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:112 +0x5f3
[46c8cb939ace]:  runtime.call128()
[46c8cb939ace]:      /usr/local/go/src/runtime/asm_amd64.s:516 +0x65
[46c8cb939ace]:  reflect.Value.Call()
[46c8cb939ace]:      /usr/local/go/src/reflect/value.go:302 +0xc0
[46c8cb939ace]:  net/rpc.(*service).call()
[46c8cb939ace]:      /usr/local/go/src/net/rpc/server.go:387 +0x257
[46c8cb939ace]:
[46c8cb939ace]:Goroutine 26 (running) created at:
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).StartApplicationInterceptor()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:369 +0x29c
[46c8cb939ace]:  github.com/aporeto-inc/trireme/enforcer.(*datapathEnforcer).Start()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/enforcer/datapath.go:299 +0x2b2
[46c8cb939ace]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).InitEnforcer()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:112 +0x5f3
[46c8cb939ace]:  runtime.call128()
[46c8cb939ace]:      /usr/local/go/src/runtime/asm_amd64.s:516 +0x65
[46c8cb939ace]:  reflect.Value.Call()
[46c8cb939ace]:      /usr/local/go/src/reflect/value.go:302 +0xc0
[46c8cb939ace]:  net/rpc.(*service).call()
[46c8cb939ace]:      /usr/local/go/src/net/rpc/server.go:387 +0x257
[46c8cb939ace]:==================

Compilation error in trireme-example

I checked out trireme-example on Fedora 28 using git as below -
'git clone https://github.com/aporeto-inc/trireme-example.git'

Then following all the steps mentioned in README.MD - https://github.com/aporeto-inc/trireme-example/blob/master/README.md , I am trying to compile using the specified command ' make build'.

I am getting the following compilation error -

(69/69) Wrote github.com/docker/[email protected]

github.com/aporeto-inc/trireme-example/vendor/go.aporeto.io/netlink-go/nflog

vendor/go.aporeto.io/netlink-go/nflog/nflog.go:287:31: too many arguments in call to packet.New
have (number, []byte, string, bool)
want (uint64, []byte, string)
vendor/go.aporeto.io/netlink-go/nflog/nflog.go:293:37: cannot call non-function ipPacket.SourceAddress (type net.IP)
vendor/go.aporeto.io/netlink-go/nflog/nflog.go:294:42: cannot call non-function ipPacket.DestinationAddress (type net.IP)
vendor/go.aporeto.io/netlink-go/nflog/nflog.go:296:34: cannot call non-function ipPacket.IPProto (type uint8)
vendor/go.aporeto.io/netlink-go/nflog/nflog.go:297:24: ipPacket.IPTotalLen undefined (type *packet.Packet has no field or method IPTotalLen)
vendor/go.aporeto.io/netlink-go/nflog/nflog.go:298:36: cannot call non-function ipPacket.SourcePort (type uint16)
vendor/go.aporeto.io/netlink-go/nflog/nflog.go:299:25: ipPacket.DestPort undefined (type *packet.Packet has no field or method DestPort)
make: *** [Makefile:23: build] Error 2

Please help to resolve this issue, as we are checking the feasibility of integrating trireme solution into security enhancements for cloud applications in our organisation

package import

go.aporeto.io/trireme-lib/controller/internal/

The following packages cannot be imported. Prompt not allowed

HandlePUEvent for the Die Event does not provide the actual PURuntime

When receiving a HandlePuEvent from the Docker Monitor, the PURuntime provided is a dummy deafult one as can be seen in this code from the docker monitor.

//handleDie event is called when a container dies. It generates a "Stop" event.
func (d *DockerMonitor) handleDieEvent(ctx context.Context, event *events.Message) error {

	puID, err := puIDFromDockerID(event.ID)
	if err != nil {
		return err
	}

	return d.config.Policy.HandlePUEvent(ctx, puID, tevents.EventStop, policy.NewPURuntimeWithDefaults())
}

As such, it makes it more difficult to extract data from the resolver without keeping a cache.

remoteEnforcer exec

remoteEnforcerTempBuildPath = "/var/run/aporeto/tmp/bin/"

在linux里，这个目录下执行文件是不允许的？？？报出权限被拒绝。.

trirme-example not working as documented with container labels

Am running trireme-example as process on rhel7 box running docker 1.12-cs3 and the step i took are as below.

start docker engine.
run trireme-example binary as process(_not as container) on the host as below
/local/trireme/trireme-example -log_dir /var/log/trireme -v=10
run a python simhttpserver as a container with label=web
docker run -l app=web -d -t qa.registry.docker.xxxx.xx.com/cpe/gs-linux6 python -m SimpleHTTPServer 80
run another container with different label as documented
docker run -i -l app=database -t qa.registry.docker.xxxx.xx.com/cpe/xx-linux6 /bin/bash
curl is able to access it. log to follow

trireme-example does not see packets from containers on overlay network and with macvlan interfaces

trireme standalone does not see any packets if a container is on docker overlay network or uses macvlan interfaces.

simple swarm overlay network setup as described in docker documentation can be used to reproduce the issue.

do we need different policy config for correct nfqueue chains? or it agnostic of policy configuration the prerouting chain should see the packet?

orchestration agnostic policy definition

Network policy definition should be orchestration agnostic. Though trireme supports k8s policy definition which makes it simpler to define policies. should be usefull to have simpler independent policy definition to support k8s,swarm etc..

Trireme with weave Net

From the documentation, Trireme works with Flannel. Does it work with Weave Net?

Add a new monitor for Kubernetes.

Add a new monitor for Kubernetes, to be used directly by projects plugging into Kubernetes such as Trireme-Kubernetes and the Enforcer launched on Kubernetes

At some point, I get Too Many Links errors everytime I start Trireme.

WARN iptablesctrl/acls.go:618 Can not delete the chain {"context": "mangle", "section": "OUTPUT", "error": "exit status 1: iptables: Too many links.\n"}

Failed to set rmem/wmem

Using version f735be9

We are failing to set net.core.rmem_max=63553920 and net.core.wmem_max=63553920 for each PU that is being created.

{"level":"info","ts":1497550983.161677,"caller":"policy/engine.go:149","msg":"Processing unit created","ID":"5942d087291ebb77c5713b7c","nativeID":"","name":"nginx3_a1961413-5e61-4563-8545-d5b809254a20_2_0"}
{"level":"info","ts":1497550983.5033548,"caller":"policy/engine.go:370","msg":"Processing unit started","ID":"5942d087291ebb77c5713b7c","nativeID":"df1ad23aa40d","name":"nginx3_a1961413-5e61-4563-8545-d5b809254a20_2_0"}
[df1ad23aa40d]:{"level":"info","ts":1497550983.6095226,"caller":"remoteenforcer/remoteenforcer_linux.go:181","msg":"Using PKI Compact Secrets"}
[df1ad23aa40d]:{"level":"error","ts":1497550983.6360884,"caller":"enforcer/datapath.go:129","msg":"Failed to set rmem","error":"exit status 255"}
[df1ad23aa40d]:{"level":"error","ts":1497550983.6386921,"caller":"enforcer/datapath.go:134","msg":"Failed to set wmem","error":"exit status 255"}

Happens for every container launched thus far.

Question about your go-ipset library reference log library

About your library: go-ipset/ipset/ipset.go go: line 28
You quoted an uppercase log library "github.com/Sirupsen/logrus", but the author later changed the library name to lowercase, which led us to throw some fatal errors all the time!

And your other library: oxy/forward/fwd.go go: line 23 is a lowercase name that references the log library: log "github.com/sirupsen/logrus"

This will lead to conflict. I hope that you can synchronize the two libraries.

protocol == 6

	for _, proto := range rule.Protocols {
		// TODO: 有疑惑。设置受trireme保护的网络作用域会调用此方法，而且协议等于6，但是生成puContext也会调用此方法，协议不可能等于6。这个单一判断无法同时满足。。
		//if strings.ToLower(proto) != constants.TCPProtoNum {
		//	continue
		//}
		for _, address := range rule.Addresses {
			for _, port := range rule.Ports {
				if err := addCache(address, port); err != nil {
					return err
				}
			}
		}
	}

trireme openshift origin v1.4.0

% oc version
oc v1.4.1+3f9807a
kubernetes v1.4.0+776c994
features: Basic-Auth

openshift v1.4.0-rc1+b4e0954
kubernetes v1.4.0+776c994

Trying to run trireme on openshift using DaemonSets. It looks like to solution we are desperately in need of. Using the DaemonSet PSK, Ive created the trireme secret but get the following error in my pods when they spin up:

% oc logs po/trireme-2dybv
I0203 14:28:32.609910 6 main.go:27] Config used: &{KubeEnv:true AuthType:PSK KubeNodeName:node01 NodeAnnotationKey: PKIDirectory: KubeConfigLocation: TriremePSK:XXXXXXXXXX== TriremeNets:[10.0.0.0/8] ExistingContainerSync:true}
W0203 14:28:32.616409 6 client_config.go:481] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0203 14:28:32.617117 6 main.go:48] Starting Trireme PSK
I0203 14:28:32.644971 6 iptables.go:608] Can't clear PREROUTING iptables command
I0203 14:28:32.647501 6 iptables.go:608] Can't clear PREROUTING iptables command
I0203 14:28:32.648981 6 iptables.go:608] Can't clear PREROUTING iptables command
I0203 14:28:32.651453 6 iptables.go:608] Can't clear POSTROUTING iptables command
E0203 14:28:32.651664 6 datapath.go:224] Error unbinding existing NFQ handler from AfInet protocol family: operation not permitted

data races happening 100% of time when stopping Trireme

c8cb939ace]:==================
[46c8cb939ace]:WARNING: DATA RACE
[a274b1e1e157]:  github.com/aporeto-inc/trireme/supervisor.NewSupervisor()
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:77 +0x7a6
[46c8cb939ace]:Read at 0x00c4203abf92 by main goroutine:
[a274b1e1e157]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).InitSupervisor()
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:142 +0x15e
[46c8cb939ace]:  github.com/coreos/go-iptables/iptables.(*IPTables).runWithOutput()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:228 +0x16f
[46c8cb939ace]:  github.com/coreos/go-iptables/iptables.(*IPTables).run()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:221 +0x6c
[46c8cb939ace]:  github.com/coreos/go-iptables/iptables.(*IPTables).NewChain()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:187 +0x129
[46c8cb939ace]:  github.com/coreos/go-iptables/iptables.(*IPTables).ClearChain()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:193 +0x84
[a274b1e1e157]:  runtime.call128()
[46c8cb939ace]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLSection()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:751 +0xb0
[a274b1e1e157]:      /usr/local/go/src/runtime/asm_amd64.s:516 +0x65
[46c8cb939ace]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLs()
[a274b1e1e157]:  reflect.Value.Call()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:741 +0xf9
[46c8cb939ace]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).Stop()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:299 +0x201
[46c8cb939ace]:  github.com/aporeto-inc/trireme/supervisor.(*Config).Stop()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:142 +0x53
[46c8cb939ace]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).EnforcerExit()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:280 +0x60
[46c8cb939ace]:  main.EnforceOnly()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:543 +0x7a4
[46c8cb939ace]:  main.main()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:637 +0xb7
[46c8cb939ace]:
[46c8cb939ace]:Previous write at 0x00c4203abf90 by goroutine 42:
[46c8cb939ace]:  github.com/coreos/go-iptables/iptables.NewWithProtocol()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:74 +0x1f1
[46c8cb939ace]:  github.com/coreos/go-iptables/iptables.New()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:60 +0x33
[46c8cb939ace]:  github.com/aporeto-inc/trireme/supervisor/provider.NewGoIPTablesProvider()
[a274b1e1e157]:      /usr/local/go/src/reflect/value.go:302 +0xc0
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/provider/iptablesprovider.go:20 +0x2f
[46c8cb939ace]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.NewInstance()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:40 +0x33
[46c8cb939ace]:  github.com/aporeto-inc/trireme/supervisor.NewSupervisor()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:77 +0x7a6
[a274b1e1e157]:  net/rpc.(*service).call()
[46c8cb939ace]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).InitSupervisor()
[46c8cb939ace]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:142 +0x15e
[46c8cb939ace]:  runtime.call128()
[46c8cb939ace]:      /usr/local/go/src/runtime/asm_amd64.s:516 +0x65
[a274b1e1e157]:      /usr/local/go/src/net/rpc/server.go:387 +0x257
[a274b1e1e157]:
[a274b1e1e157]:Goroutine 42 (finished) created at:
[a274b1e1e157]:  net/rpc.(*Server).ServeCodec()
[a274b1e1e157]:      /usr/local/go/src/net/rpc/server.go:481 +0x496
[a274b1e1e157]:  net/rpc.(*Server).ServeConn()
[46c8cb939ace]:  reflect.Value.Call()
[a274b1e1e157]:      /usr/local/go/src/net/rpc/server.go:458 +0x62b
[a274b1e1e157]:  net/rpc.(*Server).ServeHTTP()
[46c8cb939ace]:      /usr/local/go/src/reflect/value.go:302 +0xc0
[46c8cb939ace]:  net/rpc.(*service).call()
[46c8cb939ace]:      /usr/local/go/src/net/rpc/server.go:387 +0x257
[46c8cb939ace]:
[46c8cb939ace]:Goroutine 42 (finished) created at:
[46c8cb939ace]:  net/rpc.(*Server).ServeCodec()
[46c8cb939ace]:      /usr/local/go/src/net/rpc/server.go:481 +0x496
[46c8cb939ace]:  net/rpc.(*Server).ServeConn()
[46c8cb939ace]:      /usr/local/go/src/net/rpc/server.go:458 +0x62b
[46c8cb939ace]:  net/rpc.(*Server).ServeHTTP()
[46c8cb939ace]:      /usr/local/go/src/net/rpc/server.go:700 +0x506
[46c8cb939ace]:  net/http.(*ServeMux).ServeHTTP()
[46c8cb939ace]:      /usr/local/go/src/net/http/server.go:2238 +0xa2
[46c8cb939ace]:  net/http.serverHandler.ServeHTTP()
[46c8cb939ace]:      /usr/local/go/src/net/http/server.go:2568 +0xbc
[46c8cb939ace]:  net/http.(*conn).serve()
[46c8cb939ace]:      /usr/local/go/src/net/http/server.go:1825 +0x71a
[46c8cb939ace]:==================
[a274b1e1e157]:      /usr/local/go/src/net/rpc/server.go:700 +0x506
[a274b1e1e157]:  net/http.(*ServeMux).ServeHTTP()
[a274b1e1e157]:      /usr/local/go/src/net/http/server.go:2238 +0xa2
[a274b1e1e157]:  net/http.serverHandler.ServeHTTP()
[a274b1e1e157]:      /usr/local/go/src/net/http/server.go:2568 +0xbc
[a274b1e1e157]:  net/http.(*conn).serve()
[a274b1e1e157]:      /usr/local/go/src/net/http/server.go:1825 +0x71a
[a274b1e1e157]:==================

[ca41dd3ab346]:==================
[ca41dd3ab346]:WARNING: DATA RACE
[ca41dd3ab346]:Read at 0x00c4202787d2 by main goroutine:
[ca41dd3ab346]:  github.com/coreos/go-iptables/iptables.(*IPTables).runWithOutput()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:228 +0x16f
[ca41dd3ab346]:  github.com/coreos/go-iptables/iptables.(*IPTables).run()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:221 +0x6c
[ca41dd3ab346]:  github.com/coreos/go-iptables/iptables.(*IPTables).NewChain()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:187 +0x129
[ca41dd3ab346]:  github.com/coreos/go-iptables/iptables.(*IPTables).ClearChain()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:193 +0x84
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLSection()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:751 +0xb0
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLs()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:741 +0xf9
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).Stop()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:299 +0x201
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/supervisor.(*Config).Stop()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:142 +0x53
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).EnforcerExit()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:280 +0x60
[ca41dd3ab346]:  main.EnforceOnly()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:543 +0x7a4
[ca41dd3ab346]:  main.main()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:637 +0xb7
[ca41dd3ab346]:
[ca41dd3ab346]:Previous write at 0x00c4202787d0 by goroutine 42:
[ca41dd3ab346]:  github.com/coreos/go-iptables/iptables.NewWithProtocol()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:74 +0x1f1
[ca41dd3ab346]:  github.com/coreos/go-iptables/iptables.New()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:60 +0x33
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/supervisor/provider.NewGoIPTablesProvider()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/provider/iptablesprovider.go:20 +0x2f
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.NewInstance()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:40 +0x33
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/supervisor.NewSupervisor()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:77 +0x7a6
[ca41dd3ab346]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).InitSupervisor()
[ca41dd3ab346]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:142 +0x15e
[ca41dd3ab346]:  runtime.call128()
[ca41dd3ab346]:      /usr/local/go/src/runtime/asm_amd64.s:516 +0x65
[ca41dd3ab346]:  reflect.Value.Call()
[ca41dd3ab346]:      /usr/local/go/src/reflect/value.go:302 +0xc0
[ca41dd3ab346]:  net/rpc.(*service).call()
[ca41dd3ab346]:      /usr/local/go/src/net/rpc/server.go:387 +0x257
[ca41dd3ab346]:
[ca41dd3ab346]:Goroutine 42 (finished) created at:
[ca41dd3ab346]:  net/rpc.(*Server).ServeCodec()
[ca41dd3ab346]:      /usr/local/go/src/net/rpc/server.go:481 +0x496
[ca41dd3ab346]:  net/rpc.(*Server).ServeConn()
[ca41dd3ab346]:      /usr/local/go/src/net/rpc/server.go:458 +0x62b
[ca41dd3ab346]:  net/rpc.(*Server).ServeHTTP()
[ca41dd3ab346]:      /usr/local/go/src/net/rpc/server.go:700 +0x506
[ca41dd3ab346]:  net/http.(*ServeMux).ServeHTTP()
[ca41dd3ab346]:      /usr/local/go/src/net/http/server.go:2238 +0xa2
[ca41dd3ab346]:  net/http.serverHandler.ServeHTTP()
[ca41dd3ab346]:      /usr/local/go/src/net/http/server.go:2568 +0xbc
[ca41dd3ab346]:  net/http.(*conn).serve()
[ca41dd3ab346]:      /usr/local/go/src/net/http/server.go:1825 +0x71a
[ca41dd3ab346]:==================
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:40 +0x33
[46c8cb939ace]:  net/rpc.(*Server).ServeHTTP()
[46c8cb939ace]:      /usr/local/go/src/net/rpc/server.go:700 +0x506
[46c8cb939ace]:  net/http.(*ServeMux).ServeHTTP()
[46c8cb939ace]:      /usr/local/go/src/net/http/server.go:2238 +0xa2
[46c8cb939ace]:  net/http.serverHandler.ServeHTTP()
[46c8cb939ace]:      /usr/local/go/src/net/http/server.go:2568 +0xbc
[46c8cb939ace]:  net/http.(*conn).serve()
[46c8cb939ace]:      /usr/local/go/src/net/http/server.go:1825 +0x71a
[46c8cb939ace]:==================

[84cdf04e2c04]:==================
[84cdf04e2c04]:WARNING: DATA RACE
[84cdf04e2c04]:Read at 0x00c4203036f2 by main goroutine:
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:193 +0x84
[84cdf04e2c04]:  github.com/coreos/go-iptables/iptables.(*IPTables).runWithOutput()
[a274b1e1e157]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLSection()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:228 +0x16f
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:751 +0xb0
[84cdf04e2c04]:  github.com/coreos/go-iptables/iptables.(*IPTables).run()
[a274b1e1e157]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLs()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:221 +0x6c
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:741 +0xf9
[84cdf04e2c04]:  github.com/coreos/go-iptables/iptables.(*IPTables).NewChain()
[a274b1e1e157]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).Stop()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:187 +0x129
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:299 +0x201
[84cdf04e2c04]:  github.com/coreos/go-iptables/iptables.(*IPTables).ClearChain()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:193 +0x84
[a274b1e1e157]:  github.com/aporeto-inc/trireme/supervisor.(*Config).Stop()
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:142 +0x53
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLSection()
[a274b1e1e157]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).EnforcerExit()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:751 +0xb0
[e3d08ff45647]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:221 +0x6c
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:280 +0x60
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLs()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:741 +0xf9
[e3d08ff45647]:  github.com/coreos/go-iptables/iptables.(*IPTables).NewChain()
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).Stop()
[a274b1e1e157]:  main.EnforceOnly()
[e3d08ff45647]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:187 +0x129
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:543 +0x7a4
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:299 +0x201
[e3d08ff45647]:  github.com/coreos/go-iptables/iptables.(*IPTables).ClearChain()
[a274b1e1e157]:  main.main()
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/supervisor.(*Config).Stop()
[a274b1e1e157]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:637 +0xb7
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:142 +0x53
[a274b1e1e157]:
[e3d08ff45647]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:193 +0x84
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).EnforcerExit()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:280 +0x60
[84cdf04e2c04]:  main.EnforceOnly()
[00d98ff6e87e]:Previous write at 0x00c42029bff0 by goroutine 42:
[e3d08ff45647]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLSection()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:543 +0x7a4
[84cdf04e2c04]:  main.main()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/enforcerd/main.go:637 +0xb7
[00d98ff6e87e]:  github.com/coreos/go-iptables/iptables.NewWithProtocol()
[84cdf04e2c04]:
[84cdf04e2c04]:Previous write at 0x00c4203036f0 by goroutine 42:
[00d98ff6e87e]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:74 +0x1f1
[84cdf04e2c04]:  github.com/coreos/go-iptables/iptables.NewWithProtocol()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:74 +0x1f1
[84cdf04e2c04]:  github.com/coreos/go-iptables/iptables.New()
[00d98ff6e87e]:  github.com/coreos/go-iptables/iptables.New()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:60 +0x33
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/supervisor/provider.NewGoIPTablesProvider()
[00d98ff6e87e]:      /home/ubuntu/workspace/code/go/src/github.com/coreos/go-iptables/iptables/iptables.go:60 +0x33
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/provider/iptablesprovider.go:20 +0x2f
[00d98ff6e87e]:  github.com/aporeto-inc/trireme/supervisor/provider.NewGoIPTablesProvider()
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.NewInstance()
[00d98ff6e87e]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/provider/iptablesprovider.go:20 +0x2f
[00d98ff6e87e]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.NewInstance()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:40 +0x33
[e3d08ff45647]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:751 +0xb0
[00d98ff6e87e]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:40 +0x33
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/supervisor.NewSupervisor()
[00d98ff6e87e]:  github.com/aporeto-inc/trireme/supervisor.NewSupervisor()
[00d98ff6e87e]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:77 +0x7a6
[e3d08ff45647]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).cleanACLs()
[00d98ff6e87e]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).InitSupervisor()
[e3d08ff45647]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/acls.go:741 +0xf9
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:77 +0x7a6
[84cdf04e2c04]:  github.com/aporeto-inc/trireme/cmd/remoteenforcer.(*Server).InitSupervisor()
[84cdf04e2c04]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:142 +0x15e
[00d98ff6e87e]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/cmd/remoteenforcer/remoteenforcer_linux.go:142 +0x15e
[84cdf04e2c04]:  runtime.call128()
[84cdf04e2c04]:      /usr/local/go/src/runtime/asm_amd64.s:516 +0x65
[00d98ff6e87e]:  runtime.call128()
[00d98ff6e87e]:      /usr/local/go/src/runtime/asm_amd64.s:516 +0x65
[84cdf04e2c04]:  reflect.Value.Call()
[00d98ff6e87e]:  reflect.Value.Call()
[84cdf04e2c04]:      /usr/local/go/src/reflect/value.go:302 +0xc0
[84cdf04e2c04]:  net/rpc.(*service).call()
[84cdf04e2c04]:      /usr/local/go/src/net/rpc/server.go:387 +0x257
[84cdf04e2c04]:
[84cdf04e2c04]:Goroutine 42 (finished) created at:
[00d98ff6e87e]:      /usr/local/go/src/reflect/value.go:302 +0xc0
[00d98ff6e87e]:  net/rpc.(*service).call()
[84cdf04e2c04]:  net/rpc.(*Server).ServeCodec()
[84cdf04e2c04]:      /usr/local/go/src/net/rpc/server.go:481 +0x496
[84cdf04e2c04]:  net/rpc.(*Server).ServeConn()
[84cdf04e2c04]:      /usr/local/go/src/net/rpc/server.go:458 +0x62b
[84cdf04e2c04]:  net/rpc.(*Server).ServeHTTP()
[84cdf04e2c04]:      /usr/local/go/src/net/rpc/server.go:700 +0x506
[84cdf04e2c04]:  net/http.(*ServeMux).ServeHTTP()
[84cdf04e2c04]:      /usr/local/go/src/net/http/server.go:2238 +0xa2
[84cdf04e2c04]:  net/http.serverHandler.ServeHTTP()
[84cdf04e2c04]:      /usr/local/go/src/net/http/server.go:2568 +0xbc
[84cdf04e2c04]:  net/http.(*conn).serve()
[e3d08ff45647]:  github.com/aporeto-inc/trireme/supervisor/iptablesctrl.(*Instance).Stop()
[e3d08ff45647]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/iptablesctrl/iptables.go:299 +0x201
[e3d08ff45647]:  github.com/aporeto-inc/trireme/supervisor.(*Config).Stop()
[84cdf04e2c04]:      /usr/local/go/src/net/http/server.go:1825 +0x71a
[e3d08ff45647]:      /home/ubuntu/workspace/code/go/src/github.com/aporeto-inc/trireme/supervisor/supervisor.go:142 +0x53
[84cdf04e2c04]:==================

Add an intermediate log level (Trace) for Packet Headers

Actual behavior

All the Packet's header are printed in debug mode.

The goal is to add a Trace mode that will be a debug mode + Packet Header combined.

Long lived existing connections are disrupted by deploying Trireme-Lib

Actual behavior

Upon deploying Trireme-lib, it seems that existing connections sometimes fail

Expected behavior

They should be allowed to continue, or be closed depending on A policy

Steps to reproduce

Deploy Trireme-lib for a container that got a long-lived connection already setup.

panic: Failed to load Frontman.dll: The specified module could not be found.

How can I get a Frontman.dll？

BUG: Enforcing an enforcer that has already exited

While testing trireme, the test was removing PUs when it encountered the following:

{"level":"info","ts":1501555386.8419542,"caller":"policy/engine.go:507","msg":"Processing unit removed","ID":"597fe629df7696000117f505","nativeID":"/32344","name":"python-server-0c12212a-ac9c-457c-ae09-a3b2e1c5e09c-0-2"}
{"level":"info","ts":1501555386.8789887,"caller":"policy/engine.go:471","msg":"Processing unit stopped","ID":"597fe608df7696000117f43d","nativeID":"/31074","name":"python-server-0c12212a-ac9c-457c-ae09-a3b2e1c5e09c-0-0"}
[1a2a4263fd4b]:{"level":"fatal","ts":1501555386.9435802,"caller":"remoteenforcer/remoteenforcer_linux.go:377","msg":"Enforcer not inited"}
{"level":"error","ts":1501555386.9456024,"caller":"proxy/enforcerproxy.go:136","msg":"Failed to Enforce remote enforcer","error":"unexpected EOF"}
{"level":"warn","ts":1501555386.945639,"caller":"trireme/trireme.go:303","msg":"Re-initializing enforcers - connection lost"}
{"level":"error","ts":1501555386.9457128,"caller":"processmon/processmon.go:202","msg":"Cannot determine namespace of new container","error":"stat /proc/31846/ns/net: no such file or directory"}
panic: interface conversion: interface {} is nil, not *rpcwrapper.RPCHdl

goroutine 135 [running]:
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/enforcer/utils/rpcwrapper.(*RPCWrapper).DestroyRPCClient(0xc4201567e0, 0xc4212b8cc0, 0xc)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/enforcer/utils/rpcwrapper/rpc_handle.go:188 +0xac3
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/processmon.(*ProcessMon).KillProcess(0xc42000e070, 0xc4212b8cc0, 0xc)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/processmon/processmon.go:171 +0x2fd
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/supervisor/proxy.(*ProxyInfo).Unsupervise(0xc42019cc40, 0xc4212b8cc0, 0xc, 0xc42040d228, 0xc420261fc0)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/supervisor/proxy/supervisorproxy.go:81 +0xb6
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme.(*trireme).doHandleDelete(0xc4201a3680, 0xc4212b8cc0, 0xc, 0xb43711, 0x4)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/trireme.go:255 +0x3ce
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme.(*trireme).HandlePUEvent(0xc4201a3680, 0xc4212b8cc0, 0xc, 0xb43711, 0x4, 0x0, 0xb55b8f)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/trireme.go:94 +0x146
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor.(*dockerMonitor).stopDockerContainer(0xc42028d720, 0xc4212b8cc0, 0x40, 0x3, 0x1)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor/docker.go:514 +0x105
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor.(*dockerMonitor).handleDieEvent(0xc42028d720, 0xc421b04380, 0x8eb8f4, 0xc420339450)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor/docker.go:582 +0x42
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor.(*dockerMonitor).(github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor.handleDieEvent)-fm(0xc421b04380, 0xc42060e750, 0xc421a7cdbc)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor/docker.go:232 +0x34
github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor.(*dockerMonitor).eventProcessors.func1(0xc42028d720, 0x4)
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor/docker.go:319 +0x15b
created by github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor.(*dockerMonitor).eventProcessors
	/tmp/build/80754af9/go/src/github.com/aporeto-inc/enforcerd/vendor/github.com/aporeto-inc/trireme/monitor/dockermonitor/docker.go:334 +0x54

I see two issues here:

While removing PUs we hit a fatal due to s.Enforcer == nil in Server.Enforce, most probably due to us calling Server.EnforcerExit in Server.KillProcess. I am not entirely sure why these two ran into one another.
The panic was due to rpcHdl being set to nil when trying to convert it at this line:

if err := rpcHdl.(*RPCHdl).Client.Close(); err != nil {

We do not check if any error comes back from rpcHdl, _ := r.rpcClientMap.Get(contextID). Will make a PR for this.

Decoding 9 bytes of bad CBOR data can exhaust memory (BinaryJWTClaims in controller/pkg/tokens/binaryjwt.go)

Summary

Attempting to decode 9-10 bytes of malformed CBOR data (BinaryJWTClaims) causes
"fatal error: out of memory" or "runtime: out of memory" errors.

Only 1 decode attempt is needed to exhaust memory and cause these errors.

cc @sibicramesh @dstiliadis

Relevant code

trireme-lib/controller/pkg/tokens/binaryjwt.go

import (
...
"github.com/ugorji/go/codec"
...
)
...
func decode(buf []byte) (*BinaryJWTClaims, error) {
	// Decode the token into a structure.
	binaryClaims := &BinaryJWTClaims{}
	var h codec.Handle = new(codec.CborHandle)

	dec := codec.NewDecoderBytes(buf, h)
	if err := dec.Decode(binaryClaims); err != nil {

Actual behavior

Using the above decode function to decode 9-10 bytes of malformed CBOR data (BinaryJWTClaims) causes "fatal error: out of memory" or "runtime: out of memory" errors.

The malformed CBOR data simply needs to "lure a decoder into allocating very big data items (strings, arrays, maps) ..." as warned in RFC 7049 Section 8.

Expected behavior

One attempt to decode 9 bytes of malformed CBOR data (BinaryJWTClaims) should not be able to exhaust memory.

Steps to reproduce

Try to decode 9 bytes of malformed CBOR data described in RFC 7049 Section 8 (CBOR Security Considerations).

Examples of malformed CBOR data that can exhaust memory can be found on GitHub since September 2019 (possibly a lot earlier if you look beyond Go projects).

Solution proposal

trireme-lib should switch to a more secure CBOR library, which is what other projects did to fix this.

For more comparisons with ugorji/go, see fxamacker/cbor.

Background and Other Projects That Fixed This

In October 2013, RFC 7049 Section 8 (CBOR Security Considerations) warned that malformed CBOR data can be used to exhaust system resources.

Resource exhaustion attacks might attempt to lure a decoder into
allocating very big data items (strings, arrays, maps) or exhaust the
stack depth by setting up deeply nested items. Decoders need to have
appropriate resource management to mitigate these attacks.

In Sept 2019, oasislabs/oasis-core discovered tiny malformed CBOR data can exhaust memory and they traced the problem to the same CBOR library being used by trireme-lib (ugorji/go). They fixed it by switching to a more secure CBOR library.

In Feb 2020, smartcontractkit/chainlink had a CBOR security issue which was fixed by
a GitHub PR titled "Switch to more secure CBOR library". They were also using the same CBOR library as trireme-lib.

For info about CBOR and security, see Section 8 of RFC 7049 (Security Considerations).

bug/trireme-lib: ProcessingUnit not killed if remote enforcerd failed to launch

Actual behavior

When Trireme is unable to launch a remote enforcerd for a ProcessintUnit, the ProcessingUnit is not killed right after and remains running. It'll be killed in the next X (60sec) seconds though. (tested with a Docker container)

Expected behavior

The ProcessingUnit should be killed right after.

aporeto-inc / trireme-lib Goto Github PK

trireme-lib's People

Stargazers

Watchers

Forkers

trireme-lib's Issues

Requested behavior

Expected behavior

Actual behavior

Expected behavior

Steps to reproduce

Solution proposal

Installation type

Actual behavior

Expected behavior

Steps to reproduce

Solution proposal

Installation type

Actual behavior

Expected behavior

Steps to reproduce

Solution proposal

make build

github.com/aporeto-inc/trireme-example/vendor/go.aporeto.io/netlink-go/nflog

Actual behavior

Actual behavior

Expected behavior

Steps to reproduce

Summary

Relevant code

Actual behavior

Expected behavior

Steps to reproduce

Solution proposal

Background and Other Projects That Fixed This

Actual behavior

Expected behavior

Recommend Projects

Recommend Topics

Recommend Org