Git Product home page Git Product logo

martian's People

Contributors

0xflotus avatar a5malik avatar admtnnr avatar arnoldstoba avatar axamon avatar bbigras avatar bramhaghosh avatar brandonnelson3 avatar cpasillas avatar darist avatar davidkeay avatar dgiovann avatar gempesaw avatar hnovikov avatar hueich avatar jba avatar jeanbza avatar jmathes avatar justinrolston avatar kku1993 avatar konojunya avatar nbeloglazov avatar phaus avatar pytlesk4 avatar shawnps avatar surki avatar vangent avatar vivekv96 avatar wishdev avatar xuehuichao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

martian's Issues

README should focus on proxy toolkit rather than the example proxy implementation

Martian is, first and foremost, a set of tools to build custom HTTP/S proxies.

We've always had an example proxy implementation that we've maintained to show others how one could be built which has been confusing to users. We should change the README to present Martian as a toolkit for custom proxies rather than the documentation for the example implementation.

At some point we may want to move the testing proxy implementation to a separate repo and name it something else.

har: empty request timings

Would it be possible to have timings recorded as well, at least optionally?

Main use case for this is analysis of web site performance. It would be handy to be able to view the HAR as it would have been seen at the user's end (sans the browser level metrics of course).

https proxy

我需要代理访问https请求的网站,并且修改请求头,请求如何做?

Test failure on ARM

A test specifically fails on ARM arch:

ok  	github.com/google/martian	42.997s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/api"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
PASS
ok  	github.com/google/martian/api	0.019s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/auth"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
PASS
ok  	github.com/google/martian/auth	0.020s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/body"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
PASS
ok  	github.com/google/martian/body	0.020s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/cmd/proxy"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
2018/10/22 16:58:41 martian: starting proxy on [::]:44869 and api on [::]:46069
2018/10/22 16:58:41 martian: shutting down
2018/10/22 16:58:50 martian: starting proxy on [::]:35927 and api on [::]:33689
2018/10/22 16:58:50 martian: starting proxy on [::]:36391 and api on [::]:46289
2018/10/22 16:58:50 martian: shutting down
2018/10/22 16:58:50 martian: shutting down
--- FAIL: TestProxyMain (13.92s)
    --- FAIL: TestProxyMain/HttpsGenerateCert (8.71s)
        main_test.go:51: waitForProxy: did not start up within 5.0 seconds
FAIL
exit status 1
FAIL	github.com/google/martian/cmd/proxy	13.942s
error: Bad exit status from /var/tmp/rpm-tmp.AEwrYc (%check)

It works fine on any other arches.
You can have access to the full log here: https://koji.fedoraproject.org/koji/taskinfo?taskID=30397630

logging: enhanced debug logging across all core modifiers

All the core modifiers, filters, and verifiers should have increased debug log output (log.Debugf) so it's easier to trace which modifiers were called and which values were modified.

Individual Modifiers

  • auth.Filter
  • body.Modifier
  • cookie.Modifier
  • fifo.Group
  • header.ForwardedModifier
  • header.FramingModifier
  • header.BlacklistModifier
  • header.Filter
  • header.Modifier
  • header.Verifier
  • header.HopByHopModifier
  • header.ViaModifier
  • ipauth.Modifier
  • martianhttp.Modifier
  • martianurl.Modifier
  • martianurl.Filter
  • martianurl.Verifier
  • method.Verifier
  • pingback.Verifier
  • priority.Group
  • proxyauth.Modifier
  • querystring.Modfier
  • querystring.Filter
  • querystring.Verifier
  • status.Modifier
  • status.Verifier

proxy: provide support for WebSockets with MITM

Some prelimary investigation reveals several problems with supporting WebSockets, both secure and insecure. When using a proxy, browsers will send a CONNECT request to the destination when attempting to open a WebSocket, regardless of whether it is secure (ws:// or wss://).

Martian assumes CONNECT requests are requests to upgrade to TLS for HTTPS. In the non MITM case, Martian will blindly make a TCP connection by default for CONNECT requests which means that both secure and insecure WebSocket requests work.

Everything starts to break when MITM is enabled. Here's how:

To start, we'll look at an example request that is made to the proxy when attempting to open a WebSocket to echo.websocket.org.

CONNECT echo.websocket.org:80 HTTP/1.1
Host: echo.websocket.org:80
Proxy-Connection: keep-alive
Content-Length: 0
User-Agent: ...

Insecure WebSockets with MITM

Martian will wrap the connection with a TLS connection after sending a 200 OK response to the client. In the case of an insecure WebSocket request, the connection will then have a normal HTTP request sent to it for the WebSocket handshake. This fails with a TLS handshake error: tls: first record does not look like a TLS handshake.

How to Fix

This is the tricky case. We actually need to inspect the first couple bytes of traffic from the connection to be able to make a guess whether the connection is TLS or something else. The current idea is to inspect the bytes for:

  • A TLS version as part of the handshake and then wrap the connection in TLS.
  • The start of an HTTP request so we can also support that case and run any modifiers.
  • Anything unknown will be sent as a normal CONNECT tunnel and not modified.

Secure WebSockets with MITM

GET https://echo.websocket.org/?encoding=text HTTP/1.1
Host: echo.websocket.org
Upgrade: websocket
Connection: Upgrade
Content-Length: 0
Origin: http://www.websocket.org
Sec-Websocket-Extensions: permessage-deflate
Sec-Websocket-Key: 5/02dRzDuskoI7ZWhnvL1w==
Sec-Websocket-Version: 13

The TLS handshake succeeds in the secure WebSockets case, but fails with the following response to the prior request. This is because Martian will remove the Upgrade header (because it is listed in the Connection header) and thus prevent the downstream server from recognizing it as a WebSocket request.

HTTP/1.1 400 WebSocket Upgrade Failure
Content-Length: 77
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: content-type
Access-Control-Allow-Headers: authorization
Access-Control-Allow-Headers: x-websocket-extensions
Access-Control-Allow-Headers: x-websocket-version
Access-Control-Allow-Headers: x-websocket-protocol
Access-Control-Allow-Origin: http://www.websocket.org
Content-Type: text/html
Date: Tue, 24 Nov 2015 21:59:21 GMT

How to Fix

I expect this case will be slightly easier to handle with a custom modifier that checks for the Upgrade header, and if it finds websocket specified, will issue context.Hijack() and stitch the connections together.

context.Hijack()

The idea is that session.Context provides a Hijack() method similar to ResponseWriter.Hijack() that will return net.Conn, bufio.ReadWriter, error.

This will allow a modifier aware of WebSockets to take control of the connection while allowing modifiers before it to run.

A sub modification system (such as modifiers specifically designed to manipulate Websocket messages) could be created to further enhance the system.

To deal with the default case that is currently broken, we can create a modifier that will by default take any request with WebSocket headers and hijack the connection and stitch the two connections together reusing behavior similar to the CONNECT tunnel, send a 101 Switching Protocols response, and then waiting until the connection is closed.

cmd/proxy/main.go cannot configure modifier

  1. I using go build main.go to compile it to binary.
  2. Running it with ./main -v 10
  3. using http client to POST http://martian.proxy:8181 with data
 {
     "url.Filter": {
       "scope": ["response"],
       "host": "hiweeds.net",
       "modifier": {
         "header.Modifier": {
           "name": "Martian-Test",
           "value": "true"
         }
       }
     }
   }
  1. Server reply 404 page not found

I once using this program. and it can work fine. But now it fail, can anybody check that ?

HTTP/S reponse record and replay

Provide support for HTTP/S response record and replay

Users have asked for the capability to record responses from external services, and then to use those responses as a mechanism to stub out the external service during the execution of a test run.

Proposed Record Mode Workflow

  1. Put Martian into RECORD mode
  2. Instruct Martian to record responses to requests that match a provided keyGen() function.
  3. Provide Martian with the path to the db in which to save responses (bolt?).
  4. As requests that match the provided keyGen() come across, Martian captures the response that comes back from the HTTP roundtrip.
  5. For each response for request that matches the keyGen(), Martian adds a header with a known name (X-Martian-Key) where the value is a string representation of the output of keyGen().
  6. While in RECORD mode, any response modifiers will be applied before capture, maybe. Depends on where on the stack the record/replay modifier lives.
  7. Martian holds the responses in memory, and when Martian leaves RECORD mode, the responses are written to the filesystem to the provided path and filename.

Proposed Replay Mode Workflow

  1. Put Martian into REPLAY mode and tell it where the response cache db and the keyGen() used in RECORD mode
  2. Martian parses the response cache db and holds it in memory
  3. As requests that match the provided keyGen() come across, Martian grabs any matching response from the cache db, deletes the X-Martian-Key header, and returns the cached response.
  4. Martian applies any other modifiers on the stack - this is where you'd want to put modifiers that remove "dynamic" headers (datetime, etc.)
  5. Martian returns the modified cached response.

How to add filter for modifer response in martian ?

I saw the sample of add key-value pair in example, but not find how to modify the content of response in the README.

could any body show me how to do that?
For example, when the http response body have the data "hello", replace it with a word "world".

BTW, will martian decompress the http response data that compressed?

RFC 7231 4.3.6: A server MUST NOT send any Transfer-Encoding or Content-Length header fields in a 2xx (Successful) response to CONNECT

RFC 7231 4.3.6:

A server MUST NOT send any Transfer-Encoding or Content-Length header
fields in a 2xx (Successful) response to CONNECT. A client MUST
ignore any Content-Length or Transfer-Encoding header fields received
in a successful response to CONNECT.

Full bug report see curl/curl#1317

The current martian proxy handle CONNECT, will response

  • About to connect() to proxy 192.168.17.1 port 8123 (#0)
  • Trying 192.168.17.1...
  • Connected to 192.168.17.1 (192.168.17.1) port 8123 (#0)
  • Establish HTTP proxy tunnel to www.google.com:443

CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
User-Agent: curl/7.29.0
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 OK
< Content-Length: 0
<

The Content-Length must not response

flaky data racy in proxy during Close

I saw this on one of our Travis runs:

WARNING: DATA RACE
Read at 0x00c000097a08 by goroutine 30:
  internal/race.Read()
      /Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/internal/race/race.go:37 +0x38
  sync.(*WaitGroup).Add()
      /Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/sync/waitgroup.go:71 +0x16b
  github.com/google/martian.(*Proxy).handleLoop()
      /Users/travis/gopath/pkg/mod/github.com/google/[email protected]+incompatible/proxy.go:221 +0x7f
Previous write at 0x00c000097a08 by goroutine 77:
  internal/race.Write()
      /Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/internal/race/race.go:41 +0x38
  sync.(*WaitGroup).Wait()
      /Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/sync/waitgroup.go:128 +0xef
  github.com/google/martian.(*Proxy).Close()
      /Users/travis/gopath/pkg/mod/github.com/google/[email protected]+incompatible/proxy.go:145 +0xcc
  cloud.google.com/go/httpreplay/internal/proxy.(*Proxy).Close()
      /Users/travis/gopath/pkg/mod/cloud.google.com/[email protected]/httpreplay/internal/proxy/record.go:200 +0x54
  cloud.google.com/go/httpreplay.(*Replayer).Close()
      /Users/travis/gopath/pkg/mod/cloud.google.com/[email protected]/httpreplay/httpreplay.go:152 +0x4b
  gocloud.dev/internal/testing/setup.NewAWSSession2.func2()
      /Users/travis/gopath/src/gocloud.dev/internal/testing/setup/setup.go:139 +0x41
  gocloud.dev/pubsub/awssnssqs.(*harness).Close()
      /Users/travis/gopath/src/gocloud.dev/pubsub/awssnssqs/awssnssqs_test.go:195 +0x4c
  gocloud.dev/pubsub/drivertest.testMetadata()
      /Users/travis/gopath/src/gocloud.dev/pubsub/drivertest/drivertest.go:714 +0x901
  gocloud.dev/pubsub/drivertest.RunConformanceTests.func1()
      /Users/travis/gopath/src/gocloud.dev/pubsub/drivertest/drivertest.go:167 +0x6f
  testing.tRunner()
      /Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/testing/testing.go:827 +0x162
Goroutine 30 (running) created at:
  github.com/google/martian.(*Proxy).Serve()
      /Users/travis/gopath/pkg/mod/github.com/google/[email protected]+incompatible/proxy.go:216 +0x392
Goroutine 77 (finished) created at:
  testing.(*T).Run()
      /Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/testing/testing.go:878 +0x659
  gocloud.dev/pubsub/drivertest.RunConformanceTests()
      /Users/travis/gopath/src/gocloud.dev/pubsub/drivertest/drivertest.go:167 +0x5c3
  gocloud.dev/pubsub/awssnssqs.TestConformance()
      /Users/travis/gopath/src/gocloud.dev/pubsub/awssnssqs/awssnssqs_test.go:212 +0xa5
  testing.tRunner()
      /Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/testing/testing.go:827 +0x162
==================

logging: add tracing to log calls to aid in debugging

In the martian.Context for each http.Request is a short lived unique(ish) ID. It would be helpful for debugging to be able to include this "trace ID" in every log line that occurs to be able to follow the execution/modification of a request/response.

My initial idea is that a package creates a Logger instance that it holds and passes the martian.Context to that will obtain the information it needs and reformat as appropriate.

package mypkg

var log = martian.NewLogger("mypkg.Modifier")

type Modifier struct {
  ...
}

func (m *Modifier) ModifyRequest(req  *http.Request) error {
  ctx := martian.NewContext(req)

  // Logs with ctx.ID(), the package and the formatted log line. 
  log.Debugf(ctx, "modifying request: %s", req.URL)

  ...

  return nil
}

filter: consolidate filter modifier behavior and create filter conditions

Filters often end up in a large amount of duplication and boilerplate code to work with all of the APIs in Martian (Verification, JSON, whatever else). They also have the problem that it's not possible to "invert" a filters condition or provide an "else" when the condition fails.

I think it would be more maintainable going forward to create a standard filter that takes care of acting as a modifier and can accept conditions, essentially creating a rather basic strategy pattern.

The API I have in my head looks something like:

package filter

type RequestCondition interface {
  MatchRequest(req *http.Request) (bool, error)
}

type ResponseCondition interface {
  MatchResponse(res *http.Response) (bool, error)
}

type Filter struct {
  reqcond filter.RequestCondition
  rescond filter.ResponseCondition
  reqmod martian.RequestModifier
  resmod martian.ResponseModifier
}

func (f *Filter) ModifyRequest(req *http.Request) error {
  match, err := f.reqcond.MatchRequest(req)
  if err != nil {
    return err
  }
  if !match {
    return nil
  }

  return f.reqmod.ModifyRequest(req)
}

This would allow us to trivially add a filter.NotCondition wrapper around any filter.RequestCondition or filter.ResponseCondition that could be used to invert the condition and create an "else".

400 - Bad request and 404- Not founc while extracting HAR logs

Hi team,
I am trying to use martian proxy on windows 10 host to intercept traffic from my Android mobile.

I started the proxy using the command proxy -v=2 - har=true.

Proxy is started with default settings [traffic on 8080 and api on 8181]
I am getting following errorss when I try to get the har logs

GET http://localhost:8181/logs- 400 Bad Request

GET http://localhost:8080/logs- 404 Not Found

I am using Advanced Rest Client to call Martian rest APIs.

Could you please help?

verify: refactor Go/JSON API

There is an unnecessary amount of duplication between verifications and filters. Additionally, the verifiers have inconsistent behavior as to whether the verification requirement is 0, 1, or n times. It would be nice to create an API that handles the verification requirement in one place while allowing filters to be used or verification requirements.

Example API

For example, using the basic verification framework together with a url.Filter to create a verification for "all requests must be HTTPS":

v := verify.NewVerification("HTTPS enforcement")
c := v.VerifyAll()
// c := v.VerifyAny()
// c := v.VerifyCount(10)
// c := v.VerifyNone()

f := martianurl.NewFilter(&url.URL{Scheme: "https"})
f.SetRequestModifier(c)

v.SetRequestModifier(f)

Or in the JSON API:

{
  "verify.Verification": {
    "name": "HTTPS enforcement",
    "modifier": {
      "url.Filter": {
        "scheme": "https",
        "modifier": {
          "verify.All": { }
        }
      }
    }
  }
}

Implementation Details

Conditions need to be tied to their parent verify.Verification.

Go API

In the Go API this is simple enough since we can just have the Verification keep track of it's child conditions using the constructors available on Verification (VerifyAll, VerifyNone, etc.).

This will allow us to ask the verification container to generate errors without requiring complete traversal of the modifier tree. Additionally, this will allow us to remove all the existing methods on filters and groups that currently deal with passing along verification information through the stack (VerifyRequests, VerifyResponses, ResetRequestVerifications, and ResetResponseVerifications).

JSON API

The parse API is trickier. The JSON API currently does not provide any context during the parse and thus has no way to tie a modifier with the key verify.All to the outer verify.Verification.

The only solution I currently have to this is to provide a *parse.Context that can carry this information through the parse phase. It should itself be similar to net/context's Context such that we can push and pop the current verify.Verification off as needed since verify.Verifications may be nested underneath of each other, though I currently don't have a good explanation for when that would even make sense. Maybe as a general context API for others to use in case they need to store information that is nestable.

Go Module Compatibility

While the tag created in #261 is very helpful for module enabled users, the version shows up a v2.1.0+incompatible due to the lack of semantic import versioning. Can we get this repo into compatibility?

Based on the suggestions made in the wiki on Modules, it sounds like one option is to bump to v3.0.0. I am aware that this may not be a popular decision, since there would not be any API breakage, but it could be an opportunity to do so, if that was needed.

The wiki also suggests solutions for source management for packages like yours.

These appear to be the options available:

  • bump major version: v3.0.0
    • major branch
    • major subdirectory
  • bump minor version: v2.2.0
    • you will likely encounter backwards incompatible changes

New Modifier Type: Notifier

A notifier is a modifier that executes a WebHook (HTTP callback) when the modifier is evaluated in the modifier tree.

A notifier would take a target URL and a message string. When MoidfyRequest or ModifyResponse is executed, a POST request is sent to the the target URL with the message string as the request body.

Might take separate target URLs and messages for request or response to provide control over whether to execute the callback on request or response.

Proxying via curl fails when specifying --key

I am trying to setup http(s) proxy server using martian.

I generated key/certificate files and tried to run a proxy in cmd using the following command.

./proxy -addr=:8080 -tls-addr=:8081 -api-addr=:9090 -api=localhost -cert=ca.crt -key=ca.key -har -v=2

I will show a history I tried. Please tell me how to configure or what was wrong.

Connecting to http://google.com:80 using curl.

It worked fine.

curl command

$ http_proxy=localhost:8080 https_proxy=localhost:8081 curl -p http://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

martin log

2019/04/08 15:55:53 INFO:
--------------------------------------------------------------------------------
Request to http://google.com:80
--------------------------------------------------------------------------------
CONNECT http://google.com:80 HTTP/1.1
Host: google.com:80
Content-Length: 0
User-Agent: curl/7.54.0
Via: 1.1 martian-006e5d3bdb04c9a1b2f8
X-Forwarded-For: ::1
X-Forwarded-Host: google.com:80
X-Forwarded-Proto: http
X-Forwarded-Url: http://google.com:80

Connecting to https://google.com:443 using curl

It didn't work fine.

curl command

$ http_proxy=localhost:8080 https_proxy=localhost:8081 curl -p https://google.com
curl: (56) Proxy CONNECT aborted

martin log

2019/04/08 15:58:30 ERROR: martian: failed to read request: tls: oversized record received with length 20037

Connecting to https://google.com:443 using Chrome

It didn't work fine.

Chrome result

ERR_PROXY_CONNECTION_FAILED

martin log

2019/04/08 15:59:10 ERROR: martian: failed to read request: mitm: SNI not provided, failed to build certificate

configure modifier failed.

I am newbie to golang, and found https://github.com/netroby/gohttpproxy.git this httpproxy using martian, so I run it and with description in martian readme.md page, try to set the modifier, but it is not work out, main reason is as below, there is no martian.proxy in host,should I add it in hostname ?or what should I do the make the set of modifiers work.

martian logs as below.

--------------------------------------------------------------------------------
Request to http://martian.proxy/configure
--------------------------------------------------------------------------------
POST http://martian.proxy/configure HTTP/1.1
Host: martian.proxy
Content-Length: 98
Accept: */*
Content-Type: application/json
User-Agent: curl/7.49.1
Via: 1.1 martian
X-Forwarded-For: ::1
X-Forwarded-Host: martian.proxy
X-Forwarded-Proto: http

{  "header.Modifier": {    "scope": ["response"],    "name": "Test-Header",    "value": "true"  }}
--------------------------------------------------------------------------------

2016/10/21 09:03:57 ERROR: martian: failed to round trip: dial tcp: lookup martian.proxy: no such host
2016/10/21 09:03:57 INFO: 
--------------------------------------------------------------------------------
Response from http://martian.proxy/configure
--------------------------------------------------------------------------------
HTTP/1.1 502 Bad Gateway
Content-Length: 0
Warning: 199 "martian" "dial tcp: lookup martian.proxy: no such host" "Fri, 21 Oct 2016 09:03:57 GMT"

messageview attempts to decode partial data

messageview.MessageView.BodyReader tries to uncompress the body by looking only at the Content-Encoding. But it neglects to check for a Range header (or for a 206 Partial Content status). This results in an error when attempting to unzip a partial gzip file.

HAR PostData text is wrong type

har.PostData.Text, the body of a POST, has type string. This is a mistake. It should be []byte.

The HAR spec does say that text is a string, but it means a JSON string, not a Go string. POST data can contain arbitrary bytes (e.g. if the MIME type is application/octet-stream).

The reason this matters is that encoding/json.Marshal expects strings to be valid UTF-8; if they are not, it loses information by replacing invalid code points with the replacement character. So when a har.Log is marshaled to JSON, the result is wrong.

Example:

    pd := &har.PostData{Text: string([]byte{150, 151, 152})}
    bytes, err := json.Marshal(pd)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%v\n", string(bytes))

Output:

{"mimeType":"","params":null,"text":"\ufffd\ufffd\ufffd"}

cmd/proxy: integration tests

Now that github.com/google/martian/cmd/proxy is going to be our de facto proxy implementation we should write some integration tests for it to ensure we don't break it.

Enhanced Traffic Shaping

Objective

To enhance Martian’s trafficshape package so that it can simulate desired network conditions more precisely. These features are needed in order to allow developers to write tests that exercise application features that are designed to react to suboptimal network conditions.
Key features of this effort include:

  • Apply traffic shaping behaviors on a per URL basis (based on a regex)
  • Specify throughput (mbps) for byte ranges
  • Specify hangs (with duration) or exits at byte offset

Background

These features are useful to anyone who is interested in writing tests that simulate poor network conditions: especially any tests that are serving video. More specifically, the test authors will be able to specify the “shape” of the video traffic as a configuration in the test and write assertions based on the intended behavior of client in dealing with the traffic.

Currently, the trafficshape package only allows setting the bandwidth and latency for all the requests and responses; the user cannot pick a few resources for which to shape traffic. This is insufficient for the testing situation where one wants to throttle a particular resource while letting the rest of the test infrastructure use the network normally.

The trafficshape package also does not have a way to keep track of what byte offsets are currently being written back to the client for a specific resource. This can be useful when the user wants to perform different actions or have different bandwidths depending on what byte ranges are being written back to the client.

Requirements

  • Traffic shaping behavior is configured from within a test by issuing a POST request with a JSON configuration to a new configuration endpoint /configure/trafficshape
  • Traffic shaping behaviors are defined on a per resource basis
    • Match on URL or parts of URL (using martianurl.Matcher)
    • Match based on URL regex (needs a new matcher)
  • Add the ability to specify
    • Downstream bandwidth for a given byte range
    • Hang for a given time at a specific byte
    • Close connection at a specific byte
    • Number of times to repeat this behavior

Design

JSON Configuration API

{
    "trafficshape": {
        "default": {
            "bandwidth": {
                "up": 10000000,  
                "down": 1000000
            },
            "latency": {
                "up": 1000, 
                "down": 100000 
            }
        },
        "shapes": [{
            "url_regex": "example.com/video/stream",
            "throttles": [{
                "throttle": {
                    "bytes": "100-150000",
                    "bandwidth": 100000
                }
            }],
            "halts": [{
                "halt": {
                    "byte": 200000,
                    "duration": 5000,
                    "count": 1
                }
            }],
            "close_connections":[{
                "close_connection": {
                    "byte": 250000,
                    "count": 1
                }
            }]
        }]
    }
}

Default Latency and Bandwidth

These are the values that will be applied for the byte ranges that do not have throttles specified for them.

Throttle

Every throttle action takes in a byte range and a bandwidth. When the proxy is writing back bytes that fall in the range of a throttle’s byte range for the current url, it will be subject to the corresponding bandwidth.

Halt

Every halt action takes in a byte, duration and count (Explained below). The byte specifies at which byte to halt writing data back to the client and duration specifies how long to remain in this state before starting to write data back to the client.

Close_connection

Every close_connection action takes in a byte and a count (Explained below). The byte specifies what byte the proxy should be about to write back to the client before it should close the current connection with the client.

Count

Note the “count” property in the “halt” and “close_connection” actions . It is possible that in the course of writing the video, the client asks for byte range multiple times (this might happen when the existing connection is interrupted, and the client didn’t read part of the data written back by the proxy already, so it asks again for a range of the data that we have already written). So it is possible that they would want the action to be performed only the first time a specific byte is written back, or possibly more times. That is what the count specifies. A count of -1 (default) will specify that the action will happen everytime the specific byte is written back.

So the example json says:

  • Make the default bandwidth 100,000mbs, and do not add any additional latency.
  • And for any response containing a substring matching the url regex example.com/video/stream:
    • Write the bytes 100 to 150,000 (150kb) (with 150kb being exclusive) to the client at the bandwidth of 50,000 (50kb) per second. For all the other bytes outside this range, write back at the default bandwidth.
    • Upon reaching the byte 200,000 (200kb), perform an interrupt for 5000ms. Do this only the first time you are writing the byte 200kb back to the client. Do not do anything if you are writing back byte 200kb again.
    • Upon writing the byte 250,000 (250kb), close the current connection with the client. Do this 1 time only the first time when writing the byte 250kb back. (After we do this, the client may start a new connection and request the rest of the video, and we don’t want to perform the “close” again. Note, we almost always overestimate the data we write back to the client, so if we perform the close at byte 250kb, the client would probably ask for bytes, say 200kb onwards in a following request, and we do not want to close the connection again when we reach byte 250kb.)

For the byte_ranges, we support ranges to be expresses as “-5000”, which means “up till but not including byte 5000”, or “5000-”, which would mean from “5000 onwards”.

Feature request: Add "positive" at least once verifier

Background:

It would be great to verify that a request has gone through the proxy. This can be very handy when testing to verify that a specific callback has been called for example. In this case there would be an error if the verifier has still not matched any request/response when GET http://martian.proxy/verify is called.

Example interface:

Verifier:

{
  "atLeastOnce.Verifier": {
    "scope": ["request"],
    "host": "example.com",
    "path": "/testing"
  }
}

Error:

{
  "errors" : [
    {
      "message": "request(http://example.com/testing) call verify failure"
    }
  ]
}

Error when installing the package using "go get"

I tried installing the martian package using the following command

go get github.com/google/martian

This was the error I received.

$GOPATH......./src/github.com/google/martian/proxy.go:142: tr.TLSNextProto undefined (type *http.Transport has no field or method TLSNextProto)

HAR log entries missing timing data

The entries in the generated har files have all 3 members of the timings struct set to 0. After the struct is initialized, it looks like it is never filled out, so the 3 fields always stay at 0.

TestStreamsInSentOrder sometimes deadlocks

We are having a problem in Debian when building the martian where TestStreamsInSentOrder occasionally deadlocks. There's a little more info at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923819 but tl;dr in a small but consistent number of builds TestStreamsInSentOrder runs for 10 minutes until it is killed. It looks this test was previously flaky and fixed in #265. Is it possible there are still issues? I'm also doing some investigation on my end to see what I can find. A log of the failure is below.

=== RUN   TestStreamsInSentOrder
SIGQUIT: quit
PC=0x45e770 m=0 sigcode=0

goroutine 0 [idle]:
runtime.epollwait(0x7fff00000004, 0x7fff30199818, 0xffffffff00000080, 0x7fff00000001, 0xffffffff00000001, 0x1, 0x0, 0xda5f5e3000000005, 0x7f17, 0x0, ...)
	/usr/lib/go-1.11/src/runtime/sys_linux_amd64.s:671 +0x20
runtime.netpoll(0xc000028501, 0xc000028501)
	/usr/lib/go-1.11/src/runtime/netpoll_epoll.go:71 +0x132
runtime.findrunnable(0xc000026000, 0x0)
	/usr/lib/go-1.11/src/runtime/proc.go:2469 +0x51c
runtime.schedule()
	/usr/lib/go-1.11/src/runtime/proc.go:2613 +0x13a
runtime.park_m(0xc000001800)
	/usr/lib/go-1.11/src/runtime/proc.go:2676 +0xae
runtime.mcall(0x0)
	/usr/lib/go-1.11/src/runtime/asm_amd64.s:299 +0x5b

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc00010c100, 0x72cf41, 0x16, 0x73d830, 0x478386)
	/usr/lib/go-1.11/src/testing/testing.go:879 +0x383
testing.runTests.func1(0xc00010c000)
	/usr/lib/go-1.11/src/testing/testing.go:1119 +0x78
testing.tRunner(0xc00010c000, 0xc0000a5e08)
	/usr/lib/go-1.11/src/testing/testing.go:827 +0xbf
testing.runTests(0xc00008e320, 0x967200, 0x8, 0x8, 0x40d3ff)
	/usr/lib/go-1.11/src/testing/testing.go:1117 +0x2aa
testing.(*M).Run(0xc00010a000, 0x0)
	/usr/lib/go-1.11/src/testing/testing.go:1034 +0x165
main.main()
	_testmain.go:56 +0x13d

goroutine 19 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f17da5f5e30, 0x72, 0xc000046b80)
	/usr/lib/go-1.11/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00010a318, 0x72, 0xffffffffffffff00, 0x776bc0, 0x934560)
	/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00010a318, 0xc00013c000, 0x1000, 0x1000)
	/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00010a300, 0xc00013c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go-1.11/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00010a300, 0xc00013c000, 0x1000, 0x1000, 0xc0004d9630, 0x40, 0x50)
	/usr/lib/go-1.11/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00008a058, 0xc00013c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go-1.11/src/net/net.go:177 +0x68
bufio.(*Reader).fill(0xc00006a4e0)
	/usr/lib/go-1.11/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc00006a4e0, 0xc0002cd630, 0x0, 0xf)
	/usr/lib/go-1.11/src/bufio/bufio.go:242 +0x39
golang.org/x/net/websocket.hybiFrameReaderFactory.NewFrameReader(0xc00006a4e0, 0x73e108, 0xc0001ae028, 0xc000464000, 0xc000046e90)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/hybi.go:123 +0x60
golang.org/x/net/websocket.Codec.Receive(0x73d868, 0x73d870, 0xc0001ae000, 0x6a3c40, 0xc0004ed520, 0x0, 0x0)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/websocket.go:341 +0xaf
github.com/google/martian/marbl.TestStreamsInSentOrder(0xc00010c100)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler_test.go:55 +0x32a
testing.tRunner(0xc00010c100, 0x73d830)
	/usr/lib/go-1.11/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
	/usr/lib/go-1.11/src/testing/testing.go:878 +0x35c

goroutine 21 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f17da5f5f00, 0x72, 0x0)
	/usr/lib/go-1.11/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00010a118, 0x72, 0xc00005e000, 0x0, 0x0)
	/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00010a118, 0xffffffffffffff00, 0x0, 0x0)
	/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc00010a100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/go-1.11/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc00010a100, 0xc00013a140, 0x3f26326a14b63c0c, 0x44c488)
	/usr/lib/go-1.11/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00008a050, 0x5c7e7212, 0xc00004be78, 0x478386)
	/usr/lib/go-1.11/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc00008a050, 0xc00004bec8, 0x18, 0xc000070600, 0x645b85)
	/usr/lib/go-1.11/src/net/tcpsock.go:260 +0x47
net/http.(*Server).Serve(0xc000136000, 0x778ca0, 0xc00008a050, 0x0, 0x0)
	/usr/lib/go-1.11/src/net/http/server.go:2826 +0x22f
net/http.Serve(0x778ca0, 0xc00008a050, 0x775d00, 0xc00008e460, 0x0, 0x0)
	/usr/lib/go-1.11/src/net/http/server.go:2423 +0x6e
created by github.com/google/martian/marbl.TestStreamsInSentOrder
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler_test.go:38 +0x163

goroutine 4 [chan receive, 9 minutes]:
github.com/google/martian/marbl.(*Handler).streamLogs(0xc00008e460, 0xc00010e240)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler.go:77 +0x19d
github.com/google/martian/marbl.(*Handler).streamLogs-fm(0xc00010e240)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler.go:61 +0x34
golang.org/x/net/websocket.Server.serveWebSocket(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/server.go:89 +0x17e
golang.org/x/net/websocket.Server.ServeHTTP(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/server.go:70 +0x6a
github.com/google/martian/marbl.(*Handler).ServeHTTP(0xc00008e460, 0x778ee0, 0xc000146000, 0xc00010c800)
	/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler.go:61 +0xd1
net/http.serverHandler.ServeHTTP(0xc000136000, 0x778ee0, 0xc000146000, 0xc00010c800)
	/usr/lib/go-1.11/src/net/http/server.go:2741 +0xab
net/http.(*conn).serve(0xc000140000, 0x779120, 0xc00008ca40)
	/usr/lib/go-1.11/src/net/http/server.go:1847 +0x646
created by net/http.(*Server).Serve
	/usr/lib/go-1.11/src/net/http/server.go:2851 +0x2f5

rax    0xfffffffffffffffc
rbx    0xffffffff
rcx    0x45e770
rdx    0x80
rdi    0x4
rsi    0x7fff30199818
rbp    0x7fff30199e18
rsp    0x7fff301997d8
r8     0x0
r9     0x3
r10    0xffffffff
r11    0x246
r12    0xffffffffffffffff
r13    0x6a
r14    0x69
r15    0x200
rip    0x45e770
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
*** Test killed with quit: ran too long (10m0s).
FAIL	github.com/google/martian/marbl	600.017s

Allow rendering HAR logs as JSONP via query param

The popular online HAR viewer tool at http://www.softwareishard.com/har/viewer has the ability of loading (via an ajax request so even local URLs work) any HAR from its URL if this HAR is formatted as

onInputData(harJson);

If Martian allowed for this format change via, let's say, a query param (?jsonp=1) we could go to http://www.softwareishard.com/har/viewer?inputUrl=http://martian.proxy/logs?jsonp=1 to get instant rendering of the HAR timeline.

test failure in static/

Hi,

command: go test -buildmode pie -compiler gc -ldflags "-X github.com/google/martian/version=2.1.0 -extldflags '-Wl,-z,relro  '"
testing: github.com/google/martian/static
--- FAIL: TestFileExistsInBothExplictlyMappedPathAndInferredPath (0.00s)
    static_file_modifier_test.go:115: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestStaticModifierExplicitPathMapping (0.01s)
    static_file_modifier_test.go:171: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestStaticModifierOnRequest (0.00s)
    static_file_modifier_test.go:222: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestRequestOverHTTPS (0.00s)
    static_file_modifier_test.go:275: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestModifierFromJSON (0.00s)
    static_file_modifier_test.go:359: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
    static_file_modifier_test.go:398: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestStaticModifierSingleRangeRequest (0.00s)
    static_file_modifier_test.go:473: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
FAIL
exit status 1
FAIL    github.com/google/martian/static        0.020s

forwarded_modifier should preserve the original client values for x-forwarded-{proto, host, url}

When martian gets a request from another proxy, it may already have x-forwarded-... headers that represent the original client request. The current implementation (link below) overwrites those values. I would expect the values to be preserved if already present, as I expect these headers to tell me about the original client request, not the request as received from the second-to-last proxy in the chain.

func NewForwardedModifier() martian.RequestModifier {

verify: enhance errors JSON API

The package verify needs to be reworked to be more useful for end users.

The current JSON API looks like:

{
  "errors": [
    { "message": "request(url): message: got %v, want %v" }
  ]
}

Having the HTTP message type and verification type mixed in with the verification failure makes it hard to distinguish what type of failure happened and forces the consumer to blindly accept the message string as is.

The message key should continue to exist for those who want to blindly accept the full message formatting from the verifier, but should be able to use the metadata about the failure to construct their own message.

The proposed new format:

{
  "errors": [
    {
      "header.Verifier": {
        "scope": "request",
        "url": "http://example.com/path",
        "message": "[ existing message format ]",
        "actual": "Content-Type: text/plain",
        "expected": "Content-Type: image/png"
      }
    }
  ]
}

This format closely resembles the existing modifier JSON API with standard keys for the verification error.

Go Module Compatibility

While the tag created in #261 is very helpful for module enabled users, the version shows up a v2.1.0+incompatible due to the lack of semantic import versioning. Can we get this repo into compatibility?

Based on the suggestions made in the wiki on Modules, it sounds like one option is to bump to v3.0.0. I am aware that this may not be a popular decision, since there would not be any API breakage, but it could be an opportunity to do so, if that was needed.

The wiki also suggests solutions for source management for packages like yours.

These appear to be the options available:

  • bump major version: v3.0.0
    • major branch
    • major subdirectory
  • bump minor version: v2.2.0
    • you will likely encounter backwards incompatible changes

got 407 when fetching https with downstream proxy

My remote proxy which is password protected and supports https, when applies to martian as downstream proxy, 407 is got.

martian -addr=:8000 -api martian -organization Martian -downstream-proxy-url http://xxxx:password@someproxy:60000
curl -x http://localhost:8000 https://ip.cn  -v

then comes the following result:

  • Rebuilt URL to: https://ip.cn/
  • Trying ::1...
  • TCP_NODELAY set
  • Connected to localhost (::1) port 8000 (#0)
  • allocate connect buffer!
  • Establish HTTP proxy tunnel to ip.cn:443

CONNECT ip.cn:443 HTTP/1.1
Host: ip.cn:443
User-Agent: curl/7.59.0
Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required
< Connection: close
< Content-Type: text/html
< Server: someproxy
<

  • Received HTTP code 407 from proxy after CONNECT
  • CONNECT phase completed!
  • Closing connection 0
    curl: (56) Received HTTP code 407 from proxy after CONNECT

New Modifier: har.Logger

Would allow customising the har logger via the REST API and leverage the existing bodyLogging and postDataLogging options currently only accessible when martian is used as a library.

{
  "har.Logger": {
    "scope": ["request", "response"],
    "bodyLogging": false,
    "postDataLogging": false
  }
}

Use case: avoid transmitting heavy request/response bodies when they are not necessary for what's being tested. For e.g. when testing the HAR logs in a separate process that communicates with martian over the REST api.

har: header representation loses information

The har package concatenates all values with the same header together, separated by commas. This makes it impossible to round-trip. E.g. the headers

A: 1
A: 2

are represented the same as

A: 1, 2

Namely, as Header{Name: "A", Value: "1, 2"}.

The fix is for cases like the first to be represented by two distinct Header values.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.