uber / eight-track Goto Github PK

To make debugging easier, we should expose the request information inside of the generated files (e.g. method, url, headers).

The normalizeFn should only be used to generate a file key for the fixture. However, it seems https://github.com/uber/eight-track/blob/master/lib/eight-track.js#L41 actually modifies the info that is passed into the request.

Upon revisiting #17, I realized that we are allowing request to follow redirects. A quick regression test in 1f042c4 proved that my fears were true. Please add a followRedirect: false to all requests (the client requesting to our server will follow the 302's anyway).

In the land of JSON, the order of object keys should not matter.

request has been great for getting eight-track released. However, with issues like #4 and #18, it feels like it is doing too much for us. We should be using something much weaker; ideally some variant of http-proxy but I am not sure how composable it is.

https://github.com/nodejitsu/node-http-proxy

Currently, it is difficult modify response content from eight-track. I attempted to write up way to perform it via piping the response but even that does not write original data to disk making it useless (it only reads from disk).

https://gist.github.com/twolfson/9753482

The ideal API would be request-like that emits data events or calls back with the body/buffer.

// Inside of a fixed-server
response: function (req, res, next) {
  // Forward the original request
  eightTrack.forwardRequest(req, function (err, externalRes, externalBody) {
    // If there was an error, callback with it
    if (err) {
      return next(err);
    }

    // Send back twice the amount of items
    var items = JSON.parse(externalBody);
    res.send(items.concat(items));
  });
}

This is a hybrid between mocking and realistic data which is more realistic than only mocks.

/cc @zheller

During development, it is useful to have eight-track functionality to avoid abusing external services. We should consider creating eight-track-cli which would create an http server using eight-track.

I would prefer this be an external repo since it could get more complex as time goes on (e.g. support config files) and I would like to isolate said complexity.

eight-track --port 9001 --url "http://localhost:1337/" --fixture-dir "recordings"
eight-track --config ./eighttrackrc

/cc @zheller

Hey, trying to use this with some CORS requests and getting this error in Chrome when running through the middle ware:

I noticed that the body response for OPTIONS looks weird:

  "response": {
    "httpVersion": "1.1",
    "headers": {
      "content-type": "application/json; charset=utf-8",
      "strict-transport-security": "max-age=10886400",
      "timestamp": "1397159432",
      "access-control-allow-origin": "*",
      "access-control-max-age": "86400",
      "access-control-allow-methods": "GET, HEAD, POST, PUT, DELETE, OPTIONS",
      "access-control-allow-headers": "Authorization, Content-Type, If-None-Match",
      "access-control-expose-headers": "WWW-Authenticate, Server-Authorization, Timestamp, Accept-Language",
      "cache-control": "no-cache",
      "content-encoding": "gzip",
      "vary": "accept-encoding",
      "date": "Thu, 10 Apr 2014 19:50:32 GMT",
      "connection": "keep-alive",
      "transfer-encoding": "chunked"
    },
    "trailers": {},
    "statusCode": 200,
    "body": "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0003��\u0005\u0000C���\u0002\u0000\u0000\u0000"
  }
}

While #14 is a step in the right direction, the benchmark of 800ms for 100 repeated requests leaves me wanting more. We should add some timing logs to each function run in the test suite.

Off the top of my head, some more possible gains might be:

• Key generation (try to use a better serializer)
  • https://github.com/isaacs/sigmund
• Caching of generated keys (avoid md5 if it is too slow)

While attempting to install eight-track to twolfson/icomoon-phantomjs, I encountered an issue with us forwarding a request for gzipped content tthat was stringifying binary data in the response. When moving off of toString() for outgoingBuff, this fixed it nicely.

My suggestion is to:

• Store buffer data in external file (e.g. .raw, .buff, or .in/.out for incoming/outgoing buffers respectively)
• Possibly store truncated stringified version in JSON for faster viewing/debugging. Definitely add a reference to relative file path of .in/.out so we can extend on this as an indirect reference (instead of having weird brittle .ext replacement logic)

This might be a good opportunity to patch #1.

When we make a very long request, we save a correspondingly long filename. Unfortunately, sometimes it is too long to save to disk (e.g. ENAMETOOLONG).

Recently, we added some large data sets in a private repository. We are seeing up to 200ms load time for both disk load and JSON.parse. We should consider caching disk loads and JSON.parse results (with deepClone's to prevent original data mutation).

Sometimes during testing, we want to assert we are passing specific data since achieving reproducable state can be difficult. We should make it easy for people to set a pre-buffered body that we can forward requests against.

During the transition of twolfson/css-validator from nock to eight-track, I ran into a snag. The W3C's CSS validtion service uses multipart/form-data to accept POST data. An example request looks like:

    "headers": {
      "content-type": "multipart/form-data; boundary=--------------------------254222337025012242873031"
    },
    "method": "POST",
    "url": "/css-validator/validator",
    "body": "----------------------------254222337025012242873031\r\nContent-Disposition: form-data; name=\"output\"\r\n\r\nsoap12\r\n----------------------------254222337025012242873031\r\nContent-Disposition: form-data; name=\"w3cUrl\"\r\n\r\nhttp://localhost:1337/css-validator/validator\r\n----------------------------254222337025012242873031\r\nContent-Disposition: form-data; name=\"highWaterMark\"\r\n\r\n1024000\r\n----------------------------254222337025012242873031\r\nContent-Disposition: form-data; name=\"text\"\r\n\r\nbody {\n  background: url(abcd);\n}\n\r\n----------------------------254222337025012242873031--"

Part of the specification for multipart/form-data is to use a boundary to delimit different chunks of data. This delimiter must not be contained in the original data.

http://tools.ietf.org/html/rfc2388#section-4.1

To solve this problem, some libraries (e.g. felixge/node-form-data) generate a random number as their boundary (as seen in the example above).

https://github.com/felixge/node-form-data/blob/0.1.2/lib/form_data.js#L193-L202

To combat this, we some options include:

• Introduce a normalize parameter which is a function that adjusts the data we serialize against to generate a key
• Add built-in normalizing logic for multipart form (or at least an option for it)

I would lean towards the normalize function as it covers cases that we have not yet encountered. Additionally, maybe we can add a class-level function which is already set up to normalize multipart form boundaries.

If a request comes in with a Host header, we should use the new host's header to prevent issues.

Currently, this is occurring with node-zendesk + eight-track and causing the request to be rejected.

Please restore the original header on response to prevent any possible future issues.

https://github.com/mikeal/request/blob/v2.30.1/request.js#L193-L202

https://github.com/mikeal/request/blob/v2.30.1/request.js#L648

To be as accurate as possible, we should consider moving to buffer clones that will each be written via their own unique data events.

Currently, we are concatenating them all and pushing them via one major data event.

I want to create fixtures on my local machine, and do not create them on CI server (use saved fixtures from repo). So, basically I want to disable recording during CI run. Is it possible now?

Currently, you can only proxy to a hostname + port. In theory, you should be able to proxy to a subpath

eightTrack({
  url: 'http://localhost:1337/abc'
}).listen(1338);

request('http://localhost:1338/def'); // proxies to 'http://localhost:1337/abc/def')