Git Product home page Git Product logo

node-http-mitm-proxy's Introduction

HTTP MITM Proxy

HTTP Man In The Middle (MITM) Proxy written in node.js. Supports capturing and modifying the request and response data.

NPM version Downloads Test Status

Install

npm install --save http-mitm-proxy

Node.js Compatibility

The library should work starting Node.js 8.x, but testing is only expected for currently supported LTS versions of Node.js starting Node.js 12.x . use on your own risk with non LTS Node.js versions.

Typescript

type definitions are now included in this project, no extra steps required.

Example

This example will modify any search results coming from google and replace all the result titles with "Pwned!".

const Proxy = require('http-mitm-proxy').Proxy;
// or using import/module (package.json -> "type": "module")
// import { Proxy } from "http-mitm-proxy";
const proxy = new Proxy();

proxy.onError(function(ctx, err) {
  console.error('proxy error:', err);
});

proxy.onRequest(function(ctx, callback) {
  if (ctx.clientToProxyRequest.headers.host == 'www.google.com'
    && ctx.clientToProxyRequest.url.indexOf('/search') == 0) {
    ctx.use(Proxy.gunzip);

    ctx.onResponseData(function(ctx, chunk, callback) {
      chunk = Buffer.from(chunk.toString().replace(/<h3.*?<\/h3>/g, '<h3>Pwned!</h3>'));
      return callback(null, chunk);
    });
  }
  return callback();
});

console.log('begin listening on 8081')
proxy.listen({port: 8081});

You can find more examples in the examples directory

SSL

Using node-forge allows the automatic generation of SSL certificates within the proxy. After running your app you will find options.sslCaDir + '/certs/ca.pem' which can be imported to your browser, phone, etc.

API

Proxy

Context

Context functions only effect the current request/response. For example you may only want to gunzip requests made to a particular host.

WebSocket Context

The context available in websocket handlers is a bit different

Starts the proxy listening on the given port.

Arguments

  • options - An object with the following options:
  • port - The port or named socket to listen on (default: 8080).
  • host - The hostname or local address to listen on (default: 'localhost'). Pass '::' to listen on all IPv4/IPv6 interfaces.
  • sslCaDir - Path to the certificates cache directory (default: process.cwd() + '/.http-mitm-proxy')
  • keepAlive - enable HTTP persistent connection
  • timeout - The number of milliseconds of inactivity before a socket is presumed to have timed out. Defaults to no timeout.
  • httpAgent - The http.Agent to use when making http requests. Useful for chaining proxys. (default: internal Agent)
  • httpsAgent - The https.Agent to use when making https requests. Useful for chaining proxys. (default: internal Agent)
  • forceSNI - force use of SNI by the client. Allow node-http-mitm-proxy to handle all HTTPS requests with a single internal server.
  • httpsPort - The port or named socket for https server to listen on. (forceSNI must be enabled)
  • forceChunkedRequest - Setting this option will remove the content-length from the proxy to server request, forcing chunked encoding.

Example

proxy.listen({ port: 80 });

Stops the proxy listening.

Example

proxy.close();

Adds a function to the list of functions to get called if an error occures.

Arguments

  • fn(ctx, err, errorKind) - The function to be called on an error.

Example

proxy.onError(function(ctx, err, errorKind) {
  // ctx may be null
  var url = (ctx && ctx.clientToProxyRequest) ? ctx.clientToProxyRequest.url : "";
  console.error(errorKind + ' on ' + url + ':', err);
});

Allows the default certificate name/path computation to be overwritten.

The default behavior expects keys/{hostname}.pem and certs/{hostname}.pem files to be at self.sslCaDir.

Arguments

  • hostname - Requested hostname.
  • callback - The function to be called when certificate files' path were already computed.

Example 1

proxy.onCertificateRequired = function(hostname, callback) {
  return callback(null, {
    keyFile: path.resolve('/ca/certs/', hostname + '.key'),
    certFile: path.resolve('/ca/certs/', hostname + '.crt')
  });
};

Example 2: Wilcard certificates

proxy.onCertificateRequired = function(hostname, callback) {
  return callback(null, {
    keyFile: path.resolve('/ca/certs/', hostname + '.key'),
    certFile: path.resolve('/ca/certs/', hostname + '.crt'),
    hosts: ["*.mydomain.com"]
  });
};

Allows you to handle missing certificate files for current request, for example, creating them on the fly.

Arguments

  • ctx - Context with the following properties
  • hostname - The hostname which requires certificates
  • data.keyFileExists - Whether key file exists or not
  • data.certFileExists - Whether certificate file exists or not
  • files - missing files names (files.keyFile, files.certFile and optional files.hosts)
  • callback - The function to be called to pass certificate data back (keyFileData and certFileData)

Example 1

proxy.onCertificateMissing = function(ctx, files, callback) {
  console.log('Looking for "%s" certificates',   ctx.hostname);
  console.log('"%s" missing', ctx.files.keyFile);
  console.log('"%s" missing', ctx.files.certFile);

  // Here you have the last chance to provide certificate files data
  // A tipical use case would be creating them on the fly
  //
  // return callback(null, {
  //   keyFileData: keyFileData,
  //   certFileData: certFileData
  // });
  };

Example 2: Wilcard certificates

proxy.onCertificateMissing = function(ctx, files, callback) {
  return callback(null, {
    keyFileData: keyFileData,
    certFileData: certFileData,
    hosts: ["*.mydomain.com"]
  });
};

Adds a function to get called at the beginning of a request.

Arguments

  • fn(ctx, callback) - The function that gets called on each request.

Example

proxy.onRequest(function(ctx, callback) {
  console.log('REQUEST:', ctx.clientToProxyRequest.url);
  return callback();
});

Adds a function to get called for each request data chunk (the body).

Arguments

  • fn(ctx, chunk, callback) - The function that gets called for each data chunk.

Example

proxy.onRequestData(function(ctx, chunk, callback) {
  console.log('REQUEST DATA:', chunk.toString());
  return callback(null, chunk);
});

Adds a function to get called when all request data (the body) was sent.

Arguments

  • fn(ctx, callback) - The function that gets called when all request data (the body) was sent.

Example

var chunks = [];

proxy.onRequestData(function(ctx, chunk, callback) {
  chunks.push(chunk);
  return callback(null, chunk);
});

proxy.onRequestEnd(function(ctx, callback) {
  console.log('REQUEST END', (Buffer.concat(chunks)).toString());
  return callback();
});

Adds a function to get called at the beginning of the response.

Arguments

  • fn(ctx, callback) - The function that gets called on each response.

Example

proxy.onResponse(function(ctx, callback) {
  console.log('BEGIN RESPONSE');
  return callback();
});

Adds a function to get called for each response data chunk (the body).

Arguments

  • fn(ctx, chunk, callback) - The function that gets called for each data chunk.

Example

proxy.onResponseData(function(ctx, chunk, callback) {
  console.log('RESPONSE DATA:', chunk.toString());
  return callback(null, chunk);
});

Adds a function to get called when the proxy request to server has ended.

Arguments

  • fn(ctx, callback) - The function that gets called when the proxy request to server as ended.

Example

proxy.onResponseEnd(function(ctx, callback) {
  console.log('RESPONSE END');
  return callback();
});

Adds a function to get called at the beginning of websocket connection

Arguments

  • fn(ctx, callback) - The function that gets called for each data chunk.

Example

proxy.onWebSocketConnection(function(ctx, callback) {
  console.log('WEBSOCKET CONNECT:', ctx.clientToProxyWebSocket.upgradeReq.url);
  return callback();
});

Adds a function to get called for each WebSocket message sent by the client.

Arguments

  • fn(ctx, message, flags, callback) - The function that gets called for each WebSocket message sent by the client.

Example

proxy.onWebSocketSend(function(ctx, message, flags, callback) {
  console.log('WEBSOCKET SEND:', ctx.clientToProxyWebSocket.upgradeReq.url, message);
  return callback(null, message, flags);
});

Adds a function to get called for each WebSocket message received from the server.

Arguments

  • fn(ctx, message, flags, callback) - The function that gets called for each WebSocket message received from the server.

Example

proxy.onWebSocketMessage(function(ctx, message, flags, callback) {
  console.log('WEBSOCKET MESSAGE:', ctx.clientToProxyWebSocket.upgradeReq.url, message);
  return callback(null, message, flags);
});

Adds a function to get called for each WebSocket frame exchanged (message, ping or pong).

Arguments

  • fn(ctx, type, fromServer, data, flags, callback) - The function that gets called for each WebSocket frame exchanged.

Example

proxy.onWebSocketFrame(function(ctx, type, fromServer, data, flags, callback) {
  console.log('WEBSOCKET FRAME ' + type + ' received from ' + (fromServer ? 'server' : 'client'), ctx.clientToProxyWebSocket.upgradeReq.url, data);
  return callback(null, data, flags);
});

Adds a function to the list of functions to get called if an error occures in WebSocket.

Arguments

  • fn(ctx, err) - The function to be called on an error in WebSocket.

Example

proxy.onWebSocketError(function(ctx, err) {
  console.log('WEBSOCKET ERROR:', ctx.clientToProxyWebSocket.upgradeReq.url, err);
});

Adds a function to get called when a WebSocket connection is closed

Arguments

  • fn(ctx, code, message, callback) - The function that gets when a WebSocket is closed.

Example

proxy.onWebSocketClose(function(ctx, code, message, callback) {
  console.log('WEBSOCKET CLOSED BY '+(ctx.closedByServer ? 'SERVER' : 'CLIENT'), ctx.clientToProxyWebSocket.upgradeReq.url, code, message);
  callback(null, code, message);
});

Adds a module into the proxy. Modules encapsulate multiple life cycle processing functions into one object.

Arguments

  • module - The module to add. Modules contain a hash of functions to add.

Example

proxy.use({
  onError: function(ctx, err) { },
  onCertificateRequired: function(hostname, callback) { return callback(); },
  onCertificateMissing: function(ctx, files, callback) { return callback(); },
  onRequest: function(ctx, callback) { return callback(); },
  onRequestData: function(ctx, chunk, callback) { return callback(null, chunk); },
  onResponse: function(ctx, callback) { return callback(); },
  onResponseData: function(ctx, chunk, callback) { return callback(null, chunk); },
  onWebSocketConnection: function(ctx, callback) { return callback(); },
  onWebSocketSend: function(ctx, message, flags, callback) { return callback(null, message, flags); },
  onWebSocketMessage: function(ctx, message, flags, callback) { return callback(null, message, flags); },
  onWebSocketError: function(ctx, err) {  },
  onWebSocketClose: function(ctx, code, message, callback) {  },
});

node-http-mitm-proxy provide some ready to use modules:

  • Proxy.gunzip Gunzip response filter (uncompress gzipped content before onResponseData and compress back after)
  • Proxy.wildcard Generates wilcard certificates by default (so less certificates are generated)

Adds a stream into the request body stream.

Arguments

  • stream - The read/write stream to add in the request body stream.

Example

ctx.addRequestFilter(zlib.createGunzip());

Adds a stream into the response body stream.

Arguments

  • stream - The read/write stream to add in the response body stream.

Example

ctx.addResponseFilter(zlib.createGunzip());

License

Copyright (c) 2015 Joe Ferner

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:



The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.



THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

node-http-mitm-proxy's People

Contributors

acklenx avatar apollon77 avatar arasmussen avatar avistramer avatar bjowes avatar cvle avatar danbuk avatar dependabot[bot] avatar devhercule avatar felicienfrancois avatar freewil avatar joeferner avatar jonluca avatar kdzwinel avatar longnguyen2004 avatar mickhansen avatar noonat avatar normanzb avatar pauloasilva avatar rastapasta avatar rich-b avatar rickyabell avatar smsohan avatar spratt avatar sweet-peach avatar switer avatar treffynnon avatar vandenoever avatar victorb avatar wisec avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-http-mitm-proxy's Issues

[feature] Add support for websocket

Add support for proxying websocket (ws and wss protocols).

Useful to proxify website using this technology.
On top of that websocket is often mixed up with http/https in browser proxy settings.

SSL requests - certificate problem

Hi

I am having trouble with HTTS connections passing through the proxy. Its seems that somehow these do not validate against the self signed SSL keys. I am able to see that the proxy works fine if I manually make an exception in Firefox, but these should work without having to manually make an exception after adding the keys to the keychain.

See below for the quests, I even tried to use "--cacert certificate" but it still fails.

curl -v --proxy https://localhost:8080 https://google.com

  • Rebuilt URL to: https://google.com/
  • Hostname was NOT found in DNS cache
  • Trying ::1...
  • Connected to localhost (::1) port 8080 (#0)
  • Establish HTTP proxy tunnel to google.com:443

    CONNECT google.com:443 HTTP/1.1
    Host: google.com:443
    User-Agent: curl/7.37.1
    Proxy-Connection: Keep-Alive

    < HTTP/1.1 200 OK
    <
  • Proxy replied OK to CONNECT request
  • SSL certificate problem: Invalid certificate chain
  • Closing connection 0
    curl: (60) SSL certificate problem: Invalid certificate chain
    More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Thanks,
Mauro

Mixed chunks when simultaneous requests

I'm pretty sure this is an issue of my code and not this library but wanted to ask to make sure.

I'm doing the following, in purpose of caching requests locally:

var Proxy = require('http-mitm-proxy')
var proxy = Proxy()
var Immutable = require('immutable')
var crypto = require('crypto')

var port = 8081

var Cache = Immutable.Map({})

const idFromReq = (ctx) => {
  const req = ctx.clientToProxyRequest
  const url = req.url
  const host = req.headers.host
  return crypto.createHash('md5').update(host + url).digest('hex')
}

const saveCache = (ctx, chunks) => {
  const id = idFromReq(ctx)
  return Cache.set(id, Immutable.Map({
    chunks: chunks,
    statusCode: ctx.serverToProxyResponse.statusCode,
    headers: ctx.serverToProxyResponse.headers
  }))
}

proxy.onError(function (ctx, err) {
  console.error('proxy error:', err)
})

proxy.onRequest(function (ctx, callback) {
  ctx.chunks = []
  ctx.use(Proxy.gunzip)
  if (Cache.get(idFromReq(ctx)) !== undefined) {
    console.log('Using cache for ' + idFromReq(ctx))
    const cached = Cache.get(idFromReq(ctx))
    ctx.proxyToClientResponse.writeHead(cached.get('statusCode'), cached.get('headers'))
    ctx.proxyToClientResponse.end(Buffer.concat(cached.get('chunks')))
    return
  }
  proxy.onResponseData(function (ctx, chunk, dataCallback) {
    // var new_chunk = chunk.toString()
    // if (new_chunk.indexOf('integrity=') !== -1) {
    //   new_chunk = new_chunk.replace(/(integrity=".*?")/g, '')
    // }
    ctx.chunks.push(chunk)
    return dataCallback(null, chunk)
  })

  proxy.onResponseEnd(function (ctx, endCallback) {
    Cache = saveCache(ctx, ctx.chunks)
    const chunks = ctx.chunks
    return endCallback(null, Buffer.concat(chunks))
  })
  callback()
})

proxy.listen({ port: port })
console.log('listening on ' + port)

Which is working fine when I do one request and then another request, after each other. But it fails when making more than one request at the same time, seems to be an issue where the chunks are getting mixed up.

Any pointers to what I could be doing wrong?

Publish to npm?

I was wondering why this cool project is not published to npm.
Is it a choice?

You can publish it to npm by creating an account on npmjs.org and then executing the following command at the root of the project:

npm login
# then enter your login and passwod
npm publish

SSL3_GET_CLIENT_HELLO no shared cipher

I've set things up using the generated CA.perm as a trusted root CA. I see in the logs often errors like this:

W20160222-16:07:01.200(-3)? (STDERR) HTTPS_CLIENT_ERROR on : [Error: 140735123144704:error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher:../deps/openssl/openssl/ssl/s3_srvr.c:1436:

This seems to be related to problems with some kinds of web socket connections over SSL, though I'm not yet 100% clear on the association. It may be that all examples produce the error but https://github.com/joeferner/node-http-mitm-proxy/blob/master/examples/processFullResponseBody.js definitely does.

Modify all chunks at once

Hi,
is it possible to modify all chunks at once? The response is a big json document and to parse it, I need to get all chunks first.
With the default https.request I could collect all chunks with the on data event and push all concated chunks in the on end event. I found the onResponseEnd event in the script, but I don't know how to return the combined chunks at once.

Ignore Certificate Errors on down steam servers

I have a server that returns an invalid certificate, how can I set the proxy not to reject the invalid cert on a particular request?

I can set, process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0"; but that's not a nice solution, I want to check the host on the request header and accept all certificates only for certain hosts, is this possible?

Thanks

method chaining

Small feature request. IMO I'd be a bit more convenient to write:

proxy.onRequest(handleIncomingRequest.bind(this))
     .onResponse(handleIncomingResponse.bind(this))
     .onError(handleProxyError.bind(this))
     .listen({port: port});

instead of:

  proxy.onRequest(handleIncomingRequest.bind(this));
  proxy.onResponse(handleIncomingResponse.bind(this));
  proxy.onError(handleProxyError.bind(this));
  proxy.listen({port: port});

Do not close the HTTPS servers.

Hey, as long as the module is running in the system, I see a growing number of open ports. How to clean unnecessary open HTTPS servers?

I tried this way on ctx.onResponseEnd:

(Object.keys(proxy.sslServers)).forEach(function (srvName) {
    if (proxy.sslServers[srvName].server._connections === 0 && !proxy.sslServers[srvName].server._handle.reading && proxy.sslServers[srvName].server._handle.writeQueueSize === 0) {
        console.log('Can: ', proxy.sslServers[srvName].server);
        proxy.sslServers[srvName].server.close();
        delete proxy.sslServers[srvName];
        console.log('HTTPS Server: "' + srvName.yellow + '" closed.');
    } else {
        console.log('Cannot: ', proxy.sslServers[srvName].server);
    }
});

Did not work. Proxy client loses the connection and make mistakes. Please help.

Buffering requests/responses

Hello,

For my personal use, I've implemented buffering of requests and responses (including automatic gunzip) as a layer on top of this library. I don't think it is particularly useful to rewrite data at a chunk level, but admittedly good for performance.

I would like to do a pull request, but I'm at a loss on how to proceed and wanted to get some opinions and gauge interest.

Perhaps ctx.onRequestBody and ctx.onResponseBody?? This could possibly have weird interactions with .use middleware.

Let me know your thoughts

support to proxy chain

I didn't find a way to set a proxy in mitm-proxy to have multiple proxies chain.
Something like

Browser -> Node-Http-Mitm-Proxy -> Internal Squid Proxy -> Internet

Is there any workaround?

HTTP on port 443 and HTTPS on port != 443

Currently, the proxy pipe the request either to an internal http or https server depending on the port of the request.

As a result HTTP requests made on port 443 (not usual) and HTTPS requests made on any port different than 443 (more frequent) are not proxied correctly.

That's also the case for websocket requests
Here a usecase for not secure websockets on port 443: http://websocketstest.com/

What is the good way to cancel the default behavior?

Examples:

proxy.onRequest(function(ctx, callback) {
  // I want to answer the request myself, without making a request to a server  

  // callback() would create the request to the server with the ctx.proxyToServerRequestOptions
  // callback();

  // callback("Any error") would trigger a "Proxy error"
  // callback("Any error");

  // No callback may not be clean and may have side effect, like not calling next handlers
});
proxy.onResponseData(function(ctx, chunk, callback) {
  // I want to collect all chunks without sending it to the client and then send something to the client at the ResponseEnd

  // callback(chunk) would send the chunk to the client
  // callback(null, chunk);

  // callback(null, null) or callback(null, []) may break or at least send the headers and is not clean at all
  // callback(null, null);
  // callback(null, []);

  // callback("Any error") would trigger a "Proxy error"
  // callback("Any error");

  // No callback may not be clean and may have side effect, like not calling next handlers and may lead to no responseEnd event
});

Maybe the best way is to rewrite the API with a switch to prevent default operation after the handler execution.
Maybe the most transparent and intuitive way would be to test the second parameter of the callback and cancel default behavior if it is equals to false or null (but not undefined)

onResponseEnd?

First of all great library.

I'm wondering how to detect the end of a response though - onResponseData might fire multiple times for a chunked response and i don't currently see a way to detect that something is the final chunk.

Have i completely missed something or is it not implemented yet?

Performance is inferior to Firefox's nsITraceableChannel

I am developing a Firefox addon which rewrites incoming traffic using nsITraceableChannel, and I tried to decouple it from Firefox and put the logic into a separate HTTP proxy using this package. I knew doing MITM with HTTPS would use a little more resources due to SSL/gzip decoding/encoding, but the proxy is much slower, to the point that I can't even use it for testing. And it doesn't even seem like the proxy is hitting the CPU too hard, it's just slow! Is the slower performance normal, or do I definitely have some sort of bug that severely degrades performance? I am caching requests with the following helper function, based on the examples in tests/:

ctx.cacheAndModify = function(f) {
  var data = "";
  ctx.onResponseData(function(ctx, chunk, cb) {
    data += chunk.toString();
    return cb(null, null);
  });
  ctx.onResponseEnd(function(ctx, cb) {
    var opts = ctx.proxyToServerRequestOptions;
    var url = (ctx.isSSL ? "https" : "http") + "://" + opts.host +
      (opts.port === 443 || opts.port === 80 ? "" : ":" + opts.port) +
      opts.path;
    ctx.proxyToClientResponse.write(new Buffer(f(data, url)));
    return cb();
  });
};

so then I can just use ctx.cacheAndModify(handler) in onResponse. I also use Proxy.gunzip

Dies on HTTP requests with no Host: header

I was wondering if the proxy could be made to serve the cert file directly to HTTP clients. It seems to be easily confused by non-proxy HTTP requests right now.

GET / HTTP/1.0 results in:

.../http-mitm-proxy/lib/proxy.js:708                                                                                                                                                       
    host: hostPort.host,                                                                                                                                                                                                          
                  ^                                                                                                                                                                                                               

TypeError: Cannot read property 'host' of null                                                                                                                                                                                    
  at [object Object].Proxy._onHttpServerRequest (.../http-mitm-proxy/lib/proxy.js:708:19)                                                                                                  
  at emitTwo (events.js:87:13)                                                                                                                                                                                                    
  at Server.emit (events.js:172:7)                                                                                                                                                                                                
  at HTTPParser.parserOnIncoming [as onIncoming] (_http_server.js:528:12)                                                                                                                                                         
  at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)                                                                                                                                                                   

Documentation gap?

Installed the ca.pem file, trusted it. but connecting to a running example proxy raises
HTTPS_CLIENT_ERROR on : [Error: 140735258857472:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../deps/openssl/openssl/ssl/s3_pkt.c:1472:SSL alert number 48

This is likely because i am doing something wrong, but i am having a hard time understanding what i am missing by looking at the readme.

onRequestEnd

Maybe I'm missing something, but how do I know if client finished sending request body? For responses we have onResponseEnd, but there is no equivalent for requests. From what I've tested multiple chunks are possible (e.g. when trying to POST a 5m file).

HTTPS + Curl fails

Given the following command: curl -i -x https://localhost:3001 https://www.google.co.uk/

After launching the server, I'm first getting a response of Server aborted the SSL handshake

The next response is Unknown SSL protocol error in connection

The server is outputting HTTPS_CLIENT_ERROR on : [Error: TLS handshake timeout]

I've added the ca.pem into my OSX keychain.

When using a different command line tool, I get HTTPS_CLIENT_ERROR on : [Error: 19282344:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/s3_pkt.c:1472:SSL alert number 42 19282344:error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure:../deps/openssl/openssl/ssl/s3_pkt.c:1210

The server is based on your example. Any way to resolve this?

Not intercepting

Here is the code

var Proxy = require('http-mitm-proxy');
var proxy = Proxy();

proxy.onError(function(ctx, err) {
    console.error('proxy error:', err);
});

proxy.onRequest(function(ctx, callback) {
    console.log(ctx.clientToProxyRequest.headers.host)

    return callback();
});

proxy.listen({
    port: 8080
});

Im starting my server node index.js and refreshing google.com and dont see any console logging?

modifyGoogle example does not work

Hi,

I've tried out your modifyGoogle example, but I always got this error:

PROXY_TO_SERVER_REQUEST_ERROR on /: { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
PROXY_TO_SERVER_REQUEST_ERROR on /favicon.ico: { [Error: socket hang up] code: 'ECONNRESET' }

Do you have any idea why?

I installed the certificate.

Thx

need ca.crt for some androids (not ca.pem)

The ca.pem provided here does not work for my android (n). I can get it to work by generating a ca.crt from your ca.pem like so:

openssl x509 -inform PEM -outform DM -in .http-mitm-proxy/certs/ca.pem -out .http-mitm-proxy/certs/ca.crt

but this requires the openssl binary which may not always be easy to come by. Could the project here directly generate the ca.crt that many of us need?
see: justinleewells/pogo-optimizer#108

Failed to load resource: The network connection was lost.

I've got a basic HTTP/HTTPS proxy running:

var Proxy = require('http-mitm-proxy');
var proxy = Proxy();

proxy.onError( (ctx, err, errorKind) => {
  var url = (ctx && ctx.clientToProxyRequest) ? ctx.clientToProxyRequest.url : '';
  console.error(errorKind + ' on ' + url + ':', err);
});

proxy.use(Proxy.gunzip);

proxy.listen({port: 8089, sslCaDir: "/Users/peter/Development/fetching-proxy/certs"});

I notice that Safari (possibly other browsers) frequently show this kind of error in the console for JS and CSS assets:

Failed to load resource: The network connection was lost.

I'm not sure they are related, but the node console also has many:

HTTPS_CLIENT_ERROR on : { [Error: socket hang up] code: 'ECONNRESET' }

and

PROXY_TO_SERVER_REQUEST_ERROR on /510080801/: { [Error: connect ETIMEDOUT 31.13.85.8:443]

Needless to say this causes most websites to have trouble loading.

Thank you!

ssl handshake failure

I'm seeing a lot of these errors thrown when intercepting HTTPS traffic:

Error: 140735158005760:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../../vendor/node/deps/openssl/openssl/ssl/s3_pkt.c:1472:SSL alert number 48
140735158005760:error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure:../../vendor/node/deps/openssl/openssl/ssl/s3_pkt.c:1210:

I'm able to reproduce it on node 4.1.1 and 5.1.1 for multiple hosts.

Error: socket hang up, ECONNRESET

First, thanks for making this!

Second, using your examples I've got a proxy running. On some HTTPS sites it works as expected however on some it fails with the error:

proxy error: { [Error: socket hang up] code: 'ECONNRESET', sslError: undefined }

For example https://github.com and https://en.wikipedia.org fail to load. Safari, for example, indicates that it's not able to establish a secure connection to the web server.

Any suggestions?

Error: this.readyState !== WebSocket.OPEN

Hi,

We are using proxy as part of our testing infrastructure to block javascript (with webdriverio). We are now constantly, getting this error:

image

We are running on Node 7 and latest version http-mitm.

We also get the deprecation warning too.

Could you please advice, as its a bit urgent.

Thanks a lot!
Jeremy

WebSocket ping/pong

On top of the message frame kind WebSocket also have ping and pong frame types that are mostly used to keep alive a websocket connection but can also carry data.

Thoose frame kind are currently not handled by node-http-mitm-proxy (not forwarded).
As a result websocket connection may not be kept alive and broke on some websites.

SSL makeConnection race condition

Hi Joe,
Congrats for your work on this module.

As you can imagine (because of my pull requests) I am working on it to enable SSL signing on the fly.
I have already extended the module adding an onCertificateRequired and onCertificateMissing "hooks" (as soon as I can test it, I will do a pull request).

Meanwhile I am having some troubles and I would like to have your feedback.
On Proxy.prototype._onHttpServerConnect the makeConnection call is async, what I think will cause problems with incoming requests for the same hostname which will find var sslServer = this.sslServers[hostname]; undefined and start a "race condition".

Do you agree that the SSL connection logic has to be sync?
I will be working in this but I would really appreatiate your feedback.

Regards,
Paulo

Unable to verify the first certificate

Hello,
when we have self signed certificates, the proxy does triggers an error.
curl -x 127.0.0.1:8082 https://127.0.0.1 -ki
HTTP/1.1 200 OK

HTTP/1.1 504 Proxy Error
Date: Mon, 22 Feb 2016 20:41:20 GMT
Connection: keep-alive
Transfer-Encoding: chunked

PROXY_TO_SERVER_REQUEST_ERROR: Error: unable to verify the first certificate

I resolved using:
NODE_TLS_REJECT_UNAUTHORIZED=0 node myproxy.js

 //myproxy.js
Proxy().listen({port: 8082});

but I think It would be nice:

  1. Since it's a proxy, to allow self signed certs to go through by default.
  2. to have some way to easily set this option to true or false from the code.

Am I missing something?
Thanks and keep up the great work!

How to tunnel https directly ?

http-mitm-proxy without "onConnect" exposing, how can I tunnel https directly without create httpsServer ?

In some case, proxy https request doesn't need decrypting : )

Error using example

I tried the example in the project's page and it doesn't work, I get the following error even though the port is specified as 8081:

internal/net.js:17
    throw new RangeError('"port" argument must be >= 0 and < 65536');
RangeError: "port" argument must be >= 0 and < 65536

Here's the code I used:

var Proxy = require('http-mitm-proxy');
var proxy = Proxy();

proxy.onError(function(ctx, err) {
  console.error('proxy error:', err);
});

proxy.onRequest(function(ctx, callback) {
  if (ctx.clientToProxyRequest.headers.host == 'www.google.com'
    && ctx.clientToProxyRequest.url.indexOf('/search') == 0) {
    ctx.use(Proxy.gunzip);

    ctx.onResponseData(function(ctx, chunk, callback) {
      chunk = new Buffer(chunk.toString().replace(/<h3.*?<\/h3>/g, '<h3>Pwned!</h3>'));
      return callback(null, chunk);
    });
  }
  return callback();
});

proxy.listen({port: 8081});

Any ideas why this is not working? not even the first example works which may suggest something went awry with the current version

Usage question.

Thank you for making such great software available. I would like to intercept requests for a single domain and url and cause the request to be proxied to a different remote host. I've gotten as far as matching the request and sending a different response from the proxy to the client but that's it.

ENOTFOUND error -- how to catch?

Trying to figure out how to catch an ENOTFOUND error (from request made to the proxy using URL with a non-existent host that fails dns lookup) or similar request level errors -- they become uncaught exceptions and crash the process. Of course I can use

process.on('uncaughtException', function(err) ...

but that doesn't really help respond to bad request (it isn't tied to specific request) and leaves the process in an unstable state. I'm just not experienced enough with node to figure this out. (Simply avoiding bad requests is impossible as many come from secondary requests from a site, for instance it may link to a .css file on a non-existent host.)

Add linting rules

It is extremely hard to contribute to the codebase when there are no linting rules/ code style is inconsistent between files.

I suggest to use one of the existing style guides, https://github.com/dustinspecker/awesome-eslint#configs:

  • Canonical is the most strict of them all (I am biased though, since it is the package that I maintain).
  • AirBnb is the most popular.
  • Standard while popular, is very loose

@joeferner Do you agree that linting rules need to be added? Do you have preference for which style guide to use?

Gunzip throw error on partial chunks

Steps to reproduce:

  1. node examples/processFullResponseBody.js
  2. Open Chrome (with proxy setup)
  3. Navigate to http://humblebundle.com (should succeed)
  4. Refresh page, and see console output for exception below.
$ node examples/processFullResponseBody.js
listening on 8081

...some lines removed, everything below is after refresh...

starting server for humble.pubnub.com
https server started for humble.pubnub.com on 56619
starting server for www.humblebundle.com
https server started for www.humblebundle.com on 56888
starting server for humblebundle-a.akamaihd.net
https server started for humblebundle-a.akamaihd.net on 56958
starting server for checkout.stripe.com
https server started for checkout.stripe.com on 56969
starting server for www.youtube.com
https server started for www.youtube.com on 56995
starting server for connect.facebook.net
https server started for connect.facebook.net on 56999
starting server for apis.google.com
https server started for apis.google.com on 57005
starting server for platform.twitter.com
https server started for platform.twitter.com on 57008
starting server for chart.googleapis.com
https server started for chart.googleapis.com on 57012
events.js:154
      throw er; // Unhandled 'error' event
      ^

Error: unexpected end of file
    at Zlib._handle.onerror (zlib.js:363:17)

forged certificate from date can be out of range due to clock skew.

When I MitM myself chrome stable returns quite a few errors due to the cert being outside the valid date range even though my clocks are synchronized between the machine that's acting as my proxy and my desktop. Adjusting the from date to 2 days back removed all the errors. The change is trivial and I'm not sure how far back you would want to go, so I'm just attaching the diff. Perhaps a better alternative would be to match the to/from date of the original cert? I can submit a pull request for either solution if there's interest.

Thanks,
-Matt

$ git diff
diff --git a/lib/ca.js b/lib/ca.js
index 4fb83c1..7345ba2 100644
--- a/lib/ca.js
+++ b/lib/ca.js
@@ -200,7 +200,8 @@ CA.prototype.generateServerCertificateKeys = function (hosts, cb) {
   var certServer = pki.createCertificate();
   certServer.publicKey = keysServer.publicKey;
   certServer.serialNumber = this.randomSerialNumber();
-  certServer.validity.notBefore = new Date();
+  var date = new Date();
+  certServer.validity.notBefore.setDate(date.getDate()-2);
   certServer.validity.notAfter = new Date();
   certServer.validity.notAfter.setFullYear(certServer.validity.notBefore.getFullYear() + 2);
   var attrsServer = ServerAttrs.slice(0);

Error while using http-mitm-proxy.

I got this error when i am trying to use "http-mitm-proxy".

/home/shark_s/node_modules/http-mitm-proxy/lib/middleware/wildcard.js:8
const HOSTNAME_REGEX = /^(.+)(.[^.]{4,}(.[^.]{1,3})*.[^.]+)$/;
^^^^^
SyntaxError: Use of const in strict mode.
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/home/shark_s/node_modules/http-mitm-proxy/lib/proxy.js:24:27)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)

Install Issue

When I install with the command line:
npm install --save http-mitm-proxy
I get the following message and the install stops:
npm WARN install Refusing to install http-mitm-proxy as a dependency of itself

Request timeout after 2min

Currently requests timeout after 2min of inactivity
This is due to nodejs http & https servers default timeout

This may be an issue in some cases.
For example in case of a request upgraded to websocket. WebSocket connection does not timeout by default and some websocket servers does not have any timeout. In such case, the websocket connection is unexpectedly closed after 2min of inactivity.

The proxy should not enforce any timeout by default and rely on the server instead.

Documentation for proxy.onCertificateMissing

Hello,

In your documentation you have:

proxy.onCertificateMissing = function(ctx, files, callback) {
  console.log('Looking for "%s" certificates',   ctx.hostname);
  console.log('"%s" missing', ctx.files.keyFile);
  console.log('"%s" missing', ctx.files.certFile);

  // Here you have the last chance to provide certificate files data
  // A tipical use case would be creating them on the fly
  //
  // return callback(null, {
  //   key: keyFileData,
  //   cert: certFileData
  // });
  };

The required keys for the object sent to the callback are actually keyFileData and certFileData.

Proxy Authentication headers missing from HTTPS requests

Hello! I've attempted your suggested approach to implement basic authentication (included below). However I'm seeing that HTTPS requests do not include the same header information that HTTP requests include. For example, this HTTP request includes the expected headers:

The CURL request:

curl http://stackoverflow.com -v -x http://username:[email protected]:8089

* Rebuilt URL to: http://stackoverflow.com/
*   Trying 127.0.0.1...
* Connected to myproxy.com (127.0.0.1) port 8089 (#0)
* Proxy auth using Basic with user 'username'
> GET http://stackoverflow.com/ HTTP/1.1
> Host: stackoverflow.com
> Proxy-Authorization: Basic cGV0ZXI6cGFzc3dvcmQ=
> User-Agent: curl/7.43.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< date: Sat, 12 Mar 2016 16:03:29 GMT
< content-type: text/html; charset=utf-8
< transfer-encoding: chunked
< connection: close
< set-cookie: __cfduid=d0a207fda936835992317de3be8a24ecc1457798609; expires=Sun, 12-Mar-17 16:03:29 GMT; path=/; domain=.stackoverflow.com; HttpOnly
< set-cookie: prov=359b8d31-d451-4ea5-8a12-8824bb92b971; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
< cache-control: public, no-cache="Set-Cookie", max-age=60
< expires: Sat, 12 Mar 2016 16:04:29 GMT
< last-modified: Sat, 12 Mar 2016 16:03:29 GMT
< vary: *
< x-frame-options: SAMEORIGIN
< x-request-guid: 91d04c64-9c57-4475-a188-e917ec63193d
< server: cloudflare-nginx
< cf-ray: 282879fbf28c2a5b-SEA
< 

The node-http-mite-proxy request headers:

{ 
  host: 'stackoverflow.com',
  'proxy-authorization': 'Basic cGV0ZXI6cGFzc3dvcmQ=',
  'user-agent': 'curl/7.43.0',
  accept: '*/*',
  'proxy-connection': 'Keep-Alive' 
}

Whereas an HTTPS request, while showing similar header information in the CURL request, does not include the same headers in the proxy:

The CURL request

curl https://github.com -v -x http://username:[email protected]:8089

* Rebuilt URL to: https://github.com/
*   Trying 127.0.0.1...
* Connected to my proxy.com (127.0.0.1) port 8089 (#0)
* Establish HTTP proxy tunnel to github.com:443
* Proxy auth using Basic with user 'username'
> CONNECT github.com:443 HTTP/1.1
> Host: github.com:443
> Proxy-Authorization: Basic cGV0ZXI6cGFzc3dvcmQ=
> User-Agent: curl/7.43.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< 
* Proxy replied OK to CONNECT request
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: github.com
* Server certificate: NodeMITMProxyCA
> GET / HTTP/1.1
> Host: github.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 407 Proxy Authentication Required
* Authentication problem. Ignoring this.
< Proxy-Authenticate: Basic
< Date: Sat, 12 Mar 2016 16:06:34 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
< 
* Connection #0 to host local.fetching.io left intact

The headers included in the proxy:

{ 
  host: 'github.com',
  'user-agent': 'curl/7.43.0',
  accept: '*/*' 
}

My proxy code:

'use strict';

const Proxy = require('http-mitm-proxy');
const proxy = Proxy();
const logger = require('./logger.js');

proxy.onRequest( (ctx, callback) => {
  let request = ctx.clientToProxyRequest;
  let host = request.headers['host'];
  let path = request.url;
  let protocol = ctx.isSSL ? 'https' : 'http';
  let url = `${protocol}://${host}${path}`;
  let proxyAuthorization = request.headers['proxy-authorization'];

  console.log('DEBUG', url, request.headers);

  if (proxyAuthorization) {
    let encodedLoginPassword = proxyAuthorization.replace('Basic ', '');
    let loginPassword = new Buffer(encodedLoginPassword, 'base64').toString().split(':');
    let login = loginPassword[0];
    let password = loginPassword[1];
    callback();
  } else {
    let response = ctx.proxyToClientResponse;
    response.writeHead(407, {'Proxy-Authenticate': 'Basic'});
    response.end();
  }
});

proxy.onRequest( (ctx, callback) => {
  let request = ctx.clientToProxyRequest;
  let proxyAuthorization = request.headers['proxy-authorization'];
  callback();
});

proxy.onError( (ctx, err, errorKind) => {
  let url = (ctx && ctx.clientToProxyRequest) ?
    ctx.clientToProxyRequest.url :
    '';
  logger.log('error', `${errorKind} on ${url}`, err);
});

proxy.listen({
  port: 8089,
  forceSNI: true,
  silent: true,
  sslCaDir: "~/Development/proxy/certs"
});

In short, the proxy authentication works for HTTP requests but doesn't work for HTTPS requests and I'm not sure what accounts for the difference.

Thank you!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.