dvonlehman / express-request-proxy Goto Github PK

I'm interested in using express-request-proxy to dynamically proxy endpoints. I know this isn't a documented use case, but it's not so hard to do:

const requestProxy = require('express-request-proxy');

function someMiddleware(req, res, next) {
  return requestProxy({ url: dynamicUrl })(req, res, next);
}

The overhead of this extra function call shouldn't be a problem, but I'm a little hesitant to be creating the in-scope functions (such as this one) for each request when that could be avoided.

Would you be open to a PR that would pull those outside of the main export function so that they're shared among all proxies? It would require a slight modification to their signature to accept an options argument. This way, there's less overhead to generating lots and lots (and lots) of proxies.

It'd look something like:

module.exports = function() {
  return function(req, res, next) {
    var options = { ... };
    sharedThing(req, res, next, options);
  }
}

function sharedThing(req, res, next, options) {
 // lala
}

Thanks for reading!

I always experienced this:

Error: API call timed out
at Function.Error.create (/root/proxy/node_modules/simple-errors/index.js:25:11)
at Function.Error.http (/root/proxy/node_modules/simple-errors/index.js:68:21)
at unhandledApiError (/root/proxy/node_modules/express-request-proxy/lib/proxy.js:254:25)
at Request. (/root/proxy/node_modules/express-request-proxy/lib/proxy.js:86:7)
at emitOne (events.js:121:20)
at Request.emit (events.js:211:7)
at Timeout._onTimeout (/root/proxy/node_modules/request/request.js:848:16)
at ontimeout (timers.js:498:11)
at tryOnTimeout (timers.js:323:5)
at Timer.listOnTimeout (timers.js:290:5)

I found that it caches the incomplete result even it is timeout. How to avoid that?

Hi,

I want to acsess the user properties but I cannot even start because req is not defined here
https://github.com/4front/express-request-proxy#logged-in-user-properties

app.all('/api/*', requestProxy({
    url: "URL HERE" + '/*',
    userAgent:false,
    timeout: 5000,
    gzip: true,
    headers: {
        Authorization: "Bearer " + req.session.access_token
    }
}));

error

ReferenceError: req is not defined
    at Object.<anonymous> (/Users/name/git/project/server.js:100:36)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
    at Timeout.Module.runMain [as _onTimeout] (module.js:575:10)
    at tryOnTimeout (timers.js:224:11)
    at Timer.listOnTimeout (timers.js:198:5)

I'm trying to redirect POST request using requestProxy, but couldn't make it work.

Here is my code

router.post('/api/v1/device/register',
    requestProxy({
         url: "http://localhost:3000/api/v1/user/activation"
    }));

Request log

POST /api/v1/device/register 408 5449.497 ms - 975
POST /api/v1/user/activation 400 5455.902 ms - 605

I don't know why the destination url serving HTTP error 400. Please help.

Got a destination URL (which I don't control) as something.com/api?client_id=:id (rather than the more elegant /api/client/:id). When running requestProxy({url: "something.com/api?client_id=:id" }), that actual requested URL ends up being ...?client_id=%3Aid (no interpolation). Tried a quick /api/client/:id sanity-check and that interpolates fine.

I see query support, but not for tokens passed into the URL. Seems tokens only support params. Any thoughts?

I am wondering is there a way to stop express-request-proxy from decoding the URL that is proxying? I asked this question on StackOverFlow as well.

This line is the source of my problem:

I appreciate any help or hint.

I've started facing following error for api response size > 16KB when using redis as a cache. And it ends up returning partial data of 16KB.
with following log.
Error: stream.push() after EOF at readableAddChunk (_stream_readable.js:156:17) at RedisRStream.Readable.push (_stream_readable.js:134:10)

From your project I run npm audit and see it has problem with lodash, it is recommended to upgrade lodash from ver ^4.17.10 to ^4.17.15.

Hello,

I'm trying to use this module to proxy and cache some images coming from external services.
It works when I don't set the cache option.

exports.proxy = function(req, res, next)
{
    if (!req.headers['referer'])
        return res.status(400).send();

    var ref_parts = url.parse(req.headers['referer'], true);
    if (ref_parts.host != config.hostname)
        return res.status(400).send();

    return requestProxy({
        cache: redis.createClient(),
        cacheMaxAge: 3600,
        url: req.query.url
    })(req, res, next);
}

The very same code works for other requests/mime-types such as plain HTML or text.

How can we solve this?

Given the request

http://localhost:3000/epiquery/glgCurrentUser/getUserByLogin.mustache

And the mapping

console.log "Using #{app.get 'episerver'} for Epiquery"
router.all '/*', proxy
  url: "#{app.get 'episerver'}/*"
  headers:
    Authorization: app.get 'epiauth'

Where the epiquery server var is http://localhost:8080

The proxy code is escaping the slashes in the path

http://localhost:9700/glgCurrentUser%2FgetUserByLogin.mustache

The problem seems to be in request-options.js, line 47 - compiledPath regexp is escaping the path

/glgCurrentUser%2FgetUserByLogin.mustache

I'm guessing the fault is in path-to-regexp, but that's as far as I've gotten.

I am writing zipkin client instrumentation for express-request-proxy . I need a way to intercept the request so I can record it before it is sent and then intercept the response (transformers will probably work for the response).

I've already instrumented plain old request (https://github.com/openzipkin/zipkin-js/tree/master/packages/zipkin-instrumentation-request) so if there was a way to pass in a modified request to express-request-proxy like:

const request = require('request');
const wrapRequest = require('zipkin-instrumentation-request');
const proxy = require('express-request-proxy');

const wrappedRequest = wrapRequest(request, {tracer, serviceName, remoteServiceName});
app.get(proxy({url:'http://api.weather.com'}, wrappedRequest);

it would make my life a lot easier. This could also be used if others want to pass in a request.defaults() instance for express-request-proxy to use.

Another option would be to have decorateRequest like express-http-proxy has which I used here (https://github.com/openzipkin/zipkin-js/blob/master/packages/zipkin-instrumentation-express/src/wrapExpressHttpProxy.js) when instrumenting that client.

I keep getting this exception when making a request from a client

TypeError: undefined is not a function
    at proxyViaCache (c:\PROJ\Server\node_modules\express-request-proxy\lib\proxy.js:130:19)
    at c:\PROJ\Server\node_modules\express-request-proxy\lib\proxy.js:57:14
    at Layer.handle [as handle_request] (c:\PROJ\Server\node_modules\express\lib\router\layer.js:95:5)
    at next (c:\PROJ\Server\node_modules\express\lib\router\route.js:131:13)
    at cors (c:\PROJ\Server\node_modules\cors\lib\index.js:178:7)
    at c:\PROJ\Server\node_modules\cors\lib\index.js:228:17
    at originCallback (c:\PROJ\Server\node_modules\cors\lib\index.js:217:15)
    at c:\PROJ\Server\node_modules\cors\lib\index.js:222:13
    at optionsCallback (c:\PROJ\Server\node_modules\cors\lib\index.js:203:9)
    at c:\PROJ\Server\node_modules\cors\lib\index.js:208:7

Line 130 is this:

options.cache.exists(cacheKey, function(err, exists) {

I am specifying the cache like in the example, but my cache doesn't have the "exists()" method that your code is expecting. Here is how I'm initializing things (this is transpiled TypeScript):

var express = require("express");
var redis = require("redis");
var redisStreams = require("redis-streams");
var requestProxy = require("express-request-proxy");
var GW2_API_1 = require("./Shared/GW2-API");
var ms = require("milliseconds");
var cors = require("cors");
redisStreams(redis);
var app = express();

var requestProxyConfig = {
    cache: redis,
    cacheMaxAge: ms.days(1),
    url: GW2_API_1.default.endpointPrefix + "/*",
};

app.get('/gw2-api/*', cors(), requestProxy(requestProxyConfig));

var server = app.listen(8032, function () {
    var host = server.address().address;
    var port = server.address().port;
});

Any idea what the problem might be?

i want to console log success and error response of this proxy request, however i'm not able to find any option to do that:
app.get(
"/api/:resource/:id",
requestProxy({
cache: redis.createClient(),
cacheMaxAge: 60,
url: "https://someapi.com/api/:resource/:id",
query: {
secret_key: process.env.SOMEAPI_SECRET_KEY
},
headers: {
"X-Custom-Header": process.env.SOMEAPI_CUSTOM_HEADER
}
})
);

As this middleware is used to proxy to another server, should the error code on timeout be 504 (Gateway Timeout) rather than 408 (Request Timeout)?
https://github.com/4front/express-request-proxy/blob/master/lib/proxy.js#L265

The issue here is that the HTTP specification for 408 allows clients to repeat the request. It seems that modern browsers (i.e. Chrome) will retry a few times before providing the error code to the client application.
https://tools.ietf.org/html/rfc2068#section-10.4.9
vs
https://tools.ietf.org/html/rfc2068#section-10.5.5

What do you think?

I'm trying to use this nice package to proxy an API endpoint, and then transform the response I get. But I can't seem to get it working. I first tried to use your examples, but they don't seem to be updated. I had a look the tests and found the correct way to define transforms.

When using your example:

module.exports = function(options) {
    var transform = through2(); // Just to test with a simple pass-through stream.
    transform.contentType = 'application/json';
    return transform;
};

I get these errors:

 express-request-proxy applying transform  +370ms
stream.js:45
  dest.on('drain', ondrain);
      ^

TypeError: Cannot read property 'on' of undefined
    at Request.Stream.pipe (stream.js:45:7)
    at Request.pipe (/Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/request/request.js:1330:36)
    at /Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/express-request-proxy/lib/proxy.js:275:23
    at Array.forEach (native)
    at applyTransforms (/Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/express-request-proxy/lib/proxy.js:269:16)
    at Request.<anonymous> (/Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/express-request-proxy/lib/proxy.js:99:22)
    at emitOne (events.js:77:13)
    at Request.emit (events.js:169:7)
    at Request.onRequestResponse (/Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/request/request.js:951:10)
    at emitOne (events.js:77:13)
    at ClientRequest.emit (events.js:169:7)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:415:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
    at Socket.socketOnData (_http_client.js:305:20)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)

When doing it the same way you do in your tests:

module.exports = function() {
  return {
    name: 'test-transform',
    contentType: 'application/json',
    transform: through2()
  };
}

It works for the first request, the transform is properly applied, and returned to the client, but on subsequent requests I get these errors (and the server crashes):

  express-request-proxy applying transform test-transform +167ms
GET /test 200 180.097 ms - -
  express-request-proxy making GET call to http://www.mocky.io/v2/569f75ae2700002b38fb712c +6s
  express-request-proxy applying transform test-transform +151ms
stream.js:74
      throw er; // Unhandled stream error in pipe.
      ^

Error: write after end
    at writeAfterEnd (/Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:144:12)
    at DestroyableTransform.Writable.write (/Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/through2/node_modules/readable-stream/lib/_stream_writable.js:192:5)
    at Request.ondata (stream.js:31:26)
    at emitOne (events.js:77:13)
    at Request.emit (events.js:169:7)
    at IncomingMessage.<anonymous> (/Users/espenhogbakk/code/hyper/sites/cubus/wallet/code/node_modules/request/request.js:959:12)
    at emitOne (events.js:77:13)
    at IncomingMessage.emit (events.js:169:7)
    at IncomingMessage.Readable.read (_stream_readable.js:360:10)
    at flow (_stream_readable.js:733:26)
    at resume_ (_stream_readable.js:713:3)
    at doNTCallback2 (node.js:430:9)
    at process._tickCallback (node.js:344:17)

This is my setup:

var requestProxy = require('express-request-proxy');
var testTransform = require('./transforms/test');

app.get('/test', requestProxy({
  url: 'http://www.mocky.io/v2/569f75ae2700002b38fb712c',
  transforms: [
    testTransform()
  ]
}));

It seems to me to be a "caching" issue. When the library pipes to the transform on the second request, it's for some reason trying to write to the previous (closed) stream.

If i define a transform like this (wrapped in a function call):

module.exports = function() {
  return {
    name: 'test-transform',
    contentType: 'application/json',
    transform: function() { return through2() }
  };
}

And change this line in your code to call that function:

stream = stream.pipe(transform.transform());

It seems to work. I can submit a pull request with updates to the documentation and with this fix, if this is how you want to solve it. Just wanted to check first if I have misunderstood something, and configured it wrong. When reading the readme, you do say:

Transforms are functions which return a Node.js transform stream

function appenderTransform(appendText) {
return {
name: 'appender',
transform: function() {
return through2(function(chunk, enc, cb) {
console.log(chunk);
this.push(chunk + appendText);
cb();
});
}
}
}

proxy({
url: "/abc",
timeout: parseInt(req.headers.timeout) || DEFAULT_TIMEOUT,
originalQuery: req.originalUrl.indexOf("?") >= 0,
transforms :[appenderTransform('XYZ')]
})(req, res, next);

I'm getting this error:
Error: aborted
at connResetException (node:internal/errors:691:14)
at TLSSocket.socketCloseListener (node:_http_client:407:19)
at TLSSocket.emit (node:events:402:35)
at node:net:687:12
at TCP.done (node:_tls_wrap:580:7)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

I need the session because that is where I hold the Oauth2 token.

This is a dup of request/request#2485 since I found this error while using express-request-proxy. Maybe someone here has better insight?

I'm having an issue with the transform. Even when just trying to push back the data unedited, I have issues.

https://github.com/busheezy/express-request-proxy-transform-issue

My "buffer" is just a random blob of binary, it seems. Maybe I'm only getting one piece or maybe I'm doing something completely wrong.

Thanks.

Hi,

I am getting an error for the following code:

var express = require('express');
var requestProxy = require('express-request-proxy');
var memoryCache = require('memory-cache-stream');
var app = express();

app.get('/data/find', requestProxy({
    cache: memoryCache(),
    cacheMaxAge: 10,
    url: "http://dataservice:5001/getdata/find?query=*"
}));

app.listen(5000, function () {
    console.log('Listening: 5000');
});

And the error is:

Error: Expected "0" to be defined
   at Function.Error.create (/src/node_modules/express-request-proxy/node_modules/simple-errors/index.js:25:11)
   at Function.Error.http (/src/node_modules/express-request-proxy/node_modules/simple-errors/index.js:68:21)
   at proxyViaCache (/src/node_modules/express-request-proxy/lib/proxy.js:119:25)
   at /src/node_modules/express-request-proxy/lib/proxy.js:58:14
   at Layer.handle [as handle_request] (/src/node_modules/express/lib/router/layer.js:95:5)
   at next (/src/node_modules/express/lib/router/route.js:131:13)
   at Route.dispatch (/src/node_modules/express/lib/router/route.js:112:3)
   at Layer.handle [as handle_request] (/src/node_modules/express/lib/router/layer.js:95:5)
   at /src/node_modules/express/lib/router/index.js:277:22
   at Function.process_params (/src/node_modules/express/lib/router/index.js:330:12)

Deps from my package.json

  "dependencies": {
    "winston": "^2.2.0",
    "express-request-proxy": "^2.0.0",
    "express": "^4.13.4",
    "memory-cache-stream": "^1.0.4"
  },

Any idea what I am missing?

It would be great if we could pass in a custom request object for the proxy to use. This would allow us to configure advanced request options or use other libraries that wrap request to provide additional functionality e.g. https://github.com/FGRibreau/node-request-retry

In my case I am proxying and caching a somewhat unreliable API so I would like to be able to retry it a couple of times with an exponential backoff strategy if I get a 5XX response.

As far as I understand right now it is possible to insert query parameters via defining them as params within proxy.

app.use('/api/test', requestProxy({
  url: `${services.forecast.baseUrl}/something/:something`,
  query: {
    apikey: services.forecast.apiKey,
  },
  params: {
    something: 123,
  },
}))

I was looking into replacing this within the axios call.

const fetchSomethingForecast = (something) => call(
    axios,
    {
      method: 'get',
      url: '/api/test',
      params: {
        something: something,
      },
    }
  )

However, axios params go into req.query and hence the url parameter is not replaced.
My suggestion would be to consider query parameters as well while compiling the path in this line. Or do you think that this would be a performance issue?

Proxy works well with text files, but there are some issues with images (only if we enable cache ).
As a cache, I am using Redis 5.0.1
Did you try it with any image file for example jpg?

Hi,

I am proxying to a server that is setting a custom statusMessage in the response (> 400). The statusMessage is not being sent back through the proxy, as requestjs pipeDest function is not automatically setting it. Please see here: request/request#2694

Since it is not being set by requestejs automatically, can we set it before calling pipe?

https://github.com/4front/express-request-proxy/blob/e13778be505940649410f3c4f44bde58ddf69caa/lib/proxy.js#L95

Thank you!

Is this behaviour by design?

For me this fix works:

Just copy the unzip pipe code from the proxyViaCache to makeApiCall function:

      // If the api response is gzip encoded, unzip it before attempting
      // any transforms.
      if (resp.headers['content-encoding'] === 'gzip') {
        apiRequest = apiRequest.pipe(zlib.createGunzip());
      }
      
      if (_.isArray(options.transforms)) {
        apiRequest = applyTransforms(apiRequest, options.transforms, resp.headers);
      }

How would you inject a sensitive parameter in POST requests, not via means of query string, but as a POST request body instead? Is it even possible with the proxy works?

Hello @dvonlehman (I hope there is still someone who takes care of this repo),

I have received a message from Github saying that there is a security issue with Hoek 4.2.0, and when I saw which dependecy depends on hoek on my project, this one was express-request-proxy. I think it needs an update to work with a newer version of Hoek.

Here more about this issue

The npm 'request' module supports passing in a callback function that is called after the request is completed with the result or the error. There is no way to pass this callback/handler currently. This is required because making a call to an non-existent url using express-request-proxy throws an error that cannot be caught by the caller program. I have made a PR to add this: #42

When using queries like /myroute?values[]=1 through the middleware, the backend server receive as query values=1 instead of values[]=1

Example with backend :

app.get('/*', (req, res) => {
  res.json({
    url: req.url
  });
});

Request in direct :

curl -g "localhost:8080/sample?values[]=1"
# {"url": "/az?toto[]=1"}

Request through middleware

curl -g "localhost:8081/sample?values[]=1"
# {"url": "/az?toto=1"}

The backend behind my proxy requires a x-forwarded-host and x-forwarded-port header. This pull request adds the relative information from the host header.

Pull request created #11

I think there might be an issue with other express middlewares that consume the request body like body-parser.

I believe that after body-parser consumes the request body it's no longer possible to pipe the request body to the new proxy request like it's done in proxy.js because it will no longer emit an end event.

I think this might also be the underlying issue for #6 where request timeouts are observed on POST requests.