strongloop / microgateway Goto Github PK

hitting the following error when I'm trying to do a load test on the microgateway.

Error: rate-limit: limit exceeded x-ibm-unnamed-rate-limit

where is the rate limit set?

Hi,

Is there any tutorial on e oauth2.?

The API Connect toolkit provides an embedded microgateway available from npm. It should be updated to use the docker-based microgateway with ngnix and node.js.

When running on the Microgateway, calls to API endpoints fail with the message:
{"name":"PreFlowError","message":"unable to process the request"}

Create a new API with any hardcoded target URL in the Invoke properties.
If you run the Test option from the Designer, it gives the error mentioned above.

The latest version should 1.6.3.

but the package.json in master is still 1.6.1, shouldn't that be updated ?

Hi colleagues!
Sorry for offtopic question, but i have not found any other way to address my question.
I work at consulting company and currently we are building the API management for our client in Russia. We are interested in microgateway solution but apart from IBM API connect package.

However we have some doubts regarding future of microgateway, customization capabilities and also looking for involvement of Strongloop expert to project for consulting.

We've tried to contact IBM here in Russia, but there is no strongloop expertise here :(
I've failed to find any email address or contact point on website and documentation..
Could you please help us with correct contact point so i will be able to address my questions.

Thank you in advance.

The microgateway within API Connect topology is giving some errors intermittently after some time (about 10 hours) online. We put New Relic to monitor an API published on microgateway and it is capturing the following error: Socket is closed. This error repeats about 5 min.
We tried to increase memory for node.js process but the error persist.

When testing the microgateway with api designer, if calling an endpoint that returns json containing multibyte characters, the response is truncated, and the test tool displays:

SyntaxError: Unexpected end of JSON input

Looking in the browser trace we can see that the expected response which is:

{"array":[{"name": "á"}]}

is returned as:

{"array":[{"name":"á"}

missing the two closing square and curly brackets.

APAR: https://jazz609.hursley.ibm.com:9443/jazz/web/projects/APIM#action=com.ibm.team.workitem.viewWorkItem&id=56533

Enhance the REST management API to allow additional flexibility to configure the API gateway without the need for the API Connect toolkit.

When a policy has empty 'properties:' or 'required:' attributes defined, but the 'throw:' property is populated in policy.yaml, the attempt to call flow.fail ends with crash in flow-engine on:

// 1. check the policy's own handler
if (currTask[type].catch) {

as currTask is null.

Adding some property is fixing the problem.

Microgateway should not reject a GET request with payload.

I have this error
Error: api-matcher failed to match to any API
I did exactly as indicated..i'm on mac.
help please

Clean up warnings during npm install

The branch of https://github.com/strongloop/microgateway/tree/feature/refactor-datastore-2 is used for prototyping and benchmark. The benchmark data is described in https://github.ibm.com/apimesh/scrum-micro-gw/issues/247#issuecomment-1274499

Then I created 2 repos to do the real refactoring

• https://github.ibm.com/garytu/refactor-microgateway: the fundamental part - only need swaggers to run with.
• https://github.ibm.com/garytu/refactor-apic-microgateway, support APIC related functions, e.g. rate limiting, subscriptions....

The works are not completed and wrapped up in https://github.ibm.com/garytu/refactor-apic-microgateway/blob/master/TODO.md

Presently, the µgw uses 2 middlewares to pre-populate APIC context variables:

• The context middleware: for "general" context vars (system, request, and message metadata)
• The preflow middleware: handles apiconnect-specific context metadata

A large percentage of these values are not used by policy so copying them from the middleware req object to a µgw context is largely a waste of memory and CPU time. I propose a "lazy load" strategy, meaning that the values of context variables are values are pulled from the middleware req at the time the value is requested.

This may help with portability to the DP/Edge gateway because there is no need to do a wholesale convert from Edge gateway metadata to Micro gateway metadata and vice-versa. The logic to do this would also benefit the cases where a policy needs to set metadata.

Hi Team,
This looks really cool project, however do we have a good WIKI for this. I am trying to find how i can get started up and do other things, which seems to be missing from main description. Please help.

Hi Strongloop Team,

I really like how API Connect's API Designer play well with microgateway.
I have followed this tutorial and the readme.
The tutorial suggested to put the user defined policies inside a "policies" folder and configure the ".apiconnect > config" to refer to this folder.

My question is...
Is there a way I can config where the microgateway (not API Connect/API Designer) to grab the user defined policies?

The reason I am asking is.. I try to something as below:
(1) Follow the "readme" tutorial, in which those built-in policies are clone to my local environment
(2) Go to the project's root and "apic edit"
(3) Configure the ".apiconnect > config" to let API Connect know where my custom policies are stored
(3) Use the API Designer's "Assemble" and pull the "JavaScript" to the panel
(4) [Problem arises here!] API Designer said "This policy is not available on the selected gateway"

I try to put my custom policies in another folder "self-defined-policies" and configure the ".apiconnect > config" to read "self-defined-policies".
After that, API Connect can read the policies properly (without saying "This policy is not available on the selected gateway")
However when I run "sample.js" in the "readme" tutorial, it fails to reference my custom policies..

Thanks for your help

We have an API that uses the Invoke policy to make a call out to a Bluemix Cloudant database. Recently that API was updated to return gzip encoded data, this caused our code in the microgateway to fail. After the Invoke policy we have a Javascript policy and our reference to message.body.id now fails.

To workaround this we now use a Set-Variable policy to set request.headers.accept-encoding to gzip;q=0 and now our Javascript policy code works again.

Should the Invoke policy support data returned compressed?
What is the IBM best practice for this? I am sure it is not to disable compression like we did.

Thanks in advance

I not understand clearly.
After installation, we can manupile Api graphically as indicated.
https://strongloop.com/strongblog/introducing-api-microgateway-programmable-open-source-gateway-apis/.
Because I did the installation and I get
{"text":"Hello World"}

Or there is a difference between IBM api connect and Microgateway?.

Thank you

This pollutes the startup logs

In following the instructions, I got an exception because the line

var mg = require('../lib/microgw');

in sample.js

is being executed from the $HOME/microgateway directory per the instructions and the ".." is effectively looking for that library as

$HOME/lib/microgw

instead of its actual location at

$HOME/microgateway/lib.microgw

I edited the sample.js file to change this line to:

var mg = require('./lib/microgw');

(i.e. changed ".." to "." in the require call)

and the "hello world" worked. You need to fix this typo or folks not familiar with what's going on will get frustrated.

Hi, I have downloaded the open source version of Microgateway and created a Swagger yaml with SecurityDefinitions

securityDefinitions:
  clientIdHeader:
    type: apiKey
    in: header
    name: X-IBM-Client-Id

I was able to call the API with following curl command
curl http://localhost:3000/trial/echo -H "X-IBM-Client-Id:default"

I wanted to find out how can I register more specific client ids with Microgateway?

The getting started tutorial provides instructions on how to configure API definitions but the Dockerfile does not include the any references to the same instructions.

I have made changes to reflect the tutorial but wanted to get consensus before committing any changes.

.
.
COPY package.json microgateway.js ./
COPY definitions definitions/

Dockerfile

FROM node:6-alpine

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY package.json microgateway.js ./
COPY definitions definitions/
COPY lib lib/
COPY utils utils/
COPY config config/
COPY policies policies/

ARG NPM_REGISTRY
ENV npm_config_registry ${NPM_REGISTRY:-https://registry.npmjs.com}
RUN npm install --prod --quiet --depth 0

ENV NODE_ENV production

CMD [ "node", "microgateway.js" ]

Is there a way to use microgateway without creating apis with loopback. We have a use case where we have multiple microservices in different languages. Can i consider microgateway for such scenario?

We are trying to use the Microgateway standalone for our project.

When I clone the github repo and do the Getting Started the code works fine.
When I install the Npm-Package and try to adapt the getting started code I am running into the following error

Error: not able to perform the handshake with APIM, error: Error: can not load default private key
    at Object.exports.handshakeWithAPIm (/Users/tones/Documents/Projekte/JB/sbc/microgateway-playground/node_modules/microgateway/utils/utils.js:192:14)
    at performHandshake (/Users/tones/Documents/Projekte/JB/sbc/microgateway-playground/node_modules/microgateway/lib/analytics.js:251:9)
    at sendAnalytics (/Users/tones/Documents/Projekte/JB/sbc/microgateway-playground/node_modules/microgateway/lib/analytics.js:49:3)
    at Object.<anonymous> (/Users/tones/Documents/Projekte/JB/sbc/microgateway-playground/node_modules/microgateway/lib/microgw.js:37:9)

Here is the js I am using to launch the app

'use strict';

process.env.CONFIG_DIR = __dirname + '/definitions/catapi';
process.env.NODE_ENV = 'production';
process.env.PORT = 3333;

const microgateway = require('./node_modules/microgateway/lib/microgw');

microgateway.start(process.env.PORT);

If your team is having issues checking code style, try out this automated code style review tool here: stickler-ci.com. Connect your repository and create a new pull request or update an existing one to see it in action!

Jenkins is reporting a test case failure in the flow engine (https://cis-jenkins.swg-devops.com/job/ds/job/microgateway~develop/202/console):

Error: 'hello' deepEqual 'hello world'

    1) test file monitor after change
AssertionError: 'hello' deepEqual 'hello world'
    at Test.<anonymous> (test/yaml.laptop.test.js:121:16)
    at Test.assert (node_modules/supertest/lib/test.js:156:6)
    at assert (node_modules/supertest/lib/test.js:127:12)
    at node_modules/supertest/lib/test.js:124:5
    at Test.Request.callback (node_modules/supertest/node_modules/superagent/lib/node/index.js:691:12)
    at IncomingMessage.<anonymous> (node_modules/supertest/node_modules/superagent/lib/node/index.js:922:12)
    at endReadableNT (_stream_readable.js:921:12)
    at node_modules/loopback/node_modules/loopback-context/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:188:31
npm ERR! Test failed.  See above for more details.

The origin of this failure are the new tests associated with the config file change detection. It appears that one of the checked in files is being modified during the test, which shouldn't be done since it can lead to nondeterministic results. After running npm test for microgateway, I see the following change:

diff --git a/test/definitions/yaml/yaml_1.0.0.yaml b/test/definitions/yaml/yaml_1.0.0.yaml
index 4bd0913..6fd6e9c 100644
--- a/test/definitions/yaml/yaml_1.0.0.yaml
+++ b/test/definitions/yaml/yaml_1.0.0.yaml
@@ -39,7 +39,7 @@ x-ibm-configuration:
                     title: set-variable
                     actions:
                       - set: message.body
-                        value: hello
+                        value: hello world
                 - set-variable:
                     title: set-content-type
                     actions:

node 4.x is out of service
node 6.x will be end of service next year
Please assess if microgateway will work with node 8.x and node 10.x?
Thanks

current engine in package.json

 "engines": {
    "node": ">=4.0.0"
}

should change to node >=6.0.0

events.js:72
throw er; // Unhandled 'error' event
^
Error: ENOENT, open 'binaries/appmetrics/tgz/0.10/appmetrics-2.0.1-linux-x64.tgz'
/root/.nvm/v0.10.40/lib
└── (empty)

npm ERR! Linux 3.10.0-514.2.2.el7.x86_64
npm ERR! argv "node" "/root/.nvm/v0.10.40/bin/npm" "install" "-g" "apiconnect"
npm ERR! node v0.10.40
npm ERR! npm v4.3.0
npm ERR! code ELIFECYCLE
npm ERR! errno 8

npm ERR! [email protected] install: node extract_all_binaries.js
npm ERR! Exit status 8
npm ERR!
npm ERR! Failed at the [email protected] install script 'node extract_all_binaries.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the appmetrics package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node extract_all_binaries.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs appmetrics
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls appmetrics
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR! /root/.npm/_logs/2017-08-30T05_25_56_452Z-debug.log

Hello,

Actualy, we use CAS specification for legacy app.

Do you think that the integration of this component with microgateway could have an interest for you and be done quickly?

Breaking up the preflow middleware so that the swagger security code is in its own middleware, which presumably would run between the existing preflow and the flow engine.

There is npm err options@latest required by [email protected] Missing from the build
but sse is not a direct dependencies of microgateway in package.json
It is an indirect dependencies - see below but it exists in npm-shrinkwrap.json

[email protected] /Users/smartmouse/staging-5081-iFix/apiconnect/node_modules/microgateway
└─┬ [email protected]
└─┬ [email protected]
└── [email protected]

dc60server:microgateway smartmouse$
dc60server:microgateway smartmouse$ npm ls [email protected]
[email protected] /Users/smartmouse/staging-5081-iFix/apiconnect/node_modules/microgateway
└─┬ [email protected]
└─┬ [email protected]
└─┬ [email protected]
└── [email protected]

Is there a way of don't get down the existing APIs while updating the microgateway configuration by API Connect Manager?

Luiz Yanai

as title

The current API microgateway combines ngnix and node.js using a docker compose file, which results in two separate container instances. We should use a single docker image that runs both processes within a single container. This can be done using supervisord, and still leverage Dockerfile for building each Docker image (node.js & ngnix).

Hi Strongloop team, I just publish an issue strongloop-archive/strong-gateway#52 on the old gateway about licensing.

I link it here because I don't know if you still watch strong-gateway repo

I appreciate the open source, and look forward to exercising this project/platform

I don't have a lot of time right now to dig into the code, so thought I'd post the issue encountered here, hoping you can point me in the right direction

1. followed installation instructions and test below, including definition of yaml file in definitions/sample
2. node sample.js appears to start-up correctly, issuing 2 warnings on deprecated context middleware
3. curl http://localhost:3000/sample/echo -- and receive message {"name": "PreFlowError". "message": "unable to process the request"}
4. message in cli displays "Error: api-loader failed to load any API"
5. not sure if it is important, but i did note in the .datastore file that the following record is stored
   {"snapshot":"C:\Users\Patrick\Desktop\microgateway\test/definitions/sample","port":36042}

we have an instance of Microgateway correctly connected with a Manager
following the instructions of IBM, but when we make petitions from Api
Manager, these not are arrived to microgateway.

WE have a microgateway instance with env.yaml and id_rsa.pub.
We have a Manager with a catalog pointed to microgateway with env and
id_rsa.
We have not collectives.

We know the manager and microgateway are correctly connected because we
can see that in logs.

NOw, whe we make an simple API from Developer Toolkit or Manager, for
example, we have not response (404) and in Microgateway logs have not
anything, so our petitions are not correctly connected.

Please, could you help us to connect correctly Manager->Microgateway??

Automate the pulling of external policies into the micro-gateway project along with ensuring that the micro-gateway is correctly configured to consume and run the policy.

When starting the app from the AMP in designer, the gateway never starts.

The gateway logs show:

[Wed Oct 4 13:20:45 2017] com.ibm.diagnostics.healthcenter.loader INFO: Node Application Metrics 3.1.0.201710011954 (Agent Core 3.2.2)
[Wed Oct 4 13:20:45 2017] com.ibm.diagnostics.healthcenter.mqtt INFO: Connecting to broker localhost:1883
2017-10-04T17:20:45.585Z pid:81222 worker:0 INFO supervisor starting (pid 81222)
2017-10-04T17:20:45.593Z pid:81222 worker:0 INFO supervisor reporting metrics to `internal:`
2017-10-04T17:20:45.620Z pid:81222 worker:0 INFO supervisor size set to undefined
2017-10-04T17:20:45.998Z pid:81226 worker:1 [Wed Oct 4 13:20:45 2017] com.ibm.diagnostics.healthcenter.loader INFO: Node Application Metrics 3.1.0.201710011954 (Agent Core 3.2.2)
2017-10-04T17:20:46.059Z pid:81226 worker:1 [Wed Oct 4 13:20:46 2017] com.ibm.diagnostics.healthcenter.mqtt INFO: Connecting to broker localhost:1883

But the gateway never ends up starting and doesn't return an error.

This is using the latest from master and using a basic notes template loopback app 2.x

uGW will terminate if schemes on 1/x apis is http.. and the rest are https.

Background
I've built an API with local user accounts (custom user model), and with custom accessToken model (custom attributes & scopes.)
For Ex. /login issues a limited access token with time limits, but when user "registered" in the app, the remote method behind /register auto-generates 4 eternal tokens with more broader scopes and custom attributes, like livemode true/false, type public/private. The public accessToken has very limited scope and is only allow to one action. The livemode attribute in the accessToken is later used to classify events. This property is inherit by the each model resulting from the use of the token, meaning that if you use a token with livemode = false, all those transactions are consider "test mode" transactions.

I also implemented custom end-points to rolled the auto-gen eternal token. At no point in the life of a user, the user is allow to generate more eternal tokens, they can generate timed tokens (/login) or rolled the existing eternal tokens created at the registration point.

Questions
I'm trying to use the microgateway to control authentication and security.

1. How do I configure the microgateway to look at my user model and accessToken model for authentication?
   1.1 Do I need to remove these models from my app and re-implement in the microgateway?
   1.2 If the above is true: What is the equivalent of my the LB user model in the microgateway?
   1.3 Else: how do I go about integrating my existing logic?

I'll will really appreciate if someone can shine some light to these questions.
Thank for the hard-work put into LB and LB microgateway.
D.

Currently the authenticated username for a request ends up in client.org.id for basicAuth or oauth.resource-owner for oAuth.

This makes identity propagation a challenge because there is no reasonable default for a policy like ltpa-generate. We can put $(client.org.id) but oAuth customers have to figure out the right value for their case. Identity propagation is nigh impossible for APIs that use basic and oAuth.

I propose a new context variable like request.whoami, that contains the authenticated name for the user, regardless of authentication scheme.

I had to add a timeout to get one of the yaml laptop tests to pass. I'm not certain if this is a code bug, or a runtime bug.

https://github.com/strongloop/microgateway/pull/23/files#diff-8efe8d3d0c66cd0f903ee93c5e57625fR114

When we load more than 500 rps, the performance falls heavily. Because the datastore-logic (apiloader middleware) starts sending over the network snapshots data between microgateway, datastore and cache. What to do with it, can it be somehow optimized or disabled? Why it sends every time http-request (in getCurrentSnapshot) on every request to the api?

I am getting an error when deploying nginx. The easy fix is to add that to the Dockerfile but curious on whether its something in my environment.

nginx           | + openssl req -x509 -newkey rsa:2048 -days 3650 -nodes -sha256 -keyout /etc/nginx/key.pem -out /etc/nginx/cert.pem -subj /C=NN/ST=NN/L=NN/O=NN/CN=localhost
nginx           | /run.sh: line 8: openssl: command not found
nginx exited with code 127

nginx.tmpl

if [ ! -e "/etc/nginx/cert.pem" ] || [! -e "/etc/nginx/key.pem" ]
then
    openssl req -x509 -newkey rsa:2048 -days 3650 -nodes -sha256 \
     -keyout "/etc/nginx/key.pem" -out "/etc/nginx/cert.pem" \
     -subj "/C=NN/ST=NN/L=NN/O=NN/CN=localhost"
fi

I checked the base nginx image and it seems openssl does not work on the command-line

root@5a4e624bb868:/# openssl
bash: openssl: command not found