alexellis / mongodb-function Goto Github PK

View Code? Open in Web Editor NEW

52.0 4.0 8.0 56 KB

OpenFaaS Function that makes use of a connection pool to access MongoDB

Home Page: https://www.openfaas.com

License: MIT License

JavaScript 100.00%

openfaas functions faas serverless databases

mongodb-function's Introduction

OpenFaaS with MongoDB

This is a simple example of how to use connection pooling in MongoDB with OpenFaaS on Kubernetes.

In the first sequence we've had no calls made to the function, so the connection pool is not yet initialized. prepareDB() was never called.
In the second sequence prepareDB() has been called and since there was no instance of a connection in memory, we create one and you see the dotted line shows the connection being established. This will then open a connection to MongoDB in the network.
In the third sequence we see that subsequent calls detect a connection exists and go straight to the connection pool. We have one active connection and two shown with dotted lines which are about to be closed due to inactivity.

Pre-requisites

Before we can build and deploy the example we'll set up OpenFaaS on Kubernetes or Swarm followed by MongoDB. This configuration is suitable for development and testing.

Start by cloning the Github repository:

$ git clone https://github.com/alexellis/mongodb-function

Kubernetes

Install OpenFaaS with helm

https://docs.openfaas.com/deployment/kubernetes/

Install the OpenFaaS CLI

curl -sL https://cli.openfaas.com | sudo sh

Set your OPENFAAS_URL variable

$ export OPENFAAS_URL=127.0.0.1:31112

If you're using minikube or a remote machine then replace 127.0.0.1 with that IP address.

Install mongodb via helm

$ helm install stable/mongodb --name openfaas-db \
  --namespace openfaas-fn \
  --set persistence.enabled=false

Note down the name of the MongoDB instance i.e. openfaas-db-mongodb

If you want to use the fully-qualified DNS name that would be: openfaas-db-mongodb.openfaas-fn.svc.cluster.local.

Now skip ahead to "Build and test"

Docker Swarm

Install OpenFaaS with Docker

https://docs.openfaas.com/deployment/docker-swarm/

Install the OpenFaaS CLI

curl -sL https://cli.openfaas.com | sudo sh

Set your OPENFAAS_URL variable

$ export OPENFAAS_URL=127.0.0.1:8080

Create a mongodb Docker Service

$ docker service create --network=func_functions --name openfaas-db-mongodb --publish 27017:27017 mongo mongod

The entry for the stack.yml file will be the IP of your Docker Swarm manager.

Build and test

Update your stack.yml's mongo field with the MongoDB DNS entry/IP from prior steps
Replace "alexellis/" prefix from Docker Hub in stack.yml with your own account
Build/push/deploy

Pull in the node8-express template:

$ faas template pull https://github.com/openfaas-incubator/node8-express-template

Now build and push / deploy

$ faas build && faas push && faas deploy

Get a load-testing tool

This requires a local installation of Go.

$ go get -u github.com/rakyll/hey

An alternative tool would be Apache-Bench which is available for most Linux distributions via a package manager.

Run a test

Let's start by running a single request with curl:

$ curl http://$OPENFAAS_URL/function/insert-user \
  --data-binary '{"first":"Alex", "last": "Ellis"}' \
  -H "Content-Type: application/json"

Now run a load-test with hey:

$ ~/go/bin/hey -m POST -d '{"first":"Alex", "last": "Ellis"}' \
  -H "Content-Type: application/json" \
  -n 1000 -c 10 http://$OPENFAAS_URL/function/insert-user

This test posts in a JSON body with 1000 requests with 10 of those being concurrent.

Here's an abridged output from hey with the function running on a remote server with the test being run from my laptop:

Summary:
  Requests/sec: 1393.2083
...
Status code distribution:
  [200] 10000 responses

If you look at the logs of the Mongo deployment or service you will see the connection count is between 1-10 connections only during the load-test. This shows that the connection-pool is being used by our function.

On Kubernetes:

$ kubectl logs deploy/openfaas-db-mongodb -f -n openfaas-fn

On Swarm:

$ docker service logs openfaas-db-mongodb -f

mongodb-function's People

Stargazers

Watchers

Forkers

rzebian-padi kiranhk mjtworks leminhtuanh carlosjustino juandisay ebarahona ccolman

mongodb-function's Issues

Cannot access with localhost:27017

Hi there, I meet some problem that access to the mongodb service

openfaas-db-mongodb successful running on the k8s cluter
and my function (based on node8-express-template) cannot connect to mongodb
logs are:

2018/07/17 17:53:14 stderr: (node:7) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): MongoNetworkError: failed to connect to server [127.0.0.1:27017] on first connect [MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017]

mongodb not unauthorized

# export OPENFAAS_URL=127.0.0.1:31112
# curl http://$OPENFAAS_URL/function/insert-user \
>   --data-binary '{"first":"Alex", "last": "Ellis"}' \
>   -H "Content-Type: application/json"
MongoError: there are no users authenticated

Here is my stack.yml.

# cat stack.yml
provider:
  name: faas
  gateway: http://127.0.0.1:31112

functions:
  insert-user:
    lang: node8-express
    handler: ./insert-user
    image: alexellis2/insert-user
    environment:
      write_timeout: 10s
      read_timeout: 10s
      mongo: openfaas-db-mongodb

Steps to Reproduce.

# helm install stable/mongodb --name openfaas-db \
>   --namespace openfaas-fn \
>   --set persistence.enabled=false
NAME:   openfaas-db
LAST DEPLOYED: Wed Jun 13 23:39:58 2018
NAMESPACE: openfaas-fn
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Deployment
NAME                 DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
openfaas-db-mongodb  1        1        1           0          1s

==> v1/Pod(related)
NAME                                  READY  STATUS             RESTARTS  AGE
openfaas-db-mongodb-547fd5b696-mpl6n  0/1    ContainerCreating  0         1s

==> v1/Secret
NAME                 TYPE    DATA  AGE
openfaas-db-mongodb  Opaque  1     1s

==> v1/Service
NAME                 TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)    AGE
openfaas-db-mongodb  ClusterIP  10.102.106.157  <none>       27017/TCP  1s


NOTES:


** Please be patient while the chart is being deployed **

MongoDB can be accessed via port 27017 on the following DNS name from within your cluster:

    openfaas-db-mongodb.openfaas-fn.svc.cluster.local

To get the root password run:

    export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace openfaas-fn openfaas-db-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 --decode)

To connect to your database run the following command:

    kubectl run openfaas-db-mongodb-client --rm --tty -i --image bitnami/mongodb --command -- mongo --host openfaas-db-mongodb -p $MONGODB_ROOT_PASSWORD

To connect to your database from outside the cluster execute the following commands:

    export POD_NAME=$(kubectl get pods --namespace openfaas-fn -l "app=mongodb" -o jsonpath="{.items[0].metadata.name}")
    kubectl port-forward --namespace openfaas-fn $POD_NAME 27017:27017 &
    mongo --host 127.0.0.1 -p $MONGODB_ROOT_PASSWORD

[root@xxx mongodb-function]# export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace openfaas-fn openfaas-db-mongodb -o jsonpath="{.data.mongodb-root-password}" | base64 --decode)
[root@xxx mongodb-function]# kubectl run openfaas-db-mongodb-client-1 --rm --tty -i --image bitnami/mongodb --command -- mongo --host openfaas-db-mongodb.openfaas-fn -p $MONGODB_ROOT_PASSWORD
If you don't see a command prompt, try pressing enter.

> use clients
switched to db clients
> show collections
2018-06-14T03:55:58.061+0000 E QUERY    [thread1] Error: listCollections failed: {
	"ok" : 0,
	"errmsg" : "not authorized on clients to execute command { listCollections: 1.0, filter: {}, $db: \"clients\" }",
	"code" : 13,
	"codeName" : "Unauthorized"
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
DB.prototype._getCollectionInfosCommand@src/mongo/shell/db.js:941:1
DB.prototype.getCollectionInfos@src/mongo/shell/db.js:953:19
DB.prototype.getCollectionNames@src/mongo/shell/db.js:964:16
shellHelper.show@src/mongo/shell/utils.js:842:9
shellHelper@src/mongo/shell/utils.js:739:15
@(shellhelp2):1:1
> db.users.find()
Error: error: {
	"ok" : 0,
	"errmsg" : "not authorized on clients to execute command { find: \"users\", filter: {}, $db: \"clients\" }",
	"code" : 13,
	"codeName" : "Unauthorized"
}

MongoError: command insert requires authentication

Problem

Following instruction in README but I got MongoError: command insert requires authentication.

Suggesting a Solution

I tried to install mogodb with username/password following stable/mongodb chart documentation.

$ helm install --name openfaas-db \
  --namespace openfaas-fn \
  --set persistence.enabled=false,mongodbRootPassword=secretpassword,mongodbUsername=my-user,mongodbPassword=my-password,mongodbDatabase=clients \
    stable/mongodb

Change two lines of handle.js to:

const mongoURL = `mongodb://${username}:${password}@${url} + ":27017/clients`
....

MongoClient.connect(mongoURL,{useNewUrlParser: true, useUnifiedTopology: true} ,...

Change stack.yml:

      url: openfaas-db-mongodb.openfaas-fn.svc.cluster.local
      username: my-user
      password: my-password

port to python?

We tried porting this to python (below), but it seems that the module is reloaded every time, as the cache is always None. Is this functionality available only in node? Do you know of a workaround for python?

Thanks!

import os
import pymongo
import json

mongo_client = None

def handle(req):
    """
    create a record,
    get the database, get the users collection from the database,
    and insert the record into the collection.
    return a result, setting status appropriately
    """

    record = json.loads(req)
    db = prepareDB()
    users = db.users
    id = users.insert_one(record).inserted_id
    return {
      'status': 'Inserted record with id {}: {}'.format(
        id, record)
    }

def prepareDB():
    global mongo_client

    if mongo_client is None:
        url = "mongodb://" + os.environ["mongo"] + ":27017"
        mongo_client = pymongo.MongoClient(url)
    return mongo_client.clients

golang pool of postgresql

is there an example of golang with databses pool of mysql or postgresql ... ?