Git Product home page Git Product logo

chryssalid's Introduction

Chryssalid

Grow within your enemy, then burst out.

I just want to get going

Discouraged, google has since enabled python as a proper function engine, and this project now only serves as insight into the inner workings of google cloud functions.

Prerequisites

  • Docker
  • Service Account json with Google Cloud Functions Developer role
  • Pipenv
  • Project with a flask blueprint and an entry point called cloud.py

cloud.py

import flask
import cloud_functions

flask_app = flask.Flask(__name__)


@flask_app.route('/')
def hello():
    return "Hello cloud functions."


cloud_functions.register_http_trigger(flask_app)

Dockerfile

FROM stormydragon/gcf-python
CMD ["--http", "--project=<my project name>", "--name=<trigger name>"]

Run these commands

pipenv install flask
docker build --tag my_cloud_function .
docker run --rm -it -v /path/to/service-account.json:/service-account.json:ro my_cloud_function

But I want to listen to Firebase or cloud storage or something..

That's good too, simply pick a resource and event type from below and use code similar to this you can find examples of the context and data in the invocation data.md file.

cloud.py

import cloud_functions
import logging
logger = logging.getLogger(__name__)


def handler(context, data):
    logger.info(context, data)


cloud_functions.register_event_trigger(handler)

Dockerfile

FROM stormydragon/gcf-python
CMD ["--event", "--project=<my project name>", "--name=<trigger name>", "--resource=<the resource>", "--event=<the event>"]

Run these commands

pipenv install flask
docker build --tag my_cloud_function .
docker run --rm -it -v /path/to/service-account.json:/service-account.json:ro my_cloud_function

The container will package your project during build and by running it with the specified arguments will deploy the package to Google Cloud Functions

Objective

Replace google cloud function node interpreter with arbitrary code Instead of using node as a shim, simply replace node.js and be master control program of the container.

How does it work?

The node worker script google_cloud_worker/worker.js was built into the container by the cloud function deployment machinery and is untouchable by our code until the /load endpoint is called, this is triggered by a "cold start" before the first connection is made. Our code usurps the node process and executes our own, in the process handing the connected /load and listening file descriptor to it, it is then that we send the answer to load, and accept new connections on the listener.

Worker needs

Google cloud functions consists of a webserver that accepts three 'paths' as well as communication via http with a supervisor service.

Environment

  • X_GOOGLE_CODE_LOCATION
  • X_GOOGLE_ENTRY_POINT
  • X_GOOGLE_FUNCTION_TRIGGER_TYPE - HTTP_TRIGGER, ,
  • X_GOOGLE_FUNCTION_NAME
  • X_GOOGLE_FUNCTION_TIMEOUT_SEC
  • X_GOOGLE_WORKER_PORT - Web server port
  • X_GOOGLE_SUPERVISOR_HOSTNAME
  • X_GOOGLE_SUPERVISOR_INTERNAL_PORT

Variables

  • MAX_LOG_LENGTH = 5000
  • MAX_LOG_BATCH_ENTRIES = 1500
  • MAX_LOG_BATCH_LENGTH = 150000
  • SUPERVISOR_KILL_TIMEOUT_MS = 500
  • SUPERVISOR_LOG_TIMEOUT_MS

Paths

  • /load - Not used after node.
  • /check - Heartbeat, must return 200 OK
  • /execute - POST - For execution of all non-http functions
  • /execute/* - For HTTP will accept arbitrary paths.

Response headers

  • X-Google-Status: one of crash, load_error, error to indicate why the function died.

Supervisor

Accept logs and kill command from worker

  • _ah/kill - Notify of our need to die.
  • _ah/log
  {
    "Entries": [
      {
        "TextPayLoad": "...",
        "Severity": "INFO",
        "Time": "2018-01-01T00:00:00.000Z"
      }
    ]
  }

Google Pub/Sub Events and Types

Events from firebase don't wrap the context in a sepate field.

Firebase Auth

Event Resource Definition

projects/<project name>

Event Types

  • providers/firebase.auth/eventTypes/user.create
  • providers/firebase.auth/eventTypes/user.delete

Firebase Firestore

Event Resource Definition

projects/<project name>/databases/(default)/documents/<collection path>/{collectionId}

Event Types

  • providers/cloud.firestore/eventTypes/document.create
  • providers/cloud.firestore/eventTypes/document.update
  • providers/cloud.firestore/eventTypes/document.delete
  • providers/cloud.firestore/eventTypes/document.write

Firebase Realtime Database

Event Resource Definition

projects/_/instances/<instance|project name>/refs/<path>/{key}

Event Types

  • providers/google.firebase.database/eventTypes/ref.create
  • providers/google.firebase.database/eventTypes/ref.write
  • providers/google.firebase.database/eventTypes/ref.update
  • providers/google.firebase.database/eventTypes/ref.delete

Pub Sub Topics

Event Resource Definition

projects/<project name>/topics/<topic name>

Event Types

  • google.pubsub.topic.publish

Cloud Storage

Event Resource Definition

projects/<project name>/buckets/<bucket name>

Event Types

  • google.storage.object.finalize
  • google.storage.object.metadata_update
  • google.storage.object.delete
  • google.storage.object.archive

chryssalid's People

Contributors

stormydragon avatar

Stargazers

Nuno Cruces avatar ethan avatar Josh avatar

Watchers

 avatar

chryssalid's Issues

Socket passing

When transitioning from NodeJS to Python, all the open sockets are passed.
Usually three sockets are passed when Python takes over.

  • The first socket is the server listening socket.
  • The second socket I thought was a call to /check but that endpoint is never called.
  • The third socket I think is a call to the current /load function in progress when taking over Node

The current behavior is to treat the other sockets as if they are load and simply respond affirmatively to them before closing them. This is not optimal though.

First function invocation timeout

This does not seem to be limited to this version of functions, but often the first run of a newly uploaded function will time out only to work on subsequent invocations, this might be mitigated by increasing the timeout. For background events this usually does not matter as the function will be called again until success, but for HTTP it will result in an error 500.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.