Git Product home page Git Product logo

flink-native-k8s-operator's Introduction

Flink native K8s operator in Java

Flink native K8s operator

Flink native Kubernetes Operator is a control plane for running Apache Flink native application on Kubernetes. It is java implemented operator, which use fabric8 Kubernetes Client to contact with K8s apiserver. Also instead of running shell commands flink run/run-application to start a Flink application, we choose to use Flink client interfaces to do this(e.g. ApplicationDeployer). Same for the savepoint and job status. Benefit from this, the operator will have better performance to launch multiple applications.

How to Build

   mvn clean install

How to Run

  • Make Sure that FlinkApplication Custom Resource Definition is already applied onto the cluster. The CRD could be find here. If not, issue the following commands to apply:
kubectl apply -f deploy/crd.yaml
  • Build Docker Image
docker build . -t flink-native-k8s-operator:1.0
docker push
  • Start flink-native-k8s-operator deployment A new ServiceAccount "flink-native-k8s-operator" will be created with enough permission to create/list pods and services.
 kubectl apply -f deploy/flink-native-k8s-operator.yaml
  • Create a new Flink application The flink-native-k8s-operator will watch the CRD resources and submit a new Flink application once the CR it applied.
kubectl apply -f deploy/cr.yaml
  • Delete a Flink application
kubectl delete -f deploy/cr.yaml

OR

kubectl delete flinkapp {app_name}
  • Get/List Flink applications Get all the Flink applications running in the K8s cluster
kubectl get flinkapp

Describe a specific Flink application to show the status(including job status, savepoint, ect.)

kubectl describe flinkapp {app_name}

Trigger a new savepoint

kubectl edit flinkapp {app_name}
# Edit the spec of flinkapp and increase the value of `savepointGeneration`.

How to access JobManager UI

By default, we expose the JobManager rest port with ClusterIP, which means it could only be accessed in the cluster. In order to access the webUI outside of the K8s cluster, the operator will try to create a ingress entry for each application. Then you could use http://{app_name}.flink.k8s.io for the JobManager webUI.

All the Flink app share a same ingress with different rules. This will save a lot public LoadBalancer ip requirements.

You should add {app_name}.flink.k8s.io {ingress_ip} to your local /etc/hosts file.

kubectl get ingress flink-native-k8s-operator could be used to get the ingress ip address.

Future to do

  • Support native K8s session mode.
  • Support ingress for JobManager webUI
  • Support more fields updating, currently only savepointGeneration is supported.

flink-native-k8s-operator's People

Contributors

dependabot[bot] avatar fuyaoli2017 avatar wangyang0918 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.