Comments (6)
It would probably be nice to add a new CRD to support Flink-native k8s cluster. I think also implementing the application mode with new CRD will also help lower the complexity.
API improvements, for example:
- Rename current
FlinkCluster
toFlinkSessionCluster
- New CRD for native k8s Flink cluster:
FlinkCluster
- New CRDs for various job types:
FlinkJob,
BeamJob
(spec includesFlinkCluster
orFlinkSessionCluster
template for dedicated use or link existing cluster as clusterName to share it). I think newFlinkJob
could support various modes likeapplication
,per-job
on native k8s Flink cluster.
Is there any progress or plan on this feature @elanv thanks
from flink-on-k8s-operator.
It would be good to discuss ways to improve the API design when considering new features.
It would probably be nice to add a new CRD to support Flink-native k8s cluster. I think also implementing the application mode with new CRD will also help lower the complexity.
API improvements, for example:
- Rename current
FlinkCluster
toFlinkSessionCluster
- New CRD for native k8s Flink cluster:
FlinkCluster
- New CRDs for various job types:
FlinkJob,
BeamJob
(spec includesFlinkCluster
orFlinkSessionCluster
template for dedicated use or link existing cluster as clusterName to share it). I think newFlinkJob
could support various modes likeapplication
,per-job
on native k8s Flink cluster.
There are existing issues that had been created. I hope you review it.
- Issues in gcp repo
- CRD design improvement: GoogleCloudPlatform/flink-on-k8s-operator#49
- Motivation for new CRD: GoogleCloudPlatform/flink-on-k8s-operator#303
from flink-on-k8s-operator.
There is some interference between my PR and current controller. Therefore if new CRD is going to be added soon, it would be efficient to apply my PR there than reworking the PR.
@regadas In that sense, any specific work or plan for this issue? If not, I would like to proceed with this work first. How is it?
The basic idea is to keep the current FlinkCluster
but create new v1beta2 FlinkJob
CRD with the job routines extracted from it. If we need Flink k8s native cluster, we can leave the current FlinkCluster
as it is and add a new CRD, and among them, we can select the Flink cluster CRD where FlinkJob
will run.
from flink-on-k8s-operator.
@regadas for clarification, is this an issue for supporting flink native kubernetes deployment? As per this documentation.
from flink-on-k8s-operator.
Right so the idea is to support https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/overview/#application-mode as alternative way to the current way of submitting jobs: Per-job and session.
Adding Flink Kubernetes native is smth we can also have a look at.
from flink-on-k8s-operator.
It would probably be nice to add a new CRD to support Flink-native k8s cluster. I think also implementing the application mode with new CRD will also help lower the complexity.
API improvements, for example:
- Rename current
FlinkCluster
toFlinkSessionCluster
- New CRD for native k8s Flink cluster:
FlinkCluster
- New CRDs for various job types:
FlinkJob,
BeamJob
(spec includesFlinkCluster
orFlinkSessionCluster
template for dedicated use or link existing cluster as clusterName to share it). I think newFlinkJob
could support various modes likeapplication
,per-job
on native k8s Flink cluster.Is there any progress or plan on this feature @elanv thanks
Is there any one working on this?
I think we can split this issue into multi issues/steps so that others can help with it.
from flink-on-k8s-operator.
Related Issues (20)
- Cluster stuck in Updating state if PodDisruptionBudget is set
- Wrong job status after job update.
- Rework examples
- Allow Flink to ignore savepoint on restore if the states of the old and new jobs are incompatible
- Create a new cluster before deleting the old one on the job update
- FlinkCluster stuck in Updating state when PDB is used.
- Add HorizontalPodAutoScaller properties to FlinkCluster spec HOT 1
- QUESTION: how to get sample app WordCount.jar to run with version 1.15.3 and 2 taskmanager replicas HOT 3
- Caused by GSSException: No valid credentials provided(Mechanism level: Failed to find any Kerberos tgt)
- Validation Error `nodeaffinity` rule for the flinkcluster HOT 2
- HPA not creating new pods on scale event HOT 3
- Pod Affinity Feature Causing Flink Pipeline Redeployment to Fail HOT 2
- poddisruptionbudget is not allowing any disruptions HOT 1
- While using application mode, the jobmanager pod is not restarted when killed
- Flink Operator Loses Job Manager Contact during EKS upgrade HOT 10
- Question: is the latest CRD backwards compatible with the CRD from 0.30 HOT 3
- Job Manager is not brought back up HOT 1
- Application mode Job Manager restart can create multiple FlinkJobs
- Streaming Application mode Jobs can sometimes reach completed stage
- If the job submitter fails, the job keeps running
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-on-k8s-operator.