Comments (9)
To follow up, I decided to just fork and give a go at merging my stuff into this operator. I managed to get it working and if you'd be open to a PR I can definitely get one started. The fork is here: https://github.com/tinyzimmer/kafka-operator
A few main things to note:
- I added support for the user to specify
KafkaCluster.Spec.ListenersConfig.SSLSecrets.Create
. This will create a full PKI for the cluster and clients using cert-manager. - A cert-manager
ClusterIssuer
is made from the user-provided secret if the above isfalse
. I understand this would make cert-manager a core dependency and I was possibly going to look into giving the controller the logic to generate and sign certificates instead.
In regards to the additions, here are a few sample CRs for the two new resources I add:
apiVersion: banzaicloud.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
name: test-topic
namespace: kafka
spec:
clusterRef:
name: kafka
name: test-topic
partitions: 6
replicationFactor: 2
config:
"retention.ms": "800000"
"cleanup.policy": "delete"
# ... etc, etc
---
apiVersion: banzaicloud.banzaicloud.io/v1alpha1
kind: KafkaUser
metadata:
name: test-kafkauser
namespace: kafka
spec:
clusterRef:
name: kafka
name: test-kafkauser
secretName: test-kafkauser-secret
includeJKS: true
topicGrants:
- topicName: test-topic
accessType: read
- topicName: test-topic
accessType: write
If you would be interested in a PR, I'd be happy to make any fixes and changes y'all recommend. I do know I'd want to clean up a few places first. For example, the cruise control resource having it's own topic logic is no longer necessary, and there is some code left over from when I originally ran it as a sidecar to a cluster.
from koperator.
Strimzi
is doing this well... not sure if something more complicated is needed ?
Been using Kafka for years and never had to use more tweaking that what is presented in https://strimzi.io/docs/latest/#type-KafkaTopic-reference
At least it would be a starting point
from koperator.
Anyone working on this ?
from koperator.
Hi @prune998, thanks for asking. Currently we have higher priority features please check the roadmap for more details.
from koperator.
@baluchicken FWIW I am working on a kafka on Kubernetes implementation right now just using helm's incubator/kafka
hacked up to properly support SSL (and disable PLAINTEXT). I wanted topic/user mgt via CRDs and wrote a sidecar operator that can handle them using operator-sdk
and talking to the brokers via sarama.ClusterAdmin
.
It could probably be easily plugged into this project either also as a sidecar to the clusters, or since it's already go scaffolding for the most part, it could also be lifted into the main operator I think. It currently gets its broker URL and TLS config from environment variables and a mounted secret, but if it were to be brought into the main operator it would probably make more sense to get auth information per event directly from the cluster CRDs and secrets (using a clusterRef
from the KafkaUser/KafkaTopic CR specs or something).
The operator assumes a cert-manager
issuer backing the SSL for users but it could be refactored to point at some other TLS provider. I have another tiny operator that can inject a cert-manager
certificate-secret with JKS format (I use it for the brokers), but I can't decide if that's really needed or where it's final home would be. I haven't paid any attention to SASL as an option.
Maybe we can help each other out and then I could move to using a cluster that is a lot more impressive than my current setup, but still have my CRDs for CI/CD 😛 . I can put the controller on my Github, but I'd probably want to rename the apigroups and stuff first. If you'd like me to submit a PR I'd be open to any design changes you all recommend.
from koperator.
You should make a PR right away (when you are ready, or create it as WIP). I know this subject is not the top priority right now, but I'm sure there will be someone to review your code (and I will do too).
At least it will give an option for others (like me) until it is reworked or merged into master.
Thanks for your work !
from koperator.
I'll clean it up a bit more then open a PR as a WIP. I know at the very least I'll probably need to create some user documentation and stuff to go along with it.
from koperator.
Thanks for deciding to fork and contribute this.
from koperator.
Thanks for doing this @tinyzimmer
from koperator.
Related Issues (20)
- enable pprof on the metrics endpoint HOT 3
- Replace github.com/golang/mock[/mockgen] with github.com/uber-go/mock
- Cruise Control metrics reporter failing on k8s cluster with cgroup v2 HOT 3
- Fill the "Events" field with useful content
- Add file integrity checks to CI pipeline
- Validating webhook times out HOT 8
- Kraft support HOT 1
- Multiarch kafka-operator docker image
- istio merged metrics cannot be used HOT 5
- Adding a new SSL listener with hostnameOverride does not regenerate certificates HOT 1
- kafka docker image is not support arm HOT 4
- How to use grafana dashboard with koperator HOT 2
- Kafka coredump created for topic list command on Kafka broker HOT 1
- configuration for securityContext in the CRD for all pods created by operator HOT 1
- The default zookeeper service name created according to the Readme wrong
- Cruise Control is not ready (yet) HOT 2
- Issues with Kafka User creation
- failed to decode certificate
- Be able to set annotation on PVC spec
- Implement access control to Kafka consumer groups
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from koperator.