1ambda / kafka-connect-dashboard Goto Github PK

View Code? Open in Web Editor NEW

38.0 6.0 6.0 662 KB

Dashboard for Apache kafka-connect

JavaScript 71.56% CSS 1.02% HTML 0.32% Scala 26.59% Shell 0.51%

kafka-connect-dashboard's Introduction

Kafkalot

Centralize your kafka-connect instances

supports connect 0.10.0.0+
shipped with dockerized images and the docker-compose file
compatibility with confluent platform 3.0.0
easy connector instance life-cycle management with fancy UI (screenshots)
easy config validation, management using JSON Schema, JSONEditor

Future Plans

support kafka-stream
real-time metrics for connect and stream

Demo

See more screenshots

Usage

Kafkalot consist of 2 sub-projects

kafkalot-storage (REST Server): persist configurations of connects and handling commands (start, validate, etc)
kafkalot-ui (SPA): provides view for managing connectors easily

with Docker

Set these env variables before launching compose

NOTE that a connect cluster should be in the same network otherwise kafkalot can't access

KAFKALOT_STORAGE_CONNECTOR_CLUSTERHOST: kafka connect cluster host
KAFKALOT_STORAGE_CONNECTOR_CLUSTERPORT: kafka connect cluster port

$ wget https://raw.githubusercontent.com/1ambda/kafka-connect-dashboard/master/docker-compose.yml

$ KAFKALOT_STORAGE_CONNECTOR_CLUSTERHOST=$CLUSTER_HOST \
  KAFKALOT_STORAGE_CONNECTOR_CLUSTERPORT=$CLUSTER_PORT \
  docker-compose up

If you do not have a connector cluster yet, use dockerized kafka and ZK

$ wget https://raw.githubusercontent.com/1ambda/kafka-connect-dashboard/master/docker-compose.yml
$ wget https://raw.githubusercontent.com/1ambda/kafka-connect-dashboard/master/with-kafka.yml

# env variables are already configured in `with-kafka.yml`
$ docker-compose -f docker-compose.yml -f with-kafka.yml up

See docker-compose.yml and with-kafka.yml

without Docker (Not Recommended)

See Running kafkalot without Docker

kafka-connect-dashboard's People

Contributors

Stargazers

Watchers

Forkers

datafibers joswlv himansh1306 himanshpal ljavargo

kafka-connect-dashboard's Issues

Implement start, stop connector

Embed db and view for simplifying installation process

DB

It would be better to use more simple embedded databases such as

View

Provide deployment and service yml for kubernetes

[ui] Inconsistency between current connector state and ConnectorListItem's spin icon

For example,

[ui] Support dynamic paginator item size

Support docker image

Create docker images (storage, ui, mongo) and compose them
Provide docker-compose.yml and docker-compose-with-kafka-connect.yml

NPE while running file sink connector on 0.10.0.0-cp1

[2016-08-19 20:30:35,463] ERROR Failed to reconfigure connector's tasks, retrying after backoff: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:807)
java.lang.NullPointerException
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:51)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:62)
        at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:79)
        at org.apache.kafka.connect.runtime.SourceConnectorConfig.<init>(SourceConnectorConfig.java:29)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnector(DistributedHerder.java:845)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnectorTasksWithRetry(DistributedHerder.java:799)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$800(DistributedHerder.java:92)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$11$1.call(DistributedHerder.java:812)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$11$1.call(DistributedHerder.java:809)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:232)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:176)
        at java.lang.Thread.run(Thread.java:745)
[2016-08-19 20:30:35,463] ERROR Unexpected error during connector task reconfiguration:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:818)
[2016-08-19 20:30:35,463] ERROR Task reconfiguration for FileConnector1 failed unexpectedly, this connector will not be properly reconfigured unless manually triggered. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:819)

Support Task level commands (restart, trace)

Merge container with storage

We need to fetch container information inside storage to avoid CORS issues
since kafka-connect currently doesn't supoprt CORS. (including pre-fligt)

Project requires licence

This looks cool, but I don't see a license file, so I'm not sure if I can use it or not

Support graceful shutdown for kafkalot-storage

Support the host networked docker-compose file

Support plugin verification while creating connector instances

config dialog
create dialog
before executing connectors

Check confluent platform 3.0.0 compatibility

Provide the connector version API and render it

For example,

{"version":"0.10.0.0","commit":"b8642491e78c5a13"}

Serving JSON error message in Storage

For example,

{
   "error_type": "ConnectorPluginValidationFailed",
   "error_message": "io.github.1ambda.ConnectorSink has no `connector.class` field......"
}

1ambda / kafka-connect-dashboard Goto Github PK

kafka-connect-dashboard's Introduction

Kafkalot

Demo

Usage

with Docker

without Docker (Not Recommended)

kafka-connect-dashboard's People

Contributors

Stargazers

Watchers

Forkers

kafka-connect-dashboard's Issues

DB

View

Recommend Projects

Recommend Topics

Recommend Org