jbvmio / kafkactl Goto Github PK

View Code? Open in Web Editor NEW

88.0 7.0 9.0 27.14 MB

CLI for Apache Kafka Management -

Home Page: http://www.jbvm.io/kafkactl

License: Apache License 2.0

Go 98.97% Makefile 0.92% Dockerfile 0.12%

kafka zookeeper burrow topic kafka-cluster consumer-group lag offsets partition replicas

kafkactl's Introduction

kafkactl - Kafka Management Tool

kafkactl - CLI for Apache Kafka, Zookeeper and Burrow Management.

Current Features:

ToDo:

Add Security Features
Add Metrics Testing
Test Avro Serialization

kafkactl is actively developed with new features being added and tested. Thus, ongoing optimization and re-factoring will occur so ensure you are aware of the latest releases.

kafkactl tool - Get Started

From Source

Requires Go Version 1.11+

git clone https://github.com/jbvmio/kafkactl
cd kafkactl/
go build -o $GOPATH/bin/kafkactl

# If inside your go path:
GO111MODULE=on go build -o $GOPATH/bin/kafkactl

Homebrew

# First time install:
brew tap jbvmio/tap
brew install jbvmio/tap/kafkactl

# Upgrading
brew upgrade jbvmio/tap/kafkactl

Docker

docker pull docker.io/jbvmio/kafkactl:latest
docker run --rm jbvmio/kafkactl -B broker:9092 get topics --lag
docker run --rm jbvmio/kafkactl -Z zookeeper:2181 ls /brokers/ids/2
docker run --rm jbvmio/kafkactl config --sample

Manual Download and Install (Linux, Windows, Mac)

Download the latest kafkactl tool appropriate for your platform, extract and install to a valid directory, preferably located somewhere in your configured $PATH.
Run "kafkactl config --sample" to generate and display a sample config. Create your config and place as $HOME/.kafkactl.yaml

Alternatively, pass all arguments using flags instead of using a config file when running kafkactl:

# kafkactl --broker brokerhost01:9092 admin create topic newtopic01 --partitions 15 --rfactor 3

kafkactl - Example Commands

example config commands

kafkactl config --sample
kafkactl config view
kafkactl config get-contexts
kafkactl config current-context
kafkactl config use cluster1

example config file (~/.kafkactl.yaml)

contexts:
  prod-atl01:
    name: prod-atl01
    brokers:
    - broker01:9092
    - broker02:9092
    - broker03:9092
    burrow:
    - burrow01:8080
    - burrow02:8080
    - burrow03:8080
    zookeeper:
    - zkhost01:2181
    - zkhost02:2181
    - zkhost03:2181
    clientVersion: ""
  prod-atl02:
    name: prod-atl02
    brokers:
    - broker01:9092
    - broker02:9092
    - broker03:9092
    burrow:
    - burrow01:8080
    - burrow02:8080
    - burrow03:8080
    zookeeper:
    - zkhost01:2181
    - zkhost02:2181
    - zkhost03:2181
    clientVersion: ""
  prod-atl03:
    name: prod-atl03
    brokers:
    - broker01:9092
    - broker02:9092
    - broker03:9092
    burrow:
    - burrow01:8080
    - burrow02:8080
    - burrow03:8080
    zookeeper:
    - zkhost01:2181
    - zkhost02:2181
    - zkhost03:2181
    clientVersion: ""
current-context: prod-atl01
config-version: 1

Usage

kafkactl -h

kafkactl: Kafka Management Tool

Usage:
  kafkactl [flags]
  kafkactl [command]

Examples:
  kafkactl --context <contextName> get brokers

Available Commands:
  admin       Kafka Admin Actions
  burrow      Show Burrow Lag Evaluations <wip>
  config      Show and Edit kafkactl config
  describe    Get Kafka Details
  get         Get Kafka Information
  help        Help about any command
  logs        Get Messages from a Kafka Topic
  send        Send/Produce Messages to a Kafka Topic
  version     Print kafkactl version and exit
  zk          Zookeeper Actions

Flags:
  -B, --broker string      Specify a single broker target host:port - Overrides config.
      --burrow string      Specify a single burrow endpoint http://host:port - Overrides config.
      --cfg string         config file (default is $HOME/.kafkactl.yaml)
  -C, --context string     Specify a context.
  -x, --exact              Find exact matches.
  -h, --help               help for kafkactl
  -o, --out string         Change Output Format - yaml|json.
  -v, --verbose            Display additional info or errors.
      --version string     Specify a client version.
  -Z, --zookeeper string   Specify a single zookeeper target host:port - Overrides config.

kafkactl admin -h

Available Commands:
  create      Create Kafka Resources
  delete      Delete Kafka Resources
  get         Get Kafka Configurations
  move        Move Partitions using Stdin
  set         Set Kafka Configurations

kafkactl zk -h

Zookeeper Actions

Usage:
  kafkactl zk [flags]
  kafkactl zk [command]

Available Commands:
  create      Create Zookeeper Paths and Values
  delete      Delete Zookeeper Paths and Values
  ls          Print Zookeeper Paths and Values

Flags:
  -h, --help         help for zk
  -o, --out string   Change Output Format - yaml|json.

Global Flags:
  -B, --broker string      Specify a single broker target host:port - Overrides config.
      --burrow string      Specify a single burrow endpoint http://host:port - Overrides config.
      --cfg string         config file (default is $HOME/.kafkactl.yaml)
  -C, --context string     Specify a context.
  -x, --exact              Find exact matches.
  -v, --verbose            Display additional info or errors.
      --version string     Specify a client version.
  -Z, --zookeeper string   Specify a single zookeeper target host:port - Overrides config.

Example output:

kafka

# kafkactl

Brokers:  5
 Topics:  208
 Groups:  126

Cluster: (Kafka: v1.1)
  brokerhost01:9092/1
* brokerhost02:9092/2
  brokerhost03:9092/3
  brokerhost04:9092/4
  brokerhost05:9092/5

(*)Controller

# kafkactl get group mygroup05

GROUPTYPE  GROUP      COORDINATOR
consumer   mygroup05  broker03:9092/3

kafkactl get group mygroup05 --lag

GROUP      TOPIC      PART  MEMBER                    OFFSET  LAG
mygroup05  mytopic05  14    mygroup05-77885078-77abc  97      0
mygroup05  mytopic05  1     mygroup05-77885078-77abc  76      0
mygroup05  mytopic05  2     mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  9     mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  12    mygroup05-77885078-77abc  84      0
mygroup05  mytopic05  3     mygroup05-77885078-77abc  135     0
mygroup05  mytopic05  8     mygroup05-77885078-77abc  90      0
mygroup05  mytopic05  7     mygroup05-77885078-77abc  83      0
mygroup05  mytopic05  4     mygroup05-77885078-77abc  111     0
mygroup05  mytopic05  6     mygroup05-77885078-77abc  114     0
mygroup05  mytopic05  0     mygroup05-77885078-77abc  2329    0
mygroup05  mytopic05  5     mygroup05-77885078-77abc  85      0
mygroup05  mytopic05  11    mygroup05-77885078-77abc  139     0
mygroup05  mytopic05  10    mygroup05-77885078-77abc  125     0
mygroup05  mytopic05  13    mygroup05-77885078-77abc  97      0

zookeeper

# kafkactl zk ls /cluster/id

VALUE: /cluster/id
 {"version":"1","id":"wwI7vYwOShKnEqVzDhhUYqi"}

# kafkactl zk ls /

PATH: /
 cluster
 controller
 brokers
 zookeeper
 admin
 isr_change_notification
 log_dir_event_notification
 controller_epoch
 consumers
 burrow
 latest_producer_id_block
 config

# kafkactl zk --create /testpath
Successfully Created: /testpath

# echo 'my testvalue from stdin' | kafkactl zk --create /testpath/testvalue
Successfully Created: /testpath/testvalue

# kafkactl zk ls /testpath/testvalue

VALUE: /testpath/testvalue
 my testvalue from stdin

burrow

kafkactl burrow mygroup07 -x

BURROW      GROUP      TOPIC      PARTITION  LAG  TOPICLAG  STATUS
dc01-kafka  mygroup07  mytopic07  0          0    0         OK
dc01-kafka  mygroup07  mytopic07  1          0    0         OK
dc01-kafka  mygroup07  mytopic07  2          0    0         OK
dc01-kafka  mygroup07  mytopic07  3          0    0         OK
dc01-kafka  mygroup07  mytopic07  4          0    0         OK
dc01-kafka  mygroup07  mytopic07  5          0    0         OK
dc01-kafka  mygroup07  mytopic07  6          0    0         OK
dc01-kafka  mygroup07  mytopic07  7          0    0         OK
dc01-kafka  mygroup07  mytopic07  8          0    0         OK
dc01-kafka  mygroup07  mytopic07  9          0    0         OK
dc01-kafka  mygroup07  mytopic07  10         0    0         OK
dc01-kafka  mygroup07  mytopic07  11         0    0         OK
dc01-kafka  mygroup07  mytopic07  12         0    0         OK
dc01-kafka  mygroup07  mytopic07  13         0    0         OK
dc01-kafka  mygroup07  mytopic07  14         0    0         OK

kafkactl's People

Contributors

Stargazers

Watchers

Forkers

cliffwheadon dmitryilyin go-stream ateijelo zhanglei percyzhang codingknees mschlech showsmall

kafkactl's Issues

build broken on termui dependency

when attempting to build today on go 1.13, I get the following dependency error:

 go build -o $GOPATH/bin/kafkactl
build github.com/jbvmio/kafkactl: cannot load github.com/gizak/termui: module github.com/gizak/termui@latest (v3.1.0+incompatible) found, but does not contain package github.com/gizak/termui

client/metadata got error from broker -1 while fetching metadata

Hello.

I am newbie and try to use kafkactl:

podman run -it --rm --name kafkactl -v ~/.config/kafkactl/config.yml:/etc/kafkactl/config.yml \
    docker.io/deviceinsight/kafkactl:v1.11.1 \
        kafkactl --verbose get topics
[kafkactl] 2020/09/22 09:02:18 Using config file: /etc/kafkactl/config.yml
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[sarama  ] 2020/09/22 09:02:18 Initializing new client
[sarama  ] 2020/09/22 09:02:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:02:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44214->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:02:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:02:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:02:48 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama  ] 2020/09/22 09:02:48 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:18 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44222->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:18 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:18 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:18 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama  ] 2020/09/22 09:03:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44228->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:48 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama  ] 2020/09/22 09:03:49 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:49 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:04:19 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44244->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:04:19 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:04:19 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:04:19 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:04:19 Closing Client
failed to create cluster admin: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Config is very simple (sed to show just non-comments and non-empty lines):

$ sed 's/#.*$//;/^$/d' .kafkactl.yaml
contexts:
  sandbox:
    name: sandbox
    brokers:
      - ecsc00a060af.epam.com:9092

current-context: sandbox

And according to wireshark dump TCP session kafkactl send metadata request and can connect to the server, but newer receive response:

Where I am wrong?

JSON Pretty Print

It would be great to have additional flag for the kafkactl consume to display pretty printed JSON.

Raw JSON:

{"type":"UserStatusChanged","id":"e886eb90-4003-4b1f-9971-8f4bdb23dc89","timestamp":1588167444,"data":{"user":{"id":331,"status":"approved"},"from_status":"created","to_status":"approved","changed_at":"2020-04-29T13:37:18+00:00"}}

Pretty printed JSON:

{
  "type": "UserStatusChanged",
  "id": "e886eb90-4003-4b1f-9971-8f4bdb23dc89",
  "timestamp": 1588167444,
  "data": {
    "user": {
      "id": 331,
      "status": "approved"
    },
    "from_status": "created",
    "to_status": "approved",
    "changed_at": "2020-04-29T13:37:18+00:00"
  }
}

It might be useful on development phase for debugging the messages.

--json-filter example?

Hi,

I can't figure out how --json-filter is supposed to work to filter the messages as I follow a topic.
Could you, please, show an example?
Thank you!

Homebrew tap throws error

brew install jbvmio/tap/kafkactl

Error: jbvmio/tap/kafkactl: wrong number of arguments (given 1, expected 0)

Support topic configuration when creating topics

Thanks for creating this useful utility!

One feature that seems missing is creating topics with An initial configuration

example usage could possibly look like

kafkactl --broker brokerhost01:9092 \
    admin create topic newtopic01 \
       --partitions 15 \
       --rfactor 3 \
       --config cleanup.policy=compact \
       --config min.compaction.lag.ms=0 \
       --config delete.retention.ms=604800000 \
       --config min.cleanable.dirty.ratio=0.05

Blocked by jbvmio/kafka#1

Support for SSL

I've been using kafkactl setting up a kafka cluster and finding it very useful, thanks a bunch.

Now I want to put this thing in production, and that includes only listening on SSL.
But unless I'm missing something, it seems kafkactl will only connect to brokers over plaintext.

Is SSL support something that's on the roadmap? The todo lists Add Security Features but that could be many things.

Thank you for considering adding SSL support.

Tap no longer working under homebrew

When I try to install under homebrew, I get the following error:

❱ brew tap jbvmio/tap
==> Tapping jbvmio/tap
Cloning into '/opt/homebrew/Library/Taps/jbvmio/homebrew-tap'...
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 179 (delta 19), reused 27 (delta 11), pack-reused 138
Receiving objects: 100% (179/179), 1.12 MiB | 8.75 MiB/s, done.
Resolving deltas: 100% (67/67), done.
Error: Invalid formula: /opt/homebrew/Library/Taps/jbvmio/homebrew-tap/Formula/kafkactl.rb
kafkactl: /opt/homebrew/Library/Taps/jbvmio/homebrew-tap/Formula/kafkactl.rb:28: syntax error, unexpected end-of-input, expecting end
Error: Cannot tap jbvmio/tap: invalid syntax in tap!

Error building kafkactl

I followed the instructions provided on the README file of this repository. I installed golang 1.19, from Kali's APT repository. Not quite sure what the next steps are here.

PWD: /home/user/projects/tmp/kafkactl
GOPATH: /home/user/go

❯ go build -o $GOPATH/bin/kafkactl
cli/kafka/adminConfig.go:24:2: github.com/jbvmio/[email protected]: replacement directory ../kafka does not exist
cli/cmd/root.go:28:2: github.com/jbvmio/[email protected] (replaced by ../kafka): reading ../kafka/go.mod: open /home/user/projects/tmp/kafka/go.mod: no such file or directory

Error when trying to reset consumer group offset

I'm trying to reset the connect consumer group offset and get an error.

Command: kafkactl reset cgo --oldest -t users connect-cluster
Error: kafka server: The provider group protocol type is incompatible with the other members.

Schema Registry Support

A lot of kafka users will probably be on the Confluent Platform and dealing with Avro and the Schema Registry.

Using curl to interact with the Schema Registry is cumbersome and doesn't produce the most readable output.

Schema Registry support would be a neat addition to this tool.