Git Product home page Git Product logo

parkeeper's Introduction

parkeeper Build Status Docker Repository on Quay.io

One service discovery backend to rule them all. The idea behind this was to seamlessly use tools and frameworks that heavily rely on zookeeper, for example, finagle, kafka, and keep/support/maintain just one key-value store. This could facilitate the migration to the nextgen service discovery/distributed configuration services like etcd or consul.

One of these two backends are available:

  • etcd
  • consul

Unsupported ZooKeeper features (ordered by priority):

  • Reliable zxid (X-Consul-Index & X-Etcd-Index)
  • Watches
  • Ephemeral Nodes
  • Sequence Nodes
  • ACLs
  • Auth
  • Process requests in batch (Multi)
  • Reliable Stats (?)

Listing of supported requests with some notes:

etcd consul
CREATE ✅ 1 ✅
DELETE ✅ ✅
EXISTS ✅ ✅
GETDATA ✅ ✅
SETDATA ✅ ✅
GETACL 🚧 🚧
SETACL 🚧 🚧
GETCHILDREN ✅ ✅
SYNC ✅ <sup23 ✅ 2
PING ✅ ✅
GETCHILDREN2 ✅ ✅
CHECK ✅ ✅
MULTI 🚧 🚧
CREATE2 ✅1 ✅
CLOSE ✅ ✅
SETAUTH 🚧 🚧
SETWATCHES 🚧 🚧

1 Unable to create a node with a key/path equal to an existing directory. (etcd will support this in v3 api: #1855)

2 There is no similar etcd/consul request. For now, it does not proceed.

Using parkeeper is as easy as this:

docker run -p 2181:2181 quay.io/glerchundi/parkeeper -backend-url etcd://127.0.0.1:4001
docker run -p 2181:2181 quay.io/glerchundi/parkeeper -backend-url consul://127.0.0.1:8500

This project is in its early stages, use at your own risk. And of course, any feedback is appreciated as well as issues!

parkeeper's People

Contributors

glerchundi avatar

Stargazers

A Husky avatar avatar avatar Gonzalo Peci avatar Michael Aldridge avatar Markus Bachmann avatar Krzysztof Zarzycki avatar Luis Furnas avatar Emmanuel Fuentes avatar Alexey Pimenov avatar Daniel avatar avatar Ted Behling avatar Ted Behling avatar Fabian Stegemann avatar Romain avatar Dale Dude avatar Roman Levytskyi avatar Elliot Weiser avatar Will Weber avatar avatar Travis Crowder avatar Roman Bulgakov avatar Wes Maag avatar Andrew Breksa avatar Sean Kell avatar Vladimir Lazarenko avatar Hyeon Kim (김지현) avatar ted avatar Andrii Melekhovskiy avatar Tom Wilkie avatar John Bauer avatar Achille avatar avatar Igor Lu avatar Hunter Morgan avatar Yannis avatar ORCTOM avatar Noah Huppert avatar Marc-André Tremblay avatar Jan Ulferts avatar Benjamin Rizkowsky avatar Dani Comnea avatar Calvin Lee Fernandes avatar Jens Rantil avatar avatar Matt Singletary avatar avatar Will Dowling avatar Valere JEANTET avatar Craig Wickesser avatar Ralph Tice avatar Miki Oracle avatar avatar Yoann Cerda avatar Joaquim Moreno Prusi avatar netxfly avatar Daniel Menzen avatar avatar avatar Henning avatar Michael Bironneau avatar avatar Thinh Nguyen avatar Rogerio Marques avatar Paul Capestany avatar Ryan Schmukler avatar avatar Daniel Hughes avatar Geofrey Ernest avatar Boris V avatar avatar avatar G avatar Drew Nall avatar Nedumaran Rajagopal avatar Paul Forman avatar JustAdam avatar Kevin Stanton avatar Kyle McCullough avatar Przemysław Węgrzyn avatar Tim Lossen avatar Jeff Gonzalez avatar avatar Miguel Eduardo Gil Biraud avatar Ross Williams avatar Mike Lee avatar Vitaliy Levchenko avatar Petr Svoboda avatar David Reed avatar Christopher Smith avatar s7v7nislands avatar Joshua Dotson avatar Evgeniy Skidan avatar Harshad Sharma avatar Yixin Hao avatar avatar James Young avatar Tony Pujals avatar Georg Kunz avatar

Watchers

James Young avatar Hagen Rother avatar Geert avatar Dave Rawks avatar Hien Vien avatar Sean Kell avatar Holger Pillmann avatar Nick Fisher avatar Shani Elharrar avatar James Cloos avatar Achille avatar Keith Chambers avatar avatar Nils Grabbert avatar avatar

Forkers

inthecloud247 annakan justizin danieldc justadam lilithmooncohen fatz anatolyzimin favoretti spechal

parkeeper's Issues

Is this ready to support kafka?

I'm using a consul backend with parkeeper. When I connect with kafka on another machine it makes parkeeper die. Here is what the log says

2015-02-01T23:33:35Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG listening on: [::]:2181
2015-02-01T23:33:56Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client connected: 10.0.0.75:52390
2015-02-01T23:33:56Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG closed due to: EOF
2015-02-01T23:33:56Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client disconnected: 10.0.0.75:52390
2015-02-01T23:33:57Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client connected: 10.0.0.75:52391
2015-02-01T23:33:57Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG closed due to: EOF
2015-02-01T23:33:57Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client disconnected: 10.0.0.75:52391
2015-02-01T23:33:58Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client connected: 10.0.0.75:52392
2015-02-01T23:33:58Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG closed due to: EOF
2015-02-01T23:33:58Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client disconnected: 10.0.0.75:52392
2015-02-01T23:35:01Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client connected: 10.0.0.75:52396
2015-02-01T23:35:01Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG closed due to: length should be at most 1572864 bytes (received 1937006964)
2015-02-01T23:35:01Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client disconnected: 10.0.0.75:52396
2015-02-01T23:35:03Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client connected: 10.0.0.75:52398
2015-02-01T23:35:03Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG closed due to: length should be at most 1572864 bytes (received 1937006964)
2015-02-01T23:35:03Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client disconnected: 10.0.0.75:52398
2015-02-01T23:35:06Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client connected: 10.0.0.75:52399
2015-02-01T23:35:06Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG closed due to: length should be at most 1572864 bytes (received 1937006964)
2015-02-01T23:35:06Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client disconnected: 10.0.0.75:52399
2015-02-01T23:35:10Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client connected: 10.0.0.75:52400
2015-02-01T23:35:10Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG closed due to: EOF
2015-02-01T23:35:10Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG client disconnected: 10.0.0.75:52400
2015-02-01T23:36:08Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG terminated
2015-02-01T23:36:08Z ip-10-0-2-96 /opt/go/bin/parkeeper[2051]: DEBUG stopping listening on: [::]:2181
2015-02-01T23:36:37Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG listening on: [::]:2181
2015-02-01T23:37:37Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG client connected: 127.0.0.1:51306
2015-02-01T23:37:37Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG closed due to: length should be at most 1572864 bytes (received 1937006964)
2015-02-01T23:37:37Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG client disconnected: 127.0.0.1:51306
2015-02-02T07:07:42Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG client connected: 10.0.3.148:50753
2015-02-02T07:07:42Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG closed due to: EOF
2015-02-02T07:07:42Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG client disconnected: 10.0.3.148:50753
2015-02-03T02:50:22Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG client connected: 10.0.3.175:59486
2015-02-03T02:50:22Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG <- 0000002c0000000000000000000000000000177000000000000000000000001000000000000000000000000000000000
2015-02-03T02:50:22Z ip-10-0-2-96 /opt/go/bin/parkeeper[2028]: DEBUG closed due to: read tcp 10.0.3.175:59486: use of closed network connection
2015-02-03T02:52:34Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG listening on: [::]:2181
2015-02-03T02:52:38Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG client connected: 127.0.0.1:55764
2015-02-03T02:52:38Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG closed due to: EOF
2015-02-03T02:52:38Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG client disconnected: 127.0.0.1:55764
2015-02-03T02:52:50Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG client connected: 10.0.3.175:59501
2015-02-03T02:52:51Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG closed due to: EOF
2015-02-03T02:52:51Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG client disconnected: 10.0.3.175:59501
2015-02-03T02:53:24Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG client connected: 10.0.3.175:59506
2015-02-03T02:53:24Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG <- 0000002c0000000000000000000000000000177000000000000000000000001000000000000000000000000000000000
2015-02-03T02:53:24Z ip-10-0-2-96 /opt/go/bin/parkeeper[25253]: DEBUG closed due to: read tcp 10.0.3.175:59506: use of closed network connection

Easing migration off Zookeeper by adding Zookeeper support

Hi,

This looks like a great idea! For ease in migrating off Zookeeper, have you considered adding Zookeeper as a backend? That would get you immediate feature parity rather than having to wait on additional functionality from the consul/etcd projects, and let you dual (or triple) write while enabling live data quality and consistency checks.

Unclean kafka shutdown

I've been trying to get parkeeper with a consul (0.5.2) backend to work with kafka (0.8.2.2). It starts up successfully, and fills in the startup-default keys just fine, but has trouble on a control-C exit (and fails to start up subsequently).

I've done the "does-it-restart" test with zookeeper and kafka and it passes just fine.

Here is the relevant kafka shutdown log:

[2015-10-12 21:52:58,063] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2015-10-12 21:52:58,195] INFO Registered broker 0 at path /brokers/ids/0 with address 02aeb061d8dd:9092. (kafka.utils.ZkUtils$)
[2015-10-12 21:52:58,209] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
^C[2015-10-12 21:53:12,675] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2015-10-12 21:53:12,677] INFO [Kafka Server 0], Starting controlled shutdown (kafka.server.KafkaServer)
[2015-10-12 21:53:12,832] INFO [Kafka Server 0], Remaining partitions to move:  (kafka.server.KafkaServer)
[2015-10-12 21:53:12,833] INFO [Kafka Server 0], Error code from controller: -1 (kafka.server.KafkaServer)
[2015-10-12 21:53:12,834] ERROR [KafkaApi-0] error when handling request Name: ControlledShutdownRequest; Version: 0; CorrelationId: 0; BrokerId: 0 (kafka.server.KafkaApis)
kafka.common.BrokerNotAvailableException: Broker id 0 does not exist.
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply$mcV$sp(KafkaController.scala:236)
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply(KafkaController.scala:234)
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply(KafkaController.scala:234)
        at kafka.utils.Utils$.inLock(Utils.scala:535)
        at kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:234)
        at kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:154)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:66)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
        at java.lang.Thread.run(Thread.java:745)
[2015-10-12 21:53:17,835] WARN [Kafka Server 0], Retrying controlled shutdown after the previous attempt failed... (kafka.server.KafkaServer)
[2015-10-12 21:53:17,854] ERROR [KafkaApi-0] error when handling request Name: ControlledShutdownRequest; Version: 0; CorrelationId: 1; BrokerId: 0 (kafka.server.KafkaApis)
kafka.common.BrokerNotAvailableException: Broker id 0 does not exist.
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply$mcV$sp(KafkaController.scala:236)
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply(KafkaController.scala:234)
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply(KafkaController.scala:234)
        at kafka.utils.Utils$.inLock(Utils.scala:535)
        at kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:234)
        at kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:154)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:66)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
        at java.lang.Thread.run(Thread.java:745)
[2015-10-12 21:53:17,855] INFO [Kafka Server 0], Remaining partitions to move:  (kafka.server.KafkaServer)
[2015-10-12 21:53:17,856] INFO [Kafka Server 0], Error code from controller: -1 (kafka.server.KafkaServer)
[2015-10-12 21:53:22,858] WARN [Kafka Server 0], Retrying controlled shutdown after the previous attempt failed... (kafka.server.KafkaServer)
[2015-10-12 21:53:22,880] ERROR [KafkaApi-0] error when handling request Name: ControlledShutdownRequest; Version: 0; CorrelationId: 2; BrokerId: 0 (kafka.server.KafkaApis)
kafka.common.BrokerNotAvailableException: Broker id 0 does not exist.
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply$mcV$sp(KafkaController.scala:236)
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply(KafkaController.scala:234)
        at kafka.controller.KafkaController$$anonfun$shutdownBroker$1.apply(KafkaController.scala:234)
        at kafka.utils.Utils$.inLock(Utils.scala:535)
        at kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:234)
        at kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:154)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:66)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
        at java.lang.Thread.run(Thread.java:745)
[2015-10-12 21:53:22,883] INFO [Kafka Server 0], Remaining partitions to move:  (kafka.server.KafkaServer)
[2015-10-12 21:53:22,884] INFO [Kafka Server 0], Error code from controller: -1 (kafka.server.KafkaServer)
[2015-10-12 21:53:27,884] WARN [Kafka Server 0], Retrying controlled shutdown after the previous attempt failed... (kafka.server.KafkaServer)
[2015-10-12 21:53:27,887] INFO Closing socket connection to /172.17.0.1. (kafka.network.Processor)
[2015-10-12 21:53:27,888] WARN [Kafka Server 0], Proceeding to do an unclean shutdown as all the controlled shutdown attempts failed (kafka.server.KafkaServer)
[2015-10-12 21:53:27,889] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)
[2015-10-12 21:53:27,892] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)
[2015-10-12 21:53:27,893] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
[2015-10-12 21:53:27,894] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2015-10-12 21:53:28,242] INFO [Replica Manager on Broker 0]: Shut down (kafka.server.ReplicaManager)
[2015-10-12 21:53:28,243] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2015-10-12 21:53:28,244] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
[2015-10-12 21:53:28,255] INFO [Replica Manager on Broker 0]: Shut down completely (kafka.server.ReplicaManager)
[2015-10-12 21:53:28,256] INFO Shutting down. (kafka.log.LogManager)
[2015-10-12 21:53:28,263] INFO Shutdown complete. (kafka.log.LogManager)
[2015-10-12 21:53:28,266] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-10-12 21:53:28,270] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2015-10-12 21:53:28,270] INFO Session: 0x1 closed (org.apache.zookeeper.ZooKeeper)
[2015-10-12 21:53:28,271] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)

Haven't dug much further, let me know if I can get you any more info.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.