Git Product home page Git Product logo

noah's Introduction

(please read here before continuing)

Noah

"look at this effing rainbow I just made for you"

Noah is an application registry inspired by Apache ZooKeeper

What does that mean? From the ZooKeeper Home Page:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

Essentially Noah is a port of parts of that functionality into a stateless RESTish application.

Some background

It's probably worth reading the following blog posts before going any further to clear up any possible misunderstandings. Noah is not a direct replacement for ZK. It's a conceptual port. More than anything, it was an itch I needed to scratch:

Also the following post was where I sort of discussed it early on:

Things Noah does not do

  • Paxos, ZAB or any other sort of consensus protocol
  • Noah itself is not distributed (yet).
  • ACLs (yet)
  • Leader election of any kind

Things Noah can do

note that these terms are fairly overloaded depending on who you talk to

  • Service registry
  • Node registry
  • Configuration Registry
  • Group Services
  • Watches (albeit differently)

Quick Start

The quickstart guide has been moved to the wiki:

Quick Start Guide

Design Goals

Noah has a few basic design goals:

  • The system MUST support RESTful interaction for operations where REST maps properly
  • The system MUST support basic concepts of hosts, services, applications and configurations
  • The system MUST support horizontal scaling.

Additionally:

  • The system SHOULD be flexible in deployment options.
  • The system SHOULD support watches similar to ZooKeeper
  • The system SHOULD support pluggable callbacks for watches.
  • The system SHOULD support being a client of itself.

Opinionated Stack

Noah is somewhat opinionated in its stack. Noah attempts to minimize the external requirements wherever possible to allow for the widest possible installation options. However, excellent work has been done to create performant and well-tested libraries that would be foolish to ignore. To this end, the current requirements are:

The above stack provides much of the functionality needed to port over ZooKeeper concepts.

Redis

Redis is the backbone of the system. Through the native datatypes and pubsub capabilities, much of the heavy lifting has already been done.

Sinatra

Sinatra is the perfect library for creating API-only style applications. It allows you do focus on the meat of what an endpoint should do instead of the hassle of creating the endpoint.

Ohm

Ohm is quite simply the most unobtrusive and flexible orm for Redis. It gets out of the way and allows you to very easily interact directly with Redis if the need arises

EventMachine

EventMachine combined with Redis pubsub forms the basis of the Watcher and callback system.

Motivation

It's something I've wanted to do for a while. Everytime I've needed something like Zookeeper, Zookeeper has always been too bulky and had too many moving parts. I've also always needed to interact with it from more than just Java or C. Sometimes it's been Ruby and sometimes it's been Python.

In the end, we reinvent the wheel ANYWAY. Maybe we do something like have our CM tool write our application config files with a list of memcached hosts. Maybe we write our own logic around (shudder) RMI to do some chatty broadcasting around the network for finding local nodes of type X. We always reinvent the wheel in some way.

More information

Here are a list of some key wiki pages:

  • Concepts How Noah views various object types in the system
  • API The API is currently in draft state. It will be finalized before the 1.0 release.
  • Example Use Cases Some use cases for Noah and how it would fit into an existing application or infrastructure
  • Watchers and Callbacks The general idea behind how Noah would implement watches
  • Watcher/Callback Examples Some example callbacks.

noah's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

noah's Issues

Ephemeral Tagging is broken

I just realized that ephemeral tagging was never implemented.

This leads to a bit of an issue since ephemerals have some weirdness with leading/trailing slashes that need to be worked out better.

Ephemerals uses "/"+params[:splat][0] so we're going to go with that for now.

Implement versioning

Not sure how many to keep at this point. Consider implementing some form of limited versioning.

Output formatting on callbacks

The idea is to allow callback data to be custom formatted by the plugin. Essentially the plugin writer would provide a method that could be called if it exists to format the response in a custom way.

Use case: RunDeck
RunDeck requires data to be in a certain format. See here under File Formats. The idea is that, in lieu of using Ephemerals to store the XML, we would be able to push updated resource data to RunDeck so it can consume it (say, new entries to the Host primitives)

Provide `short=true` for other primitives where applicable

Right now, the configurations top-level path accepts an optional query param short=true. This provides a condensed view of all configurations. In the interest of consistency, that same option should be allowed, at minimum, on the other primitive top-levels.

This might lead into other "filters" as query params which means the whole process of filtering needs to be handled better. For now, I'm only interested in the condensed view.

Provide uri for key-only enumeration

One issue that has come up from internal usage at VA is that listing large numbers of configurations (even with existing levels of specificity) is a bit cumbersome.

We're currently storing 180+ json configuration settings under a single tag. Some of these configurations are 32k in size. Pulling back all of these is rather cumbersome to enumerate client side.

Need a method for possible getting back a truncated view of items (possibly ?truncated=true or /*/truncated) that doesn't actually return the contents of the configuration but just the names and timestamp parameters. Possible providing a link to the full configuration (though that seems redundant.)

Dig my way out of being clever with tagging, linking and watching

So I thought it would be nice and easy if folks could just append an action to the end of a URL (i.e. /tag, /link, /watch). This worked great when the 4 original primitives were in there. Ephemerals went and screwed that up.

As it stands, the current API for doing those operations will mean that Ephemeral paths cannot end in /tag,/link,/watch until that changes (if it does). I'm somewhat okay with this for now. You can still have it in the rest of the URL for an ephemeral, just not at the end.

This is a place holder to remind me how shortsighted I was.

Add url to watch messages for object location

Need to add the url to the object in the body of the watch message when it is fired. This is similar to pub category but should provide the URL of the CRUDed object to be more informative.

Authentication

Decide on and implement some form of authentication. Most likely token based similar to how links/tags work. I.E.

GET /applications/some_app/token - Generates a new token for usage
DELETE /applications/some_app/token - Pass token id as JSON body
PUT /applications/some_app/token - Allow a user generated token for access control

noah-watcher daemon doesn't support REDIS_URL env

Because of the currently terrible coupling of the watcher daemon to the Ruby API, changing the connection host doesn't work. The libraries being pulled in use REDIS_URL but the binscript does not.

As an interim to the breakout of agent into its own gem, need to build in CLI options to set that properly so it will work.

This became visible when using the noah cookbook to install which puts redis on port 6381.

Tags not work after deleting related value

The flow:

Crete two ephemerals "/mydata/eph1" and "/mydata/eph2":
PUT http://my_host:5678/ephemerals/mydata/eph1
data1
PUT http://my_host:5678/ephemerals/mydata/eph2
data2

relate it to tag "mytag":
PUT http://my_host:5678/ephemerals/mydata/eph1/tag
{"tags":"mytag"}
PUT http://my_host:5678/ephemerals/mydata/eph2/tag
{"tags":"mytag"}

Currently I can get all ephemerals by tag by:
GET http://my_host:5678/tags/mytag

Delete one of ephemerals by
DELETE http://my_host:5678/ephemerals/mydata/eph1

Now, when I try to get it by tag again I got error:
{"result":"failure","error_message":"undefined method `name' for nil:NilClass"}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.