Should we include the Lynis security scanner on Docker host nodes and then generate reports?

A GUI application that runs in the system tray and that notifies the user of updated neon-cli builds and optionally downloads and installs updates.

Implement a service that reads a schedule of activities from Consul and then fires an event by launching a specified container, with optional arguments. The idea here is to provide a central place for scheduling cluster or service activities (e.g. backups).

Implement upgrade commands:

• docker [version]
• consul [version]
• vault [version]

neon-cli deploys Redis caching.

Implement neonHIVE provisioning on AWS.

Verify that all neon-cli commands work on Windows.

Implement a service that runs on every container and queries the Docker and perhaps the OS for running containers and volumes and then updates entries in Consul. This would be useful for gathering cluster status for dashboards, etc. as well as for managing stateful services.

Ideally, this would be deployed as a Docker service if we can resolve the Unix socket issue #8.

Verify that all neon-cli commands work on OSX.

Implement a WiX or other setup application that deploys the neon-cli and any dependencies on Windows. Here are the known components we'll need to deploy:

• .NET Core run-time
• neon-cli
• Docker
• WinSCP
• PuTTY
• neon-cli updater

I looked at this sometime back and I don't remember why I didn't complete this. This would be nice to have for the initial RTW.

neon-cli deploys Consul key/value store.

neon-cli deploys Microsoft SQL Server

Complete the following post-setup health checks:

• Consul
• Vault
• Log Pipeline
• Elasticsearch cluster
• Kibana

Implement cluster deployment on the Google cloud. I haven't seen a .NET REST API wrapper yet from Google but it might be possible to embed the Google cliin the neon-cli container and exec commands using that instead.

Create logos

• neonFORHGE
• neonHIVE

neon-cli deploys Git source control.

syslog: Service=vault, Facility=4 (auth) events are coming from Vault. These should be recorded as AUDIT events by the logging pipeline.

Implement a service wrapper around the Azure and/or AWS mobile notification services. This should probably include some persistence and retry capabilities.

The neon-cluster-manager is currently deployed as local containers on the manager nodes attached to the host network. I'd much rather have this be deployed as a Docker service.

The essential constrains are:

• neon-cluster-manager requires access to the Docker socket
• The Docker socket cannot be exposed outside of the machine without security risks or having to implement complex mutual TLS authentication
• This means that neon-cluster-manager can access Docker via its Unix socket (mounted into the service) or a TCP socket on 127.0.0.1
• .NET Core does not appear to support Unix sockets out-of-the-box
• Docker services cannot be deployed on the host network

The task here is to investigate whether further into .NET Core and Unix sockets:

• Perhaps a URL like http://unix:/var/aspnet/HelloMVC/kestrel.sock would work. This seems like somewhat of a standard in the Linux world.
• Investigate .NET Core/Standard 2.0
• Port/implement an HTTP Handler that works over Unix sockets

Investigate whether NeonCluster should be deployed to the store or not.

• Sign up for a store account
• Talk to the store folks about products consisting of multiple images
• Deploy to store

Promotion Activities

• Work on my LinkedIn profile.
• Issue a press release
• Reach out to important Docker influencers
• More???

neon-cli deploys Zookeeper service discovery.

Implement neonHIVE provisioning on the Google Cloud.

neon-cli deploys Kafka queuing.

This will be a Docker service that presents a REST endpoint for accepting outbound emails. Emails can then be delivered directly or perhaps through an external email forwarder. I expect that this will be an SMTP based delivery in both cases.

Emails that could not immediately be delivered will be persisted somewhere and retried later. This implies separate persisted storage or a persisted queue. I need to think about how I should do this.

I'll probably have to deploy a centralized persisted queue or database.

Capture and present stats from HAProxy instance

• Implement a service using Redis to roll-up stats across multiple instances.
• Implement an Nginx site to render this (Overview Dashboard?)
• Dashboard site implements pass-thru to HAProxy instance stats pages

Create and deploy the NeonForge.com website.

• Choose a hosting solution. WordPress or maybe just GitHub Pages.
• Basic company information
• Blog posts
• Summary and link to NeonForge

Implement standards around deploying database services to a cluster.

• Services should have a base name: e.g. main-couchbase
• A management service will be deployed named likeneon-main-couchbase-manager will be deployed on a manager node to manage cluster configuration and monitoring.
• The actual database nodes will be deployed as containers on the target nodes. These containers will be named like neon-db-couchbase-NODENAME.
• Service volumes will be named like neon-db-couchbase.
• Service volumes will be labeled with neon-db-service=true
• Both the database containers and manager services will be labeled with neon-db-service=true and neon-db-manager=true
• Swarm nodes that will host a database containers will be labeled like neon-db-main-couchbase=true, identifying.
• Information including the node details and a connection string/config will be persisted to Consul at neon/databases/main-couchbase

See if I can get anybody to (like Selvan) to play with this before I release it.

Secure cluster Consul traffic with TLS and a self-signed certificate.

• Configure CA and certs link
• Consul HTTPS via environment variable
• Consul can't do HTTPS issue. Perhaps this is on track for 0.7.4?
• Also implement support for a real certificate, persisted to Vault.

neon-cli deploys RabbitMQ queuing.

Implement smart card encryption.

Elastic no longer publishes their images to DockerHub and they are no longer based on Ubuntu, so my images don't build when based on their new location. Either redo my images to work with the new Elastic ones or build my images directly from Elastic product downloads.

The neon-vault-proxy service pings Vault with health checks which will end up in syslog and make their way down the logging pipeline. It would be nice to filter these out before the end up in Elasticsearch, wasting space.

It would be nice to be able to expose availability set (or fault domain) as node labels for all platforms so things like Docker service deployment or persisted service replicas could be "rack" aware.

I'm not sure this is possible though. Look at this after we're able to provision on all three cloud platforms.

Implement a custom parser for dockerd syslog events.

Implement the neon ssh and neon scp commands on Macintosh OSX. We'll need to identify applications we can call, perhaps just the built-in command line tools to start.

Commands that launch a specified dashboard in a browser window.

• Kibana
• Consul UI

It would also be interesting to be able to register, list, and remove dashboards by name, e.g. mapping a name to a URI. This would extend the concept and could also be used in an global cluster portal to render dashboards as tabs (in iframes).

Build and deploy the neonHIVE.com website.

• Basic Information
• Getting Started
• Documentation
• Videos

Cloud environments have built-in load balancers that can monitor node health and direct inbound cluster traffic only to health nodes.

The same functionality could be provided in on-premise deployments by configuring a load balancer between the router and the cluster, with the load balancer forwarding traffic to healthy nodes. Commercal load balancers can often be provisioned in pairs such that that effectively share the same IP address. This is done via some kind of leader election scheme where one load balancer assigns the IP address to itself and receives and forwards traffic from the router where as the other load balancer is on standby. If the leader fails, the standby load balancer becomes the leader, assigns the IP address to itself and takes over.

It might be possible to embed functionality like this into NeonCluster:

• Allocate a shared IP address that will be used to direct router traffic into the cluster. This will not be one of the node IP addresses.

• Implement a native host service that uses Consul for leader election and assigns the IP address to host node when it becomes the leader and unassigns the address when it becomes a follower. We'll probably need to do some ARP notifications so the router picks up changes.

Implement a standard container and/or service label that identifies the log format. The idea here is to be able to identify that a service generates logs in a know format that the neon-log-collector service is able to do deeper inspection on.

Here are links to some ideas:

https://docs.docker.com/engine/admin/logging/log_tags/
moby/moby#21724

neon-cli deploys Cassandra database.

neon-cli deploys MySQL database.

Implement a setup application that deploys the neon-cli and any dependencies to OSX. Here are the known components we'll need to deploy:

• .NET Core run-time
• neon-cli
• Docker
• SCP client?
• SSH client?
• neon-cli updater

neon-cli deploys elasticsearch

neon-cli deploys Couchbase database.

Here's some useful information: link

Setup cluster as a Docker service: link

neon-cli deploys Etc key/value store.

The current image building mechanism is a hack. This needs a comprehensive refactor:

• Develop a versioning strategy
• What to do about development branches (edge versions or different repos)
• Have DockerHub build off of new GitHub image repos
• Look at Debian/Alpine base images again for size
• Use new Docker multi-stage builds to reduce .NET Core related images (separate the app DLL from the .NET Core DLLs and other dependencies).

Here are some ideas:

https://github.com/ciandcd/awesome-ciandcd#build-and-release-system
https://en.wikipedia.org/wiki/Comparison_of_continuous_integration_software
https://blog.newrelic.com/2016/09/21/docker-images-continuous-integration/
http://blog.shippable.com/immutable-containers-with-version-tags-on-docker-hub