heroku / heroku-production-check Goto Github PK

View Code? Open in Web Editor NEW

11.0 98.0 3.0 17 KB

Home Page: https://devcenter.heroku.com/articles/maximizing-availability

License: MIT License

Ruby 100.00%

heroku-production-check's Introduction

Heroku App Production Check

A Heroku client plugin to check how well your app meets Heroku's guidlines for maximizing app availability.

Setup

$ heroku plugins:install https://github.com/heroku/heroku-production-check.git

Usage

$ heroku production:check -a vault
=== Production check for vault
Cedar                         Passed
Dyno Redundancy               Passed
Production Database           Failed 	 remedy: http://bit.ly/PWsbrJ
Follower Database             Failed 	 remedy: http://bit.ly/MGsk39
SSL Endpoint                  Passed
DNS Configuration             Passed
Log Drains                    Passed

heroku-production-check's People

Stargazers

Watchers

Forkers

socialcloudtech gt-voltrao9ine

heroku-production-check's Issues

open source

Should be able to eventually make this open source (and switch the Readme to point to the https:// URL).

remedy links broken

The remedy links no longer fully work because of a change in the URL structure in @heroku/devcenter. For example:

=== Production check for beta
Cedar                         Passed
Dyno Redundancy               Passed
Production Database           Passed
Follower Database             Passed
SSL Endpoint                  Failed     remedy: http://bit.ly/PfzI7x
DNS Configuration             Passed
Log Drains                    Passed

The expansion for that bit.ly link is:

http://bit.ly/PfzI7x -> https://devcenter.heroku.com/articles/maximizing-availability#dns__ssl

But the current link is:

https://devcenter.heroku.com/articles/maximizing-availability#dns--ssl

dyno redundancy check should not apply to non-web apps

Insofar as the dyno redundancy checks only looks for multiple web dynos, it shouldn't apply at all to non-web apps:

$ heroku production:check -a sigma-production | grep Dyno
Dyno Redundancy               Failed     remedy: http://bit.ly/SSHYev

$ heroku ps -a sigma-production
=== update: `bin/update`
update.1: crashed for 10m

help text

$ heroku help production | tail -n 6
Additional commands, type "heroku help COMMAND" for more details:

  production:check  # <-- help text goes here -->
  production:off    #  takes an app out of production mode
  production:on     #  sets an app to production mode

more robust dyno redundancy logic

Current:

Ensure > 2 web processes.

Consider:

Ensure > 2 web processes if any web processes (covers non-web apps).
Ensure or warn if not > 2 web processes of any present process type.

incorrect dns check logic

It should not be acceptable to A-record to our apex IPs:

~ $ heroku sudo production:check -a gist.io
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed
~ $ host gist.io
gist.io has address 75.101.145.87
gist.io has address 174.129.212.2
gist.io has address 75.101.163.44

I'm not sure exactly how to handle special records that use CNAME-style lookups to return IPs, but to start we could at least blacklist this type of setup.

It is likely that production Heroku apps will be using ALIAS records, CDNs and other sophisticated DNS setups. Instead of checking all of the sophistication, we should just fail the DNS check if an app has A records pointed at Heroku ip addresses or if apps have CNAMES pointed at proxy.heroku.com

\cc @neilmiddleton

relax dyno redundancy

It might be unreasonable to expect that customers run multiple clockwork processes. We also want customers who use this tool to feel like they have completed the list with success.

skip ssl endpoint and dns checks for non-web apps

If an app has no web processes you can completely skip these checks:

...
SSL Endpoint... N/A
DNS configuration... N/A
...

ignore empty dev databases

Usually these are just auto-added databases that aren't being used.

no endpoint required if app happens to be using herokuapp domain

E.g.:

$ heroku domains -a tines-production
No domain names for tines-production

$ heroku addons -a tines-production
=== tines-production Configured Add-ons
heroku-postgresql:zilla   HEROKU_POSTGRESQL_GREEN

$ heroku production:check -a tines-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

SSL Endpoint should be N/A here.

dns check failed when set up through nile-ha

$ heroku domains -a anvil-production
=== Domain names for anvil-production
aichi-5565.herokussl.com
api.anvilworks.org

$ heroku production:check -a anvil-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... Failed
Follower Database... Failed
SSL Endpoint... OK
DNS Configuration... Failed
Log Drains... OK

mcgranaghan feedback

The word "checking" appears on the first line but not on any others:

$ heroku production:check
Checking Cedar... OK
Dyno Redundancy... OK
...

Perhaps to avoid this and for symmetry with other commands:

$ heroku production:check
=== Production check for tasseo-production
Cedar... OK
Dyno redundancy... OK

I failed the database checks but didn't have any databases installed:

$ heroku production:check -a tasseo-production
...
Production Database... Failed
Follower Database... Failed
...

$ heroku addons -a tasseo-production
=== tasseo-production Configured Add-ons
deployhooks:campfire
redistogo:nano
ssl:endpoint

In this case consider:

$ heroku production:check -a tasseo-production
...
Production database... N/A
Follower database... N/A
...

It would be nice to know why my checks failed / how I can fix them. Perhaps:

$ heroku production:check -a tasseo-production
...
Log drains... Failed: no drains found

See https://devcenter.heroku.com/articles/maximizing-availability for information
on production app configuration.

For symmetry with other Heroku client plugins like heroku-anvil and heroku-sudo, consider the name heroku-production-check.

Error handling could use tweaking. The current status:

$ heroku production:check -a shogun
Checking Cedar...  !    You do not have direct access or team access to shogun.

/cc @ddollar on CLI style.

Check for thin vs unicorn

Would be great if this included a concurrency check on the web server level.

cross-check with actual production apps

Here are my very initial results:

~ $ echo 'echo $1; heroku production:check -a $1; echo' > check.sh
~ $ heroku list | grep -- -production | cut -d ' ' -f 1 | xargs -n 1 sh check.sh
backstop-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... Failed
Follower Database... Failed
SSL Endpoint... OK
DNS Configuration... OK
Log Drains... Failed

bob-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

canary-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

census-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

codon-production-aspen
Checking Cedar... Failed
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

codon-production-bamboo
Checking Cedar... Failed
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

codon-production-cedar
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

codon-production-monitor
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

cron-check-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

devcloud-ci-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

elbssl-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... OK
Follower Database... Failed
SSL Endpoint... OK
DNS Configuration... OK
Log Drains... OK

exprd-canary-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

exprd-census-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

exprd-core-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... OK
Follower Database... OK
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

exprd-packaging-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

exprd-tines-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

exprd-wolfpack-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

izanami-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

maas-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... OK
Follower Database... OK
SSL Endpoint... OK
DNS Configuration... OK
Log Drains... Failed

pulse-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... Failed
Follower Database... Failed
SSL Endpoint... OK
DNS Configuration... OK
Log Drains... OK

sigma-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

ssl-ip-check-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... Failed

status-buddy-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

tasseo-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... Failed
Follower Database... Failed
SSL Endpoint... OK
DNS Configuration... OK
Log Drains... Failed

tines-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... OK
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

umpire-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... Failed
Follower Database... Failed
SSL Endpoint... OK
DNS Configuration... OK
Log Drains... Failed

vacuole-production
Checking Cedar... OK
Dyno Redundancy... OK
Production Database... OK
Follower Database... OK
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

wolfpack-raven-production
Checking Cedar... OK
Dyno Redundancy... Failed
Production Database... Failed
Follower Database... Failed
SSL Endpoint... Failed
DNS Configuration... OK
Log Drains... OK

I'll follow up with an analysis to see what these results show in terms of the apps and the checks themselves.