AppController crashes in response to failed ZooKeeper operations about gts HOT 3 CLOSED

Updated description above. As the description above indicates, this issue is really multiple bugs:

1. Failed ZooKeeper operations crash the AppController (this issue).
2. The AppController erases nginx config files when it starts up.
3. The AppController doesn't revive properly. This is a known issue (#33)

Opened issues for 2 (#67), and continuing to investigate this issue.

from gts.

The branch proposed above should fix this problem. Fixed it by manually stating a longer timeout between our client and Zookeeper (thus avoiding SessionExpired exceptions), and by retrying in cases where we couldn't successfully write to Zookeeper (instead of raising FailedZooKeeperOperationException). We now try five times to do a write and then simply skip on the write.

In the future, we should revisit when simply skipping on the write is ok, and when it isn't.

from gts.

QA'ed and merged in!

from gts.