fullstop needs the Taupage user data for audit reporting

The "INSTANCE_ID" is AWS-specific and should not be set automatically (otherwise we would have to add all kind of EC2 metadata as env variables)

Just to have another Filter option in our central log solutions, we want to separate syslog again into different log files.

The parser to use for the application.log is hardcoded to 'slf4j'. We have different log formats for our applications and would like to introduce multiple parsers for that.

This could be configurable in the user data just like the scalyr api token.

concept how to add a trusted relay

Info material:
http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-using-smtp-integrate.html

It should be simple to send from the instances (Docker containers) to send transactional emails using Amazon SES.

we need such a feature for docker command for our Zookeeper appliance

for example we want to do this by senza create:

sudo docker run -p 2181:2181 -p 2888:2888 -p 3888:3888 -d --net=host efd7a9692031

we need to add this --net=host

Thanks

see docker

The "prepare-disks" script seems to fail (sometimes) when running mdadm (to build a raid with two devices):

Jun 15 08:10:27 ip-172-31-15-220 kernel: [   23.830888] blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
Jun 15 08:10:27 ip-172-31-15-220 kernel: [   23.834668]  xvdf: unknown partition table
Jun 15 08:10:29 ip-172-31-15-220 kernel: [   25.728675] blkfront: xvdi: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
Jun 15 08:10:29 ip-172-31-15-220 kernel: [   25.733076]  xvdi: unknown partition table
Jun 15 08:10:31 ip-172-31-15-220 kernel: [   28.031018] blkfront: xvdg: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
Jun 15 08:10:31 ip-172-31-15-220 kernel: [   28.035338]  xvdg: unknown partition table
Jun 15 08:10:29 ip-172-31-15-220 taupage-init: message repeated 2 times: [ Waiting for /dev/xvdh to stabilize]
Jun 15 08:10:32 ip-172-31-15-220 taupage-init: mdadm: Cannot find /dev/xvdh: No such file or directory
Jun 15 08:10:32 ip-172-31-15-220 ntpd[1534]: ntpd exiting on signal 15
Jun 15 08:10:32 ip-172-31-15-220 taupage-init: Traceback (most recent call last):
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:   File "./init.d/10-prepare-disks.py", line 237, in <module>
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:     main()
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:   File "./init.d/10-prepare-disks.py", line 230, in main
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:     handle_volumes(args, config)
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:   File "./init.d/10-prepare-disks.py", line 206, in handle_volumes
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:     handle_raid_volumes(volumes.get("raid"))
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:   File "./init.d/10-prepare-disks.py", line 193, in handle_raid_volumes
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:     create_raid_device(raid_device, raid_config)
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:   File "./init.d/10-prepare-disks.py", line 184, in create_raid_device
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:     subprocess.check_call(call)
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:   File "/usr/lib/python3.4/subprocess.py", line 557, in check_call
Jun 15 08:10:32 ip-172-31-15-220 taupage-init:     raise CalledProcessError(retcode, cmd)
Jun 15 08:10:32 ip-172-31-15-220 taupage-init: subprocess.CalledProcessError: Command '['mdadm', '--build', '/dev/md/sampleraid1', '--level=1', '--raid-devices=2', '/dev/xvdh', '/dev/xvdi']' returned non-zero exit status 1
Jun 15 08:10:32 ip-172-31-15-220 taupage-init: Failed to start 10-prepare-disks.py
Jun 15 08:10:33 ip-172-31-15-220 kernel: [   30.036629] blkfront: xvdh: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
Jun 15 08:10:33 ip-172-31-15-220 kernel: [   30.041170]  xvdh: unknown partition table

According to the Docker docs this can be done via --privileged.
It's needed in order to run Docker in Docker for example.

More information can be found here:
https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration

We are not using Loggly anymore (we were not satisfied with their performance), thus we should remove support for it in the Taupage AMI.

The etcd discovery process should take an optional health check configuration from the taupage config like "etcd_health_check" which, when set, should be used to also check and only on success to ping the etcd cluster.

Allow to specify a health endpoint to taupage, so that we do not blindly send a SUCCESS to CF only because the docker container started but also because this health endpoint returned true

Something like:

health_endpoint: 8080:/health

for example review auditd cnofig

see mint&berry project

ATM the build process will mostly continue in case of errors.

I would like to suggest to change so that any error - and especially unhandled error - will abort the build process.

A good start is to set

set -u -e -E -C -o pipefail

in create-ami.sh, setup.sh, test.sh etc.

https://github.com/prometheus/node_exporter

The AMI should be properly tested with Serverspec. Therefore additional tests have to be written and tests need to run, when the image was created, before starting the HTTP server.

module for logging solution

Jun 15 07:55:12 ip-172-31-12-207 taupage-init: Traceback (most recent call last):
Jun 15 07:55:12 ip-172-31-12-207 taupage-init: File "/opt/taupage/runtime/Docker.py", line 297, in
Jun 15 07:55:12 ip-172-31-12-207 taupage-init: main(args)
Jun 15 07:55:12 ip-172-31-12-207 taupage-init: File "/opt/taupage/runtime/Docker.py", line 278, in main
Jun 15 07:55:12 ip-172-31-12-207 taupage-init: cmd += list(f(config))
Jun 15 07:55:12 ip-172-31-12-207 taupage-init: File "/opt/taupage/runtime/Docker.py", line 132, in get_port_options
Jun 15 07:55:12 ip-172-31-12-207 taupage-init: if '/' in host_port:
Jun 15 07:55:12 ip-172-31-12-207 taupage-init: TypeError: argument of type 'int' is not iterable

Add HipChat status message when Taupage is built. This is optional.

Documentation:
https://www.hipchat.com/docs/apiv2/auth
Example:
https://gist.github.com/danriti/7345074

According to docs Username is also accepted -u="": Username or UID so no need to calculate UID programmatically any more.

Check and fix: Docker container does not always start up on boot (devicemapper errors), but works on second try

during build, index filesystem, during runtime check filesystem

If "etcd_discovery_domain" is specified and the etcd proxy gets started, the container still does not get the "ETCD_URL" set. Should point to the running etcd proxy (that by purpose was started on the docker0 IP).

Check the following process list:

4 0 1237 1 20 0 4440 644 wait Ss ? 0:00 /bin/sh -e -c /opt/zalando/init.sh 2>&1 | logger -t zalando-init /bin/sh
0 0 1238 1237 20 0 4440 704 wait S ? 0:00 _ /bin/sh /opt/zalando/init.sh
0 0 1431 1238 20 0 4440 656 wait S ? 0:00 | _ /bin/sh ./init.d/81-register-scalyr-agent.sh
4 0 1458 1431 20 0 18020 1640 wait S ? 0:00 | _ bash /opt/zalando/installfiles/install-scalyr-agent-2.sh --set-api-key 0xxx --star
0 0 1781 1458 20 0 61228 32288 poll_s S ? 0:00 | _ apt-get -y install scalyr-agent-2
0 0 1786 1781 20 0 182260 4396 poll_s S ? 0:00 | _ /usr/lib/apt/methods/https

Sorting str and int fails, see https://github.com/zalando-stups/taupage/blob/master/runtime/usr/local/lib/python3.4/dist-packages/taupage/__init__.py#L51

OSQuery[0] is a nice tool from facebook that allows for querying information about servers in a SQL like manner.

[0] https://osquery.io/

https://github.com/CISOfy/lynis

The realm that is used to get an OAuth2 access token (e.g. for pulling the docker image form pierone) should be configurable (on AMI build time), similar to the token service url.

Check out Brandon Greggs's recommendation for tuning EC2 instances:
http://www.brendangregg.com/blog/2015-03-03/performance-tuning-linux-instances-on-ec2.html

It would be a nice feature if Taupage image would decrypt environment variables using Amazons 'kms' before setting them for the docker run time.

This would basically solve storing credentials and handling them for a lot of frameworks.

Add configuration options for failure handling (the Docker container stops running). Possible handling:

• Restart the container endlessly
• Restart the contaienr X times and then shut down
• Shutdown immidiatly

Currently, if a Docker image breaks, the server just keeps running without application.

This new property allows running "one-time" tasks/jobs.

Proposal:

shutdown: on-success # stop EC2 instance when container exits with code zero
shutdown: on-exit # stop EC2 instance when container exits       
shutdown: never # default: never stop EC2 instance

We should not hard code company names, but instead use our product names (in this case "Taupage").

We recently had issues with fullstop. During init the taupage config is pushed to fullstop for auditing. This call usual took ~5min to timeout.

This timeout should be lowered allowing the instance startup to fail faster otherwise this will compete with the healthchecks.

To really have immutable servers and to decrease security risks (see http://container-solutions.com/docker-security-cheat-sheet/), we should allow running Docker containers with read-only file system (Docker "--read-only" flag).

We should probably think about making this the default setting for new applications.

zalando-stups/fullstop#109

Investigate if we want to support New Relic Docker integration.