mapbox / cwput Goto Github PK

Seeing this for the disk metrics:

Invalid type for parameter MetricData[0].Value, value: 20, type: <type 'unicode'>, valid types: <type 'float'>, <class 'decimal.Decimal'>, <type 'int'>, <type 'long'>

Removing the quotes around the value in checks/disk works -- will write tests to see if I can reproduce and then PR.

If running several hundred instances, all using cwput, this can cause you to hit the cloudwatch API call limit. A random vary of +/- 5 seconds will help stagger the calls.

It looks like the format changed for the latest docker release from having spaces between the value and the unit to no longer having spaces:

Old format:

 Data Space Used: 14.84 GB
 Data Space Total: 309.2 GB
 Data Space Available: 294.4 GB

New Format:

 Data Space Used: 55.52GB
 Data Space Total: 3.073TB
 Data Space Available: 3.017TB

We'll need to change the docker info parsing script to handle this before we can upgrade docker.

cc/ @rclark @arunasank @brendanmcfarland

On EC2s that run docker and use the devicemapper storage type, there's been a change that breaks the disk check in cwput.

On docker version 1.12.6:

$ df -x tmpfs -x devtmpfs
Filesystem                 1K-blocks     Used Available Use% Mounted on
/dev/xvda1                   8123812  1984972   6038592  25% /
/dev/mapper/docker-persist 495279944 70200096 424811812  15% /mnt/persist

On docker version 17.03.1-ce:

$ df -x tmpfs -x devtmpfs -T
Filesystem                 Type  1K-blocks    Used  Available Use% Mounted on
/dev/xvda1                 ext4    8123812 1762756    6260808  22% /
/dev/mapper/docker-persist ext4 1230497624  105800 1229750320   1% /mnt/persist
/dev/dm-4                  ext4  230032572   69692  218254876   1% /mnt/persist/docker/devicemapper/mnt/0e1709e79c7fbb0869c4b251bebc1a0894a394c316f7861884f09968615b7ba7
/dev/dm-5                  ext4  230032572  171772  218152796   1% /mnt/persist/docker/devicemapper/mnt/b15f0017892182e0cfd112ce1d783e35de8ee92d1d5727703ff31e30bf078a9d
/dev/dm-6                  ext4  230032572  542612  217781956   1% /mnt/persist/docker/devicemapper/mnt/2ccf85dff56206156852d2e03bebc6fc833f910c91ec055807162c9d278ce09a
/dev/dm-9                  ext4  230032572 1876692  216447876   1% /mnt/persist/docker/devicemapper/mnt/5e58062b2eac74475aa00b91c8b061ca20da01b111cef9f97d5d3bf80ede3996
/dev/dm-7                  ext4  230032572 1879400  216445168   1% /mnt/persist/docker/devicemapper/mnt/ff97570bec4ead1db5a03acd36d7485fe65c1308d2f2c801f1c9a0e59534b09d
/dev/dm-8                  ext4  230032572 1879088  216445480   1% /mnt/persist/docker/devicemapper/mnt/106051b44b39f1054f6d11b5bcb4326a280cc496f9a0cf20a58f637d5c57639b

On the new version, each physical disk partition that contributes to docker's logical volume now appears in the df call that the disk check makes. As a result, the disk check attempts to put a metric for each of those physical partitions.

Then, when it comes time to actually put metrics, there are more than 20 data points included in the request, which is not allowed.

We need to find a way to exclude these additional partitions from the disk check, in a way that's agnostic as to whether or not you're running docker at all.

cc @vsmart

We've deleted the CI user for this project as part of archiving the repo. If we want to revive active development on this repo we'll need to recreate the CI user in a different account.

Currently cwput polls AWS to find an autoscaling group name, and uses that as a second dimension by which to track metrics: https://github.com/mapbox/cwput/blob/master/etc/checks/disk#L30

This means that a stack that does not use autoscaling cannot use cwput. Instead, what if second tracking of a metric was user-defined - passed as parameter to cwput?

cc @ianshward @yhahn

Hi,

I appreciate your work, and just wanted to inform you of maybe a bug.. but I am not sure.
I am using an Amazon EC2 instance running Ubuntu 14.04 LTS and if i manually run the command

cwput.bash

it is posting metrics to Amazon, however, it does not automatically do this every 60 seconds as written in your code "the forever loop".

This project provides some metrics out of the box. Document them.

Log to /var/log/cwput.log

If you have processes which use larged shared memory segments (like OSRM) we don't properly report the amount of memory used/available.

Additionally this project should probably switch to looking at /proc/meminfo directly and parsing what it needs from there.

Newer versions of the linux kernel will have MemAvailable but the still widely used Ubuntu 14.04 doesn't, we should handle this for folks.

I believe we're killing metrics that take more than 30 seconds without logging that fact; see https://github.com/mapbox/cwput/blob/master/bin/cwput.bash#L21

We should figure out how we can capture that information better.