Git Product home page Git Product logo

gocollect's Introduction

GoCollect

bettercodehub goreportcard

GoCollect collects various pieces of system info and publishes them to a central server.

The intent of GoCollect is to create a map of your servers with slow and never changing data items. Where you may use Zabbix for semi-realtime monitoring of integer values like current CPU usage, you use GoCollect to collect values like hard drive serial numbers, IPMI IP-addresses and versions of installed OS packages.

Installing

make && make install
cp /etc/gocollect.conf.sample /etc/gocollect.conf
# edit /etc/gocollect.conf
# then start/restart gocollect using your favorite init method

You may need to set up a go path first:

mkdir -p ~/go
export GOPATH=~/go

And check this out inside that:

git clone https://github.com/ossobv/gocollect \
  $GOPATH/src/github.com/ossobv/gocollect

And install prerequisites:

go get github.com/ossobv/go-getopt

Possibly set env to old style module handling:

# go.mod file not found in current directory or any parent directory...
go env -w GO111MODULE=off  # sets ~/.config/go/env: GO111MODULE=off

Packaging for Debian

Prerequisites:

$ sudo apt-get install git-buildpackage dh-make dh-systemd

Optional:

$ cat > .git/gbp.conf << EOF
[DEFAULT]
upstream-branch=main
debian-branch=debian

[buildpackage]
upstream-tag = v%(version)s
EOF

Running:

$ git checkout debian
$ gbp buildpackage -sa \
    --git-debian-branch=debian --git-upstream-tag='v%(version)s'

Note that release candidate versions must be tagged as v1.2_rc3. gbp-buildpackage rewrites the underscore to a debian-style tilde. Pre-release development versions shall be called v1.3_dev which sorts before v1.3_rc1, which in turn sorts before v1.3.

Packaging a tarball

To to create a tarball with the latest version, including a config file, do this:

$ cd gocollect-client
$ TGZ_CONFIG=/path/to/gocollect.conf make tgz
...
Created: gocollect-v0.4~rc6+1.g83d4-md5conf-c0f48c3.tar.gz

You can then extract and run that archive on the target machine like this:

$ cat gocollect-v0.4~rc6+1.g83d4-md5conf-c0f48c3.tar.gz | sudo tar -xzvC /
$ sudo /etc/init.d/gocollect start

You may need to install additional dependencies first to get all collectors to work properly. For example smartmontools or ipmitool.

Golang notes to self

License

The source code is licensed according to the GNU GPLv3+; see LICENSE.

The artwork – the GoCollect logo – is licensed according to the Attribution-NonCommercial-ShareAlike 4.0 International Creative Commons license (CC BY-NC-SA 4.0); see LICENSE.CC.BY-NC-SA.4-0.txt.

gocollect's People

Contributors

dependabot[bot] avatar urth avatar wdoekes avatar

Stargazers

avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

pombredanne

gocollect's Issues

Have --test-key imply --one-shot 

# gocollect -k os.keys
gocollect: --test-key only works together with --one-shot.

If --test-key is the only option given it may just as well imply --one-shot.

Upgrade from UPSTART to SYSTEMD fails to auto-heal startup scripts

This is only a problem because we use multi-distro packages. If we used a different counter on the different distro-releases (~14.04, ~16.04) it would auto-heal because an install does fix the setup.

Should we add a collector that checks whether it auto-starts and if not, correct it?

Fix problems with lshw updating clock speed 

          {
            "id" : "cpu",
            "class" : "processor",
            "claimed" : true,
            "handle" : "DMI:0034",
            "description" : "CPU",
            "product" : "Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz",
            "vendor" : "Intel Corp.",
            "physid" : "34",
            "businfo" : "cpu@0",
            "version" : "Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz",
            "slot" : "CPU1",
            "units" : "Hz",
!           "size" : 875468000,
            "capacity" : 4000000000,
            "width" : 64,
            "clock" : 100000000,
            "configuration" : {
              "cores" : "8",
              "enabledcores" : "8",
              "threads" : "16"
            },

That bit in app.lshw keeps updating on many platforms:

sed -e '/"id"[[:blank:]:]*"cpu",/,/"size"[[:blank:]:]*[0-9]\+/{s/"size"[[:blank:]:]*[0-9]\+/"size": -1/g}'

This would fix things.

collect lldpd information

Use case:

  • Helps locating hosts in the network (as you can see its neighbors)
  • When we do large deployments this should be able to support us in adding patches in netbox in an automated way (as long as its a single patch between 2 devices).
  • Could help in verification of network topology in netbox, correct interface names, etc.
  • Verify lldpd configuration

Its not a high priority item for now, we could actually do this on the switch side already as an intermediate step.

It looks easy to implement: when lldpd is installed (ubuntu, debian, cumulus), read lldp neighbors with lldpctl (supports json and other output formats).

# lldpctl  -h
Usage:   lldpctl [OPTIONS ...] [COMMAND ...]
Version: lldpd 0.9.9

-d          Enable more debugging information.
-u socket   Specify the Unix-domain socket used for communication with lldpd(8).
-f format   Choose output format (plain, keyvalue, json, json0, xml).

see manual page lldpcli(8) for more information

lldpd does need to be configured to some extend. But thats out of scope for gocollect and should be handled by provisioning process.

dependency on smartmontools pulls unneeded packages on virtual servers

Depends: libc6 (>= 2.2.5), dmidecode, iproute | iproute2, kmod | module-init-tools, lshw, openssh-client, smartmontools

Dependency on smartmontools pulls unneeded packages on virtual servers.

Possible solution would be creating an additional (meta)package for hardware servers.

lockf.py should not be 755 in debian builds (0.4~rc10) 

2016/07/12 09:51:52 collector[lockf.py]: /usr/share/gocollect/collectors/lockf.py error: exit status 1

GoCollect should not attempt to execute it. We should (a) remove the shebang en (b) move it to a helpers dir so it's clear that it's not a collector itself.

Collect cpu vulnerabilities (and mitigation state) as reported by linux kernel

Newer Linux kernel report cpu vulnerabilities and if they are mitigated on the system in /sys/devices/system/cpu/vulnerabilities/, would be great to collect any files in that path with their contents.

example output

cat /sys/devices/system/cpu/vulnerabilities/{l1tf,meltdown,spec_store_bypass,spectre_v1,spectre_v2}
Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
Mitigation: PTI
Mitigation: Speculative Store Bypass disabled via prctl and seccomp
Mitigation: __user pointer sanitization
Mitigation: Full generic retpoline, IBPB, IBRS_FW

Start creating collectors in Go as well

We need go-based collectors to explore dockers /containers/json output. Parsing that JSON is no fun with just awk. And cheating by using python is just wrong.

So, we'll need:

  • shellcollectors
  • internalcollectors

And for virt_docker.go, we'll want to be able to run a shell-script collector from the internal-collector. (Possibly with extra envvars?) We'll want to call os.pkg but with a different root-path.

hostname -f does not always give anything 

core@ossoio-k8s-wp-node1 ~ $ hostname
ossoio-k8s-wp-node1
core@ossoio-k8s-wp-node1 ~ $ hostname -f 2>/dev/null || echo no results
no results

I'd rather see ossoio-k8s-wp-node1.without.fqdn here than (empty).

Fix debian release-upgrade problem (upstart2systemd)

When doing a release-upgrade from Xenial to Systemd, the startup config is not reinstalled. So you're left with a not-working upstart config.

You could apt-get install --reinstall gocollect but it'd be nicer if this was fixed automatically.

Possible methods:

  • create a collector that sets up the startup scripts (fugly!)
  • give all packages an increasing version number (0.4.0-1~ubu14.04 and 0.4.0-1~ubu16.04)
    (if the package management accepts this after the build) (should test)
  • ???

Fix open lintian issues 

E: gocollect changes: bad-distribution-in-changes-file stable
W: gocollect source: out-of-date-standards-version 3.9.6 (current is 3.9.7)
W: gocollect: hardening-no-relro usr/sbin/gocollect
W: gocollect: binary-without-manpage usr/sbin/gocollect
N: 2 tags overridden (1 error, 1 warning)

dpkg lockf during dist-upgrade, not nice 

root      1269  0.0  2.3 427356 93296 pts/4    S+   15:51   0:00                                  \_ /usr/bin/python3 /tmp/ubuntu-release-upgrader-8fr77y46/xenial --mode=server --frontend=DistUpgradeViewT
root      1361  0.0  0.0   4508   792 pts/4    S+   15:51   0:00                                      \_ sh -c service gocollect status >/dev/null && if ! service gocollect restart; then echo '(suppressin
root      1372  0.0  0.0   4508  1708 pts/4    S+   15:51   0:00                                          \_ /bin/sh /etc/init.d/gocollect restart
root      1401  0.2  0.2  14928 11380 pts/4    Sl+  15:51   0:00                                              \_ /usr/sbin/gocollect
root      1647  0.0  0.0   8648   784 pts/4    S    15:51   0:00                                                  \_ timeout 180s /usr/share/gocollect/collectors/os.pkg
root      1648  0.0  0.0   4508   696 pts/4    S    15:51   0:00                                                      \_ /bin/sh /usr/share/gocollect/collectors/os.pkg
root      1649  0.0  0.1  30148  6524 pts/4    S    15:51   0:00                                                          \_ python /usr/share/gocollect/collectors/util/lockf.py 120 /var/lib/dpkg/lock /va
root      1650  0.0  0.0   4508   104 pts/4    S    15:51   0:00                                                          \_ /bin/sh /usr/share/gocollect/collectors/os.pkg

ipmi collector modprobes modules when ipmitool is unavailable. 

#!/bin/sh
# vim: set ts=8 sw=4 sts=4 et ai:
# REQUIRES: coreutils(printf tr)
# REQUIRES: kmod(modprobe)
# REQUIRES: sed(sed)
# REQUIRES: util-linux(lscpu)

modprobe ipmi_msghandler
modprobe ipmi_devintf
modprobe ipmi_si

Those modules should not be modprobed when ipmitool is unavailable.

Collect more ipmi stuff

Like:

root@node1# ipmitool mc info | sed -ne '/^Device Available/q;s/[[:blank:]]*:/:/;s/[[:blank:]]*$//p' | tr '\n' , ; echo
Device ID: 32,Device Revision: 1,Firmware Revision: 1.23,IPMI Version: 2.0,Manufacturer ID: 10876,Manufacturer Name: Supermicro,Product ID: 6683 (0x1a1b),Product Name: Unknown (0x1A1B),

Create metadata collector for custom host-specific data

A collector could be: core.meta which reads out all values from /var/lib/gocollect/core.meta (a custom json blob).

It could contain stuff like:

{'roles': ['acceptance', 'loadbalancer'],
 'labels': ['unimportant']}

If we place it here, and not in core.id, we don't get it on the initial register-call. Is that necessary? Probably not, because we want to be able to update it afterwards anyway...

Certificate pinning through HPKP

The server communicates the HPKP policy to the user agent via an HTTP response header field named Public-Key-Pins (or Public-Key-Pins-Report-Only for reporting-only purposes).

The HPKP policy specifies hashes of the subject public key info of one of the certificates in the website's authentic X.509 public key certificate chain (and at least one backup key) in pin-sha256 directives, and a period of time during which the user agent shall enforce public key pinning in max-age directive, optional includeSubDomains directive to include all subdomains (of the domain that sent the header) in pinning policy and optional report-uri directive with URL where to send pinning violation reports. At least one of the public keys of the certificates in the certificate chain needs to match a pinned public key in order for the chain to be considered valid by the user agent.

The central gocollect server shall use the Public-Key-Pins header.
The gocollect daemon shall use that header (store the HPKP values in /var/lib/gocollect for reuse).

See for possible package https://github.com/tam7t/hpkp.
HTTP client can be updated like this:

client := &http.Client{}
...
client.Transport = &http.Transport{
    DialTLS: dialConf.NewDialer(),
}
resp, err := client.Get(...)

Create golang collectors/virt.docker.go collector

See #21.

And for virt_docker.go, we'll want to be able to run a shell-script collector from the internal-collector. (Possibly with extra envvars?) We'll want to call os.pkg but with a different root-path.

(See #39: we'll need to parse json to find paths, so we'll have to use a builtin collector.)

regid or _regid 

#   The {ip4} and {fqdn} parameters are taken from the core.id script.
#   The {_collector} parameter is the collector name/key.
#   The {_regid} parameter is the (unique) identifier obtained from
#   the registry.
#
#   For now, we've decided to do without auth on the collector server.
#   We won't do automatic billing based on the imported values and the
#   source IP is used as an additional identifier.
push_url = https://URL/client/v1/update/{regid}/{_collector}/

Fix problems with (zfs) host storage updating os.storage values on guest 

$ df -B1 --output
Filesystem                  Type        Inodes  IUsed     IFree IUse%    1B-blocks       Used        Avail Use% File Mounted on
data/containers/walter-dev  zfs      368357339 476187 367881152    1% 196089872384 7734820864 188355051520   4% -    /

# df -B1 --output        
Filesystem                     Type        Inodes  IUsed     IFree IUse%    1B-blocks       Used        Avail Use% File Mounted on
data/containers/jessie-builder zfs      368094416 213264 367881152    1% 194461302784 6106251264 188355051520   4% -    /

or

$ df -B1048576 --output
Filesystem                  Type        Inodes  IUsed     IFree IUse% 1M-blocks  Used  Avail Use% File Mounted on
data/containers/walter-dev  zfs      368375899 476187 367899712    1%    187015  7377 179639   4% -    /

# df -B1048576 --output
Filesystem                     Type        Inodes  IUsed     IFree IUse% 1M-blocks  Used  Avail Use% File Mounted on
data/containers/jessie-builder zfs      368112952 213264 367899688    1%    185462  5824 179639   4% -    /

The blocks and the inodes keep mutating. We don't want frequent updates. Can we detect this and drop the values?

core.meta relative path doesn't work

When using a relative config path, commit bdad644 doesn't work as promised.

# gocollect --config=etc/gocollect.conf -sk core.meta
(local results)

# gocollect --config=$(pwd)/etc/gocollect.conf -sk core.meta
(results from relative path)

This is because of the daemonize chdir("/").

dpkg failure during regular upgrade from 0.5.0-1 to 0.5.0-2 on trusty 

Unpacking gocollect-osso (0.5.0-2~ubu14.04) over (0.5.0-1~ubu14.04) ...
Selecting previously unselected package linux-image-4.4.0-62-generic.
Preparing to unpack .../linux-image-4.4.0-62-generic_4.4.0-62.83~14.04.1_amd64.deb ...
Done.
...
Preparing to unpack .../linux-headers-generic-lts-xenial_4.4.0.62.48_amd64.deb ...
Unpacking linux-headers-generic-lts-xenial (4.4.0.62.48) over (4.4.0.59.46) ...
Processing triggers for ureadahead (0.100.0-16) ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
gocollect: unrecognized service
E: Sub-process /usr/bin/dpkg exited unexpectedly

Unrecognised service failure in post-install trigger? Causing install to fail?

# service gocollect status
gocollect: unrecognized service
# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

iU  gocollect                           0.5.0-2~ubu14.04                  amd64        Collect rarely changing system info for server mapping.
iU  gocollect-osso                      0.5.0-2~ubu14.04                  all          OSSO-specific configuration files for GoCollect

dpkg --reconfigure -a fixed things...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.