risuorg / risu Goto Github PK

Automation Troubleshooting Framework to validate and report configuration, software installed, etc with bash, python, and your language of choice.

Home Page: https://risuorg.github.io

License: GNU General Public License v3.0

Shell 41.77% Python 47.82% HTML 9.78% Go 0.33% Dockerfile 0.30%

python troubleshooting bash automation support shell compliance security-audit linux hardening

risu's People

Contributors

Stargazers

Watchers

risu's Issues

Add sorting to the output

When we execute citellus.py -l we get unsorted output which is not desired since we can have multiple checks that we need to group together for example sriov.

Report on OOM's ?

Should we report on OOM's as indicators of other issues?

Finding plugins on just being executable could cause problems

When unpacking tarball of system snapshot or sosreport on a shared system for collab, it is common to need a chmod -R +rwx to enable access to other users, in this case, citellus finds lot of executable files which tries to execute (which is also dangerous).

I would suggest creating a header in the executable so both conditions should be covered:

file be executable
header containing # CITELLUS_PLUGIN or something like that

This should be compatible in bash, python, etc

If +1 I'll code this

Re-write the plugins

Yet again a new way of running the checks. To be compatible with containers and composable rules.

The idea is to run the independent checks per component/service.

An example workflow for mysqld tuning:

Check the logs for the mysqld running
If the previous check is true, then execute the tuning parameters check.

Right now the workflow is:

Check the node type
Run all the checks against that node.

Write python template

Prepare python template in https://github.com/zerodayz/citellus/tree/master/templates

tox failed with "ValueError: Can't find the JSON-Schema-Test-Suite directory. Set the 'JSON_SCHEMA_TEST_SUITE' environment variable or run the tests from alongside a checkout of the suite."

Hi,

I try to run tox under the project directory, it failed with:

ValueError: Can't find the JSON-Schema-Test-Suite directory. Set the 'JSON_SCHEMA_TEST_SUITE' environment variable or run the tests from alongside a checkout of the suite.

Could you please help to check it? Thanks!

--help doesn't work Include help message about commands supported

[user@host folder]$ ~/citellus/citellus --help
unknown option: --help

reuse code by defining functions

For example:

write a function that recursively imports 'functions.sh'
so ceilometer folder tests imports
if it's stored in /home/iranzo/GSS/DEVEL/citellus/citellus/plugins/openstack/ceilometer
will do import ceilometer/functions.sh and openstack/functions.sh and plugins/functions.sh
and on the template, indicate an import stub
that will take care of this

See how to do that for example for python plugins (as wrapper don't uses this)

Set a naming convention for exported function/vars

Like OSP_CEIL_$VAR
or OSP_$VAR

If running 'live', first argument is plugin folder

Galera, control open files and connections

Enable plugins to be executed in 'DEBUG' via Environment variable $CITELLUS_DEBUG

Implement tests remaining from ENG discussion

Logs in production set to debug permanently, leading to massive log growth and filesystem filling up
Root filesystem full (or near full)
No database backups
NTPd not configured/not working, leading to clock skew that causes RabbitMQ to not function properly
No async updates applied to OSP7 environment since install, meaning tons of critical issues left unpatched
Fencing not configured, meaning if there is a failure, there is no automatic recovery
Cinder deployed using LVM as a backend, which is not production ready/supported
Telemetry data loaded to MongoDB with no expiration, causing MongoDB to fill up and become very large
Keystone tokens not pruned, leading to very large token table

Travis-CI with Gerrit

configure travis-ci to run tests and verify code in gerrit

Exclude filter not working?

When using this command line, I would expect filters which contain sriov and clock in the path to be excluded:

$ ~/citellus/citellus/citellus.py .  -x sriov -x clock|grep -v skipped|grep -v okay
INFO:citellus:using default plugin path
_________ .__  __         .__  .__                
\_   ___ \|__|/  |_  ____ |  | |  |  __ __  ______
/    \  \/|  \   __\/ __ \|  | |  | |  |  \/  ___/
\     \___|  ||  | \  ___/|  |_|  |_|  |  /\___ \ 
 \______  /__||__|  \___  >____/____/____//____  >
        \/              \/                     \/ 
found #38 tests at 
mode: fs snapshot .
# /home/remote/piranzo/citellus/citellus/plugins/openstack/network/sriov.sh: failed
    virtual function is disabled
    vfio_iommu module is not loaded
    missing intel_iommu=on on cmdline
    missing iommu=pt on cmdline
# /home/remote/piranzo/citellus/citellus/plugins/openstack/rabbitmq/rpc_issues.sh: failed
    /var/log/neutron/metadata-agent.log (917 times)
    /var/log/nova/nova-compute.log (10 times)
# /home/remote/piranzo/citellus/citellus/plugins/openstack/traceback.sh: failed
    /var/log/ceilometer/compute.log (757 times)
    /var/log/neutron/metadata-agent.log (3 times)
    /var/log/nova/nova-compute.log (43 times)
# /home/remote/piranzo/citellus/citellus/plugins/system/clock-0-ntp-services.sh: failed
    no chrony service is active
# /home/remote/piranzo/citellus/citellus/plugins/system/clock-1-chrony.sh: failed
    no chrony service is active

But I see that the plugins are executed and status is reported

HWREQ

Hi,
We need to think in a different way to check HW requirements

Most of the shell script is the same but some values and the url's with the recomendations, so they should define some values, and then use an external script to do the actual checking.

This will reduce the lines of codes to mantain/patch in case of error.

Something like for example

hwreq.sh:

MEMTOTAL=6000000
DISKTOTAL=100000

checkhw.sh $MEMTOTAL $DISKTOTAL

Where checkhw.sh contains the actual code of HWREQ and the parameters passed define the values to compare against

Description in citellus.py is empty

Abillity to execute the specified plugins folders in CLI

citellus.py -l plugins/system plugins/pacemaker

Move citellus into 'citellus' subdir?

Usually program has a folder wiith the documentation, etc and another one for the program or python libraries.

Should we move citellus and related tools into citellus folder inside the repo and leave in the root the *.md and others required to be there like tox.ini ?

steal cluster check as plugin

Python script to check several values on cluster area
https://github.com/OndrejHome/sd-cluster-check/blob/master/sd-cluster-check

Run against a live system

Most of the code is based for running against sosreport, but for most of the files, a dictionary could be executed to convert those 'sosreport' paths to 'live paths' or to instead run sosreport on the system to a temporary folder and report from there.

Identical pieces of code should be on a common folder and symlinked

Scripts like cronjob contains same values except from URL, this should be simplified in a similar way to the HW REQ

Change the variables names to be unique

it is good practice to make sure we don't shadow the system variables to prepend a unique prefix.

Unique prefix can be CITELLUS_

Nova start instances on boot check

Convert Sergii script into plugin

#!/bin/bash

# check if nova is configured to resume guests power state at hypervisor startup

LIBVIRTCONF=/etc/sysconfig/libvirt-guests
NOVACONF=/etc/nova/nova.conf
NOVASETTING="^resume_guests_state_on_host_boot"

LIBVIRTBOOT=$(grep ^ON_BOOT $LIBVIRTCONF | awk -F "=" '{print $2}' | sed 's/ //')
LIBVIRTOFF=$(grep ^ON_SHUTDOWN $LIBVIRTCONF | awk -F "=" '{print $2}' | sed 's/ //')
NOVASTRING=$(awk '/\[DEFAULT\]/,/\[api_database\]/' $NOVACONF | grep $NOVASETTING | awk -F "=" '{print $2}' | sed 's/ //')

cat << EOF

ON_BOOT=$LIBVIRTBOOT
ON_SHUTDOWN=$LIBVIRTOFF

resume_guests_state_on_host_boot=$NOVASTRING

EOF

if [[ "$LIBVIRTBOOT" == "ignore" && "$LIBVIRTOFF" == "shutdown" && "$NOVASTRING" == "true" ]]; then
  echo "$(tput setaf 2)>>>$(tput sgr0) compute node is configured to restore guests state at startup $(tput setaf 2)<<<$(tput sgr0)"
  exit 0
else
  echo "$(tput setaf 1)>>>$(tput sgr0) compute node is NOT configured to restore guests state at startup $(tput setaf 1)<<<$(tput sgr0)"
  exit 1
fi

ceph recommended values?

"pool_name":"test"
"min_size":1 //<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Needs to be increased to 2
"pg_num":100 //<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Needs to be a power of two

Should we perform a check for hints like this?

create template plugin with basic skel to be copied and used to create new ones

Providing an empty skeleton of a plugin, with required fields and comments could make it easier for new developers.

ERROR: unknown environment 'py35'

When running 'tox' on a RHEL 7 system, python-tox from 'epel' complains about py35

Consider #citellus on Freenode

Consider providing irc channel like many other projects and document it across the program

clock-1-chrony.sh: failed 'syntax error'

/home/remote/piranzo/citellus/citellus/plugins/system/clock-1-chrony.sh: failed

clock offset is 
(standard_in) 1: syntax error

Write a python wrapper for Shell files unit testing

Write a wrapper plugin in tests/ that can search for .sh tests with patterns:

test_$COMPONENT_fail.sh
test_$COMPONENT_pass.sh
test_$COMPONENT_skip.sh

that take care of creating fake files in a fake folder that is passed to citellus for checking that results are expected ones

bash vs python framework execution differences

bash marks correctly debug.sh as failed, when python okay.

[root@undercloud-0 citellus]# ./citellus /root/sosreport-controller-1.localdomain-20170705201135/ plugins/openstack/
...
citellus: found 8 tests
mode: fs snapshot /root/sosreport-controller-1.localdomain-20170705201135
# plugins/openstack/crontab_heat_stack-purge.sh: okay 
# plugins/openstack/crontab_keystone_cleanup.sh: okay 
# plugins/openstack/debug.sh: failed 
    disabled in /etc/ceilometer/ceilometer.conf
    disabled in /etc/cinder/cinder.conf
    disabled in /etc/cinder/rootwrap.conf
    disabled in /etc/glance/glance-api.conf
    disabled in /etc/glance/glance-cache.conf
    disabled in /etc/glance/glance-glare.conf
    disabled in /etc/glance/glance-registry.conf
    disabled in /etc/glance/glance-scrubber.conf
    disabled in /etc/glance/glance-swift.conf
    disabled in /etc/heat/heat.conf
    disabled in /etc/keystone/keystone.conf
    disabled in /etc/keystone/logging.conf
    disabled in /etc/neutron/dnsmasq-neutron.conf
    disabled in /etc/neutron/neutron.conf
    disabled in /etc/neutron/neutron_lbaas.conf
    disabled in /etc/neutron/rootwrap.conf
    disabled in /etc/neutron/services_lbaas.conf
    disabled in /etc/nova/nova.conf
    disabled in /etc/nova/rootwrap.conf
    disabled in /etc/puppet/auth.conf
    disabled in /etc/puppet/puppet.conf
    disabled in /etc/sahara/rootwrap.conf
    disabled in /etc/sahara/sahara.conf
    disabled in /etc/selinux/semanage.conf
    disabled in /etc/swift/account-server.conf
    disabled in /etc/swift/container-reconciler.conf
    disabled in /etc/swift/container-server.conf
    disabled in /etc/swift/object-expirer.conf
    disabled in /etc/swift/object-server.conf
    disabled in /etc/swift/proxy-server.conf
    disabled in /etc/swift/swift.conf
# plugins/openstack/hardware_memory_recommendations.sh: failed 
    memory is lower than 16gb ram
# plugins/openstack/keystone_cleanup_last-run.sh: okay 
    2017-07-05 00:02:22.458
# plugins/openstack/keystone_cleanup_runs.sh: okay 
    1
# plugins/openstack/mysql_keystone_tokendb.sh: skipped 
# plugins/openstack/version.sh: okay 
    mitaka

[root@undercloud-0 citellus]# python ./citellus.py /root/sosreport-controller-1.localdomain-20170705201135/ plugins/openstack/
...
found 8 tests at plugins/openstack/
mode: fs snapshot False
# plugins/openstack/keystone_cleanup_runs.sh: skipped
# plugins/openstack/crontab_keystone_cleanup.sh: skipped
# plugins/openstack/hardware_memory_recommendations.sh: skipped
# plugins/openstack/mysql_keystone_tokendb.sh: skipped
# plugins/openstack/keystone_cleanup_last-run.sh: skipped
# plugins/openstack/version.sh: okay
# plugins/openstack/crontab_heat_stack-purge.sh: skipped
# plugins/openstack/debug.sh: okay

Fix ceilometer expiration.sh not to run on compute

Fix cinder lvm.sh not to run on compute

Citellus should export LANG=C to avoid problems in greps, etc

citellus: found 81468 tests
mode: fs snapshot $folder/

./$folder/boot/grub2/grub.cfg: ESC[33mskipped ESC[m^O

./grub.cfg: línea 60: terminal_output: no se encontró la orden
./grub.cfg: línea 87: menuentry: no se encontró la orden
insmod: ERROR: could not load module efi_gop: No such file or directory
insmod: ERROR: could not load module efi_uga: No such file or directory
insmod: ERROR: could not load module ieee1275_fb: No such file or directory
insmod: ERROR: could not load module vbe: No such file or directory
insmod: ERROR: could not load module vga: No such file or directory
insmod: ERROR: could not load module video_bochs: No such file or directory
insmod: ERROR: could not load module video_cirrus: No such file or directory
insmod: ERROR: could not load module gzio: No such file or directory
insmod: ERROR: could not load module part_msdos: No such file or directory
insmod: ERROR: could not load module xfs: No such file or directory
./grub.cfg: línea 97: search: no se encontró la orden
./grub.cfg: línea 99: linux16: no se encontró la orden
./grub.cfg: línea 100: initrd16: no se encontró la orden
./grub.cfg: línea 101: error sintáctico cerca del elemento inesperado `}'
./grub.cfg: línea 101: `}'

Citellus, when running in snapshot mode, takes the argument as the place to find plugins

When executed against sosreport, it tries to find plugins to run for all the files in sosreport:

<snip>

citellus: found 81468 tests
mode: fs snapshot $folder/
# ./$folder/boot/grub2/grub.cfg: ESC[33mskipped ESC[m^O
    ./grub.cfg: línea 60: terminal_output: no se encontró la orden
    ./grub.cfg: línea 87: menuentry: no se encontró la orden
    insmod: ERROR: could not load module efi_gop: No such file or directory
    insmod: ERROR: could not load module efi_uga: No such file or directory
    insmod: ERROR: could not load module ieee1275_fb: No such file or directory
    insmod: ERROR: could not load module vbe: No such file or directory
    insmod: ERROR: could not load module vga: No such file or directory
    insmod: ERROR: could not load module video_bochs: No such file or directory
    insmod: ERROR: could not load module video_cirrus: No such file or directory
    insmod: ERROR: could not load module gzio: No such file or directory
    insmod: ERROR: could not load module part_msdos: No such file or directory
    insmod: ERROR: could not load module xfs: No such file or directory
    ./grub.cfg: línea 97: search: no se encontró la orden
    ./grub.cfg: línea 99: linux16: no se encontró la orden
    ./grub.cfg: línea 100: initrd16: no se encontró la orden
    ./grub.cfg: línea 101: error sintáctico cerca del elemento inesperado `}'
    ./grub.cfg: línea 101: `}'

Reduce plugins printed to the ones that passes filters

Plugins reported as:

LOG.debug(msg=_('Found plugins: %s') % plugins

Are printed before filters are applied.

It will be cleaner to output result after filterin and filterout are executed so only the ones to be executed are reported

Remove legacy bash wrapper

Hi,
As the bash wrapper is falling behind on features, we should remove it to make it easier for users and update relevant documentation about using 'citellus.py' one

reporting image_size_cap in glance lower than XX Gb ?

Consider gerrithub.io for reviews

Log in with github and later we can setup the required configuration for end users so they can submit automatically at gerrithub

Move the execution of the code from sequential to parallel

$subject

Update project details to not list just openstack but generic tests

ceph-status Error initializing cluster client: Error('error calling conf_read_file: errno EINVAL',)

/home/remote/piranzo/citellus/citellus/plugins/openstack/ceph/ceph-status.sh: failed

Error initializing cluster client: Error('error calling conf_read_file: errno EINVAL',)

problems locating logs in sosreport

The plugins/system/kernel_panic.sh plugin is looking for:

 .../sos_commands/logs/journalctl_--no-pager_--boot

But a recent sos report actually contains:

journalctl_--all_--this-boot_--no-pager
journalctl_--all_--this-boot_--no-pager_-o_verbose

Is this an anomaly? Or do we need to update the filename in citellus?

Horizon dashboard 'debug' status is not picked

List of config files is written as:

for i in $(sed -n -r -e 's/^openstack-([a-z])-.$/\1/p' ${CITELLUS_ROOT}/installed-rpms
| sort | uniq); do ls ${CITELLUS_ROOT}/etc/$i/.conf 2>/dev/null | grep '/etc/[^/]/[^/]*.conf';
done)

So it tries to locate /etc/$PROGRAM/$PROGRAM.conf

Horizon instead uses: /etc/openstack-dasbhard/local_settings so it's not reported

when executing against only one script, # ['']: okay script name is not printed.

[root@undercloud-0 citellus]# ./citellus.py /root/sosreport-controller-1.localdomain-20170705201135/ /tmp/test/
_________ .__  __         .__  .__                
\_   ___ \|__|/  |_  ____ |  | |  |  __ __  ______
/    \  \/|  \   __\/ __ \|  | |  | |  |  \/  ___/
\     \___|  ||  | \  ___/|  |_|  |_|  |  /\___ \ 
 \______  /__||__|  \___  >____/____/____//____  >
        \/              \/                     \/ 
found #1 tests at /tmp/test/
mode: fs snapshot /root/sosreport-controller-1.localdomain-20170705201135/
# ['']: okay
[root@undercloud-0 citellus]# chmod u+x /tmp/test/hi2.sh 
[root@undercloud-0 citellus]# ./citellus.py /root/sosreport-controller-1.localdomain-20170705201135/ /tmp/test/
_________ .__  __         .__  .__                
\_   ___ \|__|/  |_  ____ |  | |  |  __ __  ______
/    \  \/|  \   __\/ __ \|  | |  | |  |  \/  ___/
\     \___|  ||  | \  ___/|  |_|  |_|  |  /\___ \ 
 \______  /__||__|  \___  >____/____/____//____  >
        \/              \/                     \/ 
found #2 tests at /tmp/test/
mode: fs snapshot /root/sosreport-controller-1.localdomain-20170705201135/
# ['/hi.sh', '/hi2.sh']: okay

Use releases

https://help.github.com/articles/creating-releases/

Citellus doesn't print the failed rule if its the first one.

If the first rule is failed, its not getting printed out.

# ./citellus.py -l plugins/system/ plugins/pacemaker/

found #12 tests at plugins/system/, plugins/pacemaker/
mode: live
# plugins/pacemaker/fence_device.sh: skipped
# plugins/pacemaker/nodes_number.sh: skipped
# plugins/pacemaker/stonith_enabled.sh: skipped
# plugins/pacemaker/stopped_resources.sh: skipped
# plugins/system/baremetal.sh: failed
    KVM
    
# plugins/system/disk_usage.sh: okay
# plugins/system/hardware_virtualization.sh: okay
# plugins/system/kernel_panic.sh: okay
# plugins/system/rh-release.sh: okay
# plugins/system/selinux_config.sh: okay
# plugins/system/selinux_runtime.sh: okay

# ./citellus.py -l plugins/system/ 

found #7 tests at plugins/system/
mode: live
# plugins/system/disk_usage.sh: okay
# plugins/system/hardware_virtualization.sh: okay
# plugins/system/kernel_panic.sh: okay
# plugins/system/rh-release.sh: okay
# plugins/system/selinux_config.sh: okay
# plugins/system/selinux_runtime.sh: okay

Running citellus against non openstack wrongly identifies node as controller

Hi,
When running citellus against non-openstack sosreport, the if sequence detects the node as controller, but without version.

I would say that we should omit node identification as per openstack role, but still run other tests as we can still spot other issues like, let's say for example hardware issues or missconfigured files, etc that could apply for non-openstack specific files.

Network connectivity test skipped on python

Dumping stdout in case of failure.

Thought we are only dumping stderr. However it seems python framework dumps out everything if check failed.

# /root/citellus/plugins/openstack/systemd_services.sh: failed
openstack-ceilometer-compute.service
openstack-nova-compute.service

  openstack-ceilometer-compute.service                                        loaded    active   failed   OpenStack ceilometer compute agent
  openstack-nova-compute.service                                              loaded    active   failed   OpenStack Nova Compute Server

When listing plugins, Path should not be needed

 iranzo   master  …  citellus  tests  UT-tests  127  ../../citellus/citellus.py --list-plugins
ERROR:citellus:When not running in Live mode, snapshot path is required

This is because the code for checking live or snapshot mode is checked before if we're just listing plugins:

risuorg / risu Goto Github PK

risu's People

Contributors

Stargazers

Watchers

Forkers

risu's Issues

/home/remote/piranzo/citellus/citellus/plugins/system/clock-1-chrony.sh: failed

./$folder/boot/grub2/grub.cfg: ESC[33mskipped ESC[m^O

/home/remote/piranzo/citellus/citellus/plugins/openstack/ceph/ceph-status.sh: failed

Recommend Projects

Recommend Topics

Recommend Org