filippo125 / gstatus Goto Github PK

                               GSTATUS
                               =======

Overview
========
A gluster trusted pool (aka cluster), consists of several key components; 
nodes, volumes and bricks. In glusterfs 3.4/3.5, there isn't a single command
that can provide an overview of the cluster's health. This means that admins
currently assess the cluster health by looking at several commands to piece
together a picture of the cluster's state.

This isn't ideal - so 'gstatus' is an attempt to provide an easy to use,
highlevel view of a cluster's health through a single command. The tool gathers
information by issuing gluster commands, to build an object model covering
Nodes, Volumes and Bricks. With this data in place, checks are performed across
this meta data and errors reported back to the user. In later releases, this
data could be analysed in different ways to add further checks, incorporating
deployment best practices, freespace triggers etc.

Pre-Requisites
==============
+ glusterfs 3.4 or above
  The tool issues commands to gluster using the --xml parameter. Scraping the
  output is another possibility, but by using the xml option, the program 
  should be more stable and more easily extendable
  
+ volume types
  the initial release of gstatus supports distributed and 
  distributed-replicated volumes ONLY


Dependencies
============
- python 2.6 or above
- gluster CLI
- gluster version 3.4 or above


Installation
============
Installing the tool can be done in two ways, through python-seuptools or simply
by running the main script (gstatus.py) directly.

* Using python-setuptools
1. To install using this method extract the archive file
   - tar xvzf gstatus.tar.gz
2. cd to the gstatus directory
3. run the setup.py program
   - python setup.py install
4. invoke the script
   gstatus <options>

This will install a hook for gstatus in /usr/bin so running the script doesn't
need the .py extension.

* running from the archive
1. Extract the archive file
   - tar xvzf gstatus.tar.gz
2. cd to the gstatus directory
3. invoke the script using
   ./gstatus.py <options>

Upgrade Notes
=============
1. If you're upgrading via rpm, first remove the existing version and then install
the new one


Volume States
=============
A gluster volume is made from individual filesystems (gluster bricks) across 
multiple nodes. From an end/user of application perspective, this complexity
is abstracted but the status of individual bricks does have a bearing on the
data availability of the volume. For example, even without replication the
loss of a single brick in the volume will not cause the volume itself to be
unavilable, but instead manifest as inaccessible files in the filesystem.
 
To help understand the impact a brick or node outage can have on data access,
gstatus introduces some additional volume status descriptions.

UP
Volume is started and available, all bricks are up

UP(Degraded)
This state is specific to replicated volumes, where at least one brick is down
within a replica set. Data is still 100% available due to the alternate
replica(s),  but the resilience of the volume to further failures within the
same replica set flags this volume as 'degraded'. 

UP(Partial)
Effectively this means that allthough some bricks in the volume are online, 
there are others that are down to a point where areas of the filesystem will be
missing. For a distrubuted volume, this state is seen if ANY brick is down, 
whereas for a replicated volume a complete replica set needs to be down before
the volume state transitions to PARTIAL. 

DOWN
Bricks are down, or the volume has not yet been started 


Running the tool
================

Running gstatus with a -h flag shows the following options;

[root@gfs-ec1 gstatus]# ./gstatus.py -h
Usage: gstatus.py [options]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -s, --state           show highlevel health of the cluster
  -v, --volume          volume info (default is ALL, or supply a volume name)
  -b, --backlog         Look deeper at self heal state
  -a, --all             show all cluster information (-s with -v)
  -u UNITS, --units=UNITS
                        display capacity units in DECimal or BINary format (GB
                        vs GiB)
  -l, --layout          show brick layout when used with -v, or -a
  -o OUTPUT_MODE, --output-mode=OUTPUT_MODE
                        produce output in different formats - json, keyvalue
                        or console(default)
  -D, --debug           turn on debug mode
  -w, --without-progress
                        turn off progress updates to user during data
                        gathering
  -t TIMEOUT, --timeout=TIMEOUT
                        gluster command timeout value (secs)



* Understanding the output

- Simply invoking gstatus without any options, will give a 3 line summary of
  the cluster. This will tell you overall health, capacity and gluster version.
  If there are issues detected in the cluster, the UNHEALTHY state will be 
  suffixed by a number in brackets - the number denotes the number of errors
  detected within the cluster.

- Using -a will show high level cluster elements, as well as volume information
  You'll notice  that there are a number of what appear to be fractions listed
  against items like Nodes, Bricks etc.

  These fractions actually show you the current 'up' state against the 
  potential up state. So a 'Node' field that shows 2/4, actually means the
  cluster has 4 nodes, but only 2 are in an up state.

  e.g.
[root@rhs1-1 gstatus-tests]# gstatus
   
      Status: HEALTHY           Capacity: 32.00 GiB(raw bricks)
   Glusterfs: 3.4.0.59rhs                 10.00 GiB(raw used)
  OverCommit: Yes                         48.00 GiB(usable from volumes)

[root@rhs1-1 gstatus-tests]# gstatus -a
 
      Status: HEALTHY           Capacity: 32.00 GiB(raw bricks)
   Glusterfs: 3.4.0.59rhs                 10.00 GiB(raw used)
  OverCommit: Yes                         48.00 GiB(usable from volumes)

   Nodes    :  5/ 5		Volumes:  2 Up
   Self Heal:  5/ 4		          0 Up(Degraded)
   Bricks   :  8/ 8		          0 Up(Partial)
   Clients  :     5               0 Down

<snip>

- Capacity information is derived from the brick information taken from a 
  'vol status detail' command. The accuracy of this number therefore depends
  on the nodes/bricks being all up - elements missing from the configuration
  will simply be missing from the calculation.

- A 'flag' called 'OverCommit' is provided that indicates whether a brick is
  used by multiple volumes. Although technically valid, this exposes the 
  user to capacity conflicts across different volumes when quota's are not used.

- running the tool with the -l flag (with -a, or -v) will show a line diagram
  depicting the brick layout within the volume. This feature was is inspired 
  by work of Fred van Zwieten and Niels De Vos in the 'lsgvt' project.

TIP: To keep the tool responsive, gstatus will not perform a vol heal <vol> info
command - since this can take time. However, if you suspect self heal is active
you can use the -b setting to check for a backlog of file heal operations.


The examples directory included in the archive contains a number of scenarios
that show the behaviour of the script under different error conditions.

 
Tested on
=========
- 4 node vm cluster, running Red Hat Storage 2.1 (RHEL 6.4, glusterfs 3.4)
- 4 node vm cluster, based on RHEL 6.5 with upstream glusterfs 3.5 beta 3
- 6 node vm cluster, based on CentOS 7.1, glusterfs 3.7 beta2


Known Issues
============

1) Normally every node will be capacity nodes. However, when nodes are added to
   the cluster for quorum purposes, the Self Heal field shows more nodes with Self 
   Heal active than needed. This is due to the arbitration nodes also running the 
   self head daemon.