redhat-cip / hardware Goto Github PK

View Code? Open in Web Editor NEW

46.0 23.0 40.0 2.77 MB

Python module for hardware detection and classification

License: Apache License 2.0

Python 100.00%

hardware's Introduction

hardware

Hardware detection and classification utilities

Homepage: https://github.com/redhat-cip/hardware

Features

detect hardware features of a Linux systems:
- RAID
- hard drives
- IPMI
- network cards
- DMI infos
- memory settings
- processor features
filter hardware according to hardware profiles

Install

Installing from pypi:

pip install -U hardware

Usage

Run the hardware-detect program:

hardware-detect --human

Runtime dependencies

The hardware detection is divided in modules that detects a specific hardware type. Each module have its own dependencies.

Therefore, we cannot enforce installing all the dependencies as some are not relevant regarding a particular hardware type. To avoid a situation where we cannot use/install hardware because of one of those deps, we do prefer let users installing the one they need.

The hardware detection code will ignore all the missing deps and continue, so not installing a deps is not fatal.

Please find bellow the list of dependencies per module:

Areca

cli64 from http://www.areca.com.tw/support/s_linux/linux.htm

Logical disks

hdparm
smartmontools

Networking

ethtool
lldp from http://open-lldp.org/
ibstat if you have infiniband devices from https://www.openfabrics.org/

System

ipmitool from https://sourceforge.net/projects/ipmitool/
pciutils
lshw from http://www.ezix.org/project/wiki/HardwareLiSter

Raid controllers

for HP controllers: hpacucli from http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_d6ebba0f5cd642edace4648b9a
for Dell controllers: megacli from http://www.avagotech.com/docs-and-downloads/raid-controllers/raid-controllers-common-files/8-07-14_MegaCLI.zip

hardware's People

Contributors

Stargazers

Watchers

hardware's Issues

Ubuntu 20.04 LTS lshw not compatible

It looks like the version of lshw shipped with Ubuntu 20.04 (Focal) is not compatible with this library due to the way it outputs JSON now. It's wrapped as an array - which hardware lib doesn't expect.

[
{
  "id" : "thegibson",
  "class" : "system",
  "claimed" : true,
  "description" : "Computer",
  "width" : 64,
  "capabilities" : {
    "smp" : "Symmetric Multi-Processing",
    "vsyscall32" : "32-bit processes"
  },
  "children" : [
    {
      "id" : "core",
      "class" : "bus",
      "claimed" : true,
      "description" : "Motherboard",
      "physid" : "0",
      "children" : [
        {
          "id" : "memory",
          "class" : "memory",
          "claimed" : true,
          "description" : "System memory",
          "physid" : "0",
          "units" : "bytes",
          "size" : 4294967296
        },
        {
          "id" : "cpu",
          "class" : "processor",
          "claimed" : true,
          "product" : "Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz",
          "vendor" : "Intel Corp.",
          "physid" : "1",
          "businfo" : "cpu@0",
          "width" : 64,
          "capabilities" : {
...

This had a downstream effect of breaking Ironic Python Agent built against this/Ubuntu 20.04 (not that it's supported yet).

GPU?

will you also detect and list GPUs?

hardware-detect disk benchmark fails to parse modern fio output

The output of fio parsed by the disk benchmark has changed around 2017: axboe/fio@d694a6a

The disk benchmark code in hardware-detect fails to parse its output.

Please use specific logger

Currently logs from hardware are seen in discoverd logs as

Feb 03 12:05:25 install-server.etest ironic-discoverd[41142]: INFO:root:Reading state from /etc/edeploy/state
Feb 03 12:05:25 install-server.etest ironic-discoverd[41142]: INFO:root:testing compute
Feb 03 12:05:26 install-server.etest ironic-discoverd[41142]: INFO:root:testing control

It's not clear where these logs come from. Use logging.getLogger('name') to get reasonably named logger.

`tox -epep8` fails since commit "a8d10ce detect: Fixing numa cpu_count computation"

Fails are E225 missing whitespace around operator, E303 too many blank lines and E501 line too long.

Should we include pep8 tests in the Travis CI test suite?

Improving the output...

Meanwhile, I can run apt install python3-hardware and then hardware-detect --human but the output

comes with a few error messages that could be easily catched:

/bin/sh: 1: cli64: not found
Info: No Areca controller found
Cannot find megacli on the system
read_smart: Reading S.M.A.R.T information on /dev/sdb
read_smart: Reading S.M.A.R.T information on /dev/sdb with -d ata
read_smart: no device /dev/sdb
read_smart: Reading S.M.A.R.T information on /dev/sda
read_smart: Reading S.M.A.R.T information on /dev/sda with -d ata
read_smart: no device /dev/sda
modprobe: FATAL: Module ipmi_smb not found in directory /lib/modules/6.1.0-5-amd64
Info: Probing ipmi_si failed
Info: Probing ipmi_devintf failed
IANA PEN registry open failed: No such file or directory
Info: No Infiniband device found
IANA PEN registry open failed: No such file or directory
/bin/sh: 1: Syntax error: end of file unexpected
Unable to run hp-conrep: 
[('hpa', 'slots', 'count', '2'),

The one with cli64: not found, Syntax error?, Unable to run...

Mem benchmark fails with Python 3

MEM benchmarking fails with error "a bytes-like object is required, not 'str'"

hardware-detect --benchmark mem
...
Memory Performance: 8 logical CPU to test (ETA: 350 seconds)
Benchmarking memory @1K from CPU 0 for 5 seconds (1 threads)
Traceback (most recent call last):
  File "/usr/bin/hardware-detect", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 981, in main
    bm_mem.mem_perf(hrdw)
  File "/usr/lib/python3.6/site-packages/hardware/benchmark/mem.py", line 152, in mem_perf
    block_size, 1, cpu_nb)
  File "/usr/lib/python3.6/site-packages/hardware/benchmark/mem.py", line 91, in run_sysbench_memory_threaded
    if "transferred" in line:
TypeError: a bytes-like object is required, not 'str'

Original report:
https://bugzilla.redhat.com/show_bug.cgi?id=1753653

A bunch of files in hardware/cardiff/VMs directory make packaging very slow

Hi,

Building python-hardware in fedora is taking quite long (~20-30 mins). While investigating on it i've found that directory:

hardware/cardiff/VMs

contains ~ 77MB (+18000 files) of .hw files apparently containing data.

Are those files used/useful in the package or is it something that may be cleaned?

Best regards,

Alfredo

pylint will never complete

Travis CI alternatives

Travis team announced in this post that their platform is changing model, and it will not be entirely free anymore.
They will still have a free tier that could be enough for this project, and they also announced in another post that open source testing will always be free, but we should still find alternatives, just in case.
For example Github actions can be used to run unit and linting tests for python, but we can explore other solutions as well.

CPU benchmark fails with Python 3

CPU benchmarking fails with error "a bytes-like object is required, not 'str'"

hardware-detect --benchmark cpu

[...]

Traceback (most recent call last):
  File "/bin/hardware-detect", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 1022, in main
    _main(options)
  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 962, in _main
    bm_cpu.cpu_perf(hrdw)
  File "/usr/lib/python3.6/site-packages/hardware/benchmark/cpu.py", line 118, in cpu_perf
    run_sysbench_cpu(hw_lst, max_time, 1, processor_num)
  File "/usr/lib/python3.6/site-packages/hardware/benchmark/cpu.py", line 52, in run_sysbench_cpu
    if "total number of events" in line:
TypeError: a bytes-like object is required, not 'str'

Original report:
https://bugzilla.redhat.com/show_bug.cgi?id=1709663

hardware-detect fails on ppc64le due to KeyError

Vendor ID isn't in the lscpu output:

[root@ibm-p9wr-13 ~]# LANG=en_US.UTF-8 lscpu -x
Architecture:        ppc64le
Byte Order:          Little Endian
CPU(s):              144
On-line CPU(s) mask: ffffffffffffffffffffffffffffffffffff
Thread(s) per core:  4
Core(s) per socket:  18
Socket(s):           2
NUMA node(s):        6
Model:               2.2 (pvr 004e 1202)
Model name:          POWER9, altivec supported
CPU max MHz:         3800.0000
CPU min MHz:         2300.0000
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            10240K
NUMA node0 CPU(s):   ffffffffffffffffff
NUMA node8 CPU(s):   ffffffffffffffffff000000000000000000
NUMA node252 CPU(s): 0
NUMA node253 CPU(s): 0
NUMA node254 CPU(s): 0
NUMA node255 CPU(s): 0

Which causes

[root@ibm-p9wr-13 ~]# hardware-detect
/bin/sh: cli64: command not found
Info: detect_areca: No controller found
Cannot find megacli on the system
read_smart: Reading S.M.A.R.T information on /dev/sdb
read_smart_ata: Found S.M.A.R.T information on /dev/sdb
read_smart: Reading S.M.A.R.T information on /dev/sda
read_smart_ata: Found S.M.A.R.T information on /dev/sda
connect: Connection refused 
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
Traceback (most recent call last):
  File "/usr/bin/hardware-detect", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 997, in main
    _main(options)
  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 924, in _main
    if not detect_system(hrdw):
  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 666, in detect_system
    get_cpus(hw_lst)
  File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 725, in get_cpus
    processor), 'vendor', lscpu['Vendor ID']))
KeyError: 'Vendor ID'
[root@ibm-p9wr-13 ~]#

Exception in network_perf when no network drives detected

I'm using 2feb4b8

I'm seeing the following issue:

(hardware) [centos@TestBifrost inspect-hardware-results]$ hardware-cardiff -I ipmi -p 'results/extra_hardware_kef1i-a00*'


Traceback (most recent call last):
  File "/home/centos/venvs/hardware/bin/hardware-cardiff", line 11, in <module>
    sys.exit(main())
  File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/cardiff.py", line 643, in main
    analyze_data(global_params, pattern, ignore_list, detail)
  File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/cardiff.py", line 202, in analyze_data
    rampup_value, current_dir)
  File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/cardiff.py", line 147, in compare_performance
    detail, rampup_value, current_dir)
  File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/check.py", line 174, in network_perf
    results[system] = Series(series, index=net)
  File "/home/centos/venvs/hardware/lib/python2.7/site-packages/pandas/core/series.py", line 249, in __init__
    .format(val=len(data), ind=len(index)))
ValueError: Length of passed values is 1, index implies 0

The error seems to arise due to a mismatch in length between the index argument and the series argument.

generate.generate fails when a model consisting of a single tuple is passed

Here's a test case to demonstrate the failure:

def test_generate_only_tuple(self):
    tup = ('bar', 'baz')
    model = {
        'foo': tup
    }
    result = generate.generate(model)
    self.assertEqual(len(result), 1)
    self.assertEqual(result[0]['foo'], tup)

The result will contain the model 16387064 times.

Splitting hardware-cardiff into a new project?

The cardfiff module brings two extremely heavy dependencies: numpy and pandas. At least on Red Hat systems, pandas ends up pulling X11 libraries. I've noticed that there are no cross-dependencies between hardware.cardiff and the rest of the library. What if we create a new project and move it there?

RAID controllers

there's a few more in the wild that need different tools to talk to them:

find them listed inside:

https://github.com/alexmyczko/autoexec.bat/blob/master/xymon/raidhw

Publish new release with python3 support

In fedora, it's requested to build python3 versions however latest release 0.18 does not support it. I've seen some commits related to python3, could we get a new release which supports it?

Adding Riccardo to collaborators?

Hi folks!

This is a formal proposal to give Riccardo Pittau aka @elfosardo commit rights. Riccardo is one of the most active (probably the most active) contributors in terms of both code and review.

What do you think? @tbreeds @ErwanAliasr1?

PEP8 style change

Currently the allowed line length for tox tests is 160 characters, which is a bit too much according to standard PEP8 convention.
We should remove the max-line-length option from tox.ini and change the code accordingly.
It's always possible to have special cases and exclude checks to favour readability, for example using # noqa

This project's maintenance

Hi @ErwanAliasr1 @fredericlepied,

Sorry for using issue tracker for question, but I'd like to (publicly) ask about the state of maintenance of this library. The thing is, we still rely on it in TripleO, but according to #42 (comment) it essentially not maintained, and pull requests have not been merged since June. The last release is from year 2016.
Do you need help with maintaining the project? I can offer at least basic help with sorting out trivial pull requests and bug reports, and making releases. If you wish so, we can discuss transferring it to the OpenStack git/gerrit infrastructure.

Cheers,
Dmitry

/cc @tbreeds

Module ipaddr is not python 3.x compatible

hardware-detect fails with python3

Running hardware-detect with python3 fails with following error:

Traceback (most recent call last):
File "/bin/hardware-detect", line 10, in
sys.exit(main())
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 900, in main
_main(options)
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 829, in _main
detect_disks(hrdw)
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 323, in detect_disks
detect_utils.read_SMART(hw_lst, "/dev/%s" % name)
File "/usr/lib/python3.6/site-packages/hardware/detect_utils.py", line 353, in read_SMART
if (line.startswith("Device does not support SMART") or
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

Taking a look into the code, the same issue will be found in several places over detect_utils.py.

Defining behavior if lshw segfaults

As per issue #245 on edeploy, if lshw segaults, we are no more pushing information about NICs and some other components.

In the SpinalStack case, that's a big issue as it breaks upload.py which requires mac addr & so.
In the wrangling case, that will surely have impacts too.

I would suggest stopping everything and report a failure if we don't have any output from lshw.

What do you think ?

Splitting output by parts

I'm in a process of using the output of python hardware to manage a 50K server farm.
One of the feature I'll need is to be able to split the output per parts to ease the conformity checking from the installed part vs a list of possible parts (initial + remplacement parts).
The current output is a single file, I'm wondering if we could output each component in a separate file but keep a single one for compat.

Any thoughts on this ?

get_ddr_timing should use i2ctools

The command used to get the ddr timings (ddr-timings-$ARCH) has been lost in time and it's not really supported anymore [1]
We could migrate the ddr timing detection mechanism to a common easily available tool like decode-dimms, included in the i2c-tools suite.
It provides a nice output that can be easily parsed, here's an example:

# decode-dimms version $Revision$

Memory Serial Presence Detect Decoder
By Philip Edelbrock, Christian Zuckschwerdt, Burkart Lingner,
Jean Delvare, Trent Piepho and others


Decoding EEPROM                                  8-0050           8-0051
Guessing DIMM is in                              bank 1           bank 2

---=== SPD EEPROM Information ===---
EEPROM CRC of bytes 0-125                        OK (0x64DE)
# of bytes written to SDRAM EEPROM               384
Total number of bytes in EEPROM                  512
Fundamental Memory type                          DDR4 SDRAM
SPD Revision                                     1.1
Module Type                                      SO-DIMM
EEPROM CRC of bytes 128-253                      OK (0x2355)

---=== Memory Characteristics ===---
Maximum module speed                             2400 MHz (PC4-19200)
Size                                             16384 MB
Banks x Rows x Columns x Bits                    16 x 16 x 10 x 64
SDRAM Device Width                               8 bits
Ranks                                            2
Rank Mix                                         Symmetrical
AA-RCD-RP-RAS (cycles)                           17-17-17-39
Supported CAS Latencies                          18T, 17T, 16T, 15T, 14T, 13T, 12T, 11T, 10T

---=== Timings at Standard Speeds ===---
AA-RCD-RP-RAS (cycles) as DDR4-2400              17-17-17-39
AA-RCD-RP-RAS (cycles) as DDR4-2133              15-15-15-35
AA-RCD-RP-RAS (cycles) as DDR4-1866              13-13-13-30
AA-RCD-RP-RAS (cycles) as DDR4-1600              11-11-11-26

---=== Timing Parameters ===---
Minimum Cycle Time (tCKmin)                      0.833 ns
Maximum Cycle Time (tCKmax)                      1.600 ns
Minimum CAS Latency Time (tAA)                   13.750 ns
Minimum RAS to CAS Delay (tRCD)                  13.750 ns
Minimum Row Precharge Delay (tRP)                13.750 ns
Minimum Active to Precharge Delay (tRAS)         32.000 ns
Minimum Active to Auto-Refresh Delay (tRC)       45.750 ns
Minimum Recovery Delay (tRFC1)                   350.000 ns
Minimum Recovery Delay (tRFC2)                   260.000 ns
Minimum Recovery Delay (tRFC4)                   160.000 ns
Minimum Four Activate Window Delay (tFAW)        21.000 ns
Minimum Row Active to Row Active Delay (tRRD_S)  3.300 ns
Minimum Row Active to Row Active Delay (tRRD_L)  4.900 ns
Minimum CAS to CAS Delay (tCCD_L)                5.000 ns
Minimum Write Recovery Time (tWR)                15.000 ns
Minimum Write to Read Time (tWTR_S)              2.500 ns
Minimum Write to Read Time (tWTR_L)              7.500 ns

---=== Other Information ===---
Package Type                                     Monolithic
Maximum Activate Count                           Unlimited
Post Package Repair                              One row per bank group
Soft PPR                                         Supported
Module Nominal Voltage                           1.2 V
Thermal Sensor                                   No

---=== Physical Characteristics ===---
Module Height                                    30 mm
Module Thickness                                 2 mm front, 2 mm back
Module Reference Card                            E revision 1

---=== Manufacturer Data ===---
Module Manufacturer                              SK Hynix
DRAM Manufacturer                                SK Hynix
Manufacturing Location Code                      0x01
Manufacturing Date                               2018-W37
Assembly Serial Number                           0x2D37BA35       0x2D37E675
Part Number                                      HMA82GS6AFR8N-UH    


Number of SDRAM DIMMs detected and decoded: 2

[1] https://github.com/redhat-cip/edeploy/blob/master/build/sources/timings.c

Drop Python 2.7 support

The time has come!
Or, well, it's coming very soon!
https://pythonclock.org/

We should start considering, or better actively doing something, about dropping Python 2.7 support.
First step should be removing testing support, then refactoring the code to remove compatibility.
I believe testing can be dropped now or very early after beginning of next year (00:01 January 1st sounds like a great time and the perfect way to celebrate the start of 2020).
We can wait a bit longer for the compatibility break, just in case bugs need to be fixed, and can be done in multiple steps.

Thoughts?

TypeError: cannot use a string pattern on a bytes-like object

Seems to have an issue with Python 3 during the HP SmartArray detection.
I'm running Debian 10, no issue with MegaCLI.

root@debian:~# hardware-detect 
/bin/sh: 1: cli64: not found
Info: detect_areca: No controller found
Traceback (most recent call last):
  File "/usr/local/bin/hardware-detect", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/hardware/detect.py", line 962, in main
    detect_hpa(hrdw)
  File "/usr/local/lib/python3.7/dist-packages/hardware/detect.py", line 82, in detect_hpa
    if not cli.launch():
  File "/usr/local/lib/python3.7/dist-packages/hardware/hpacucli.py", line 216, in launch
    self.process.expect(PROMPT_REGEXP)
  File "/usr/local/lib/python3.7/dist-packages/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "/usr/local/lib/python3.7/dist-packages/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 103, in expect_loop
    idx = self.new_data(incoming)
  File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 29, in new_data
    index = searcher.search(window, len(data))
  File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 293, in search
    match = s.search(buffer, searchstart)
TypeError: cannot use a string pattern on a bytes-like object

root@debian:~# pip show hardware
Name: hardware
Version: 0.22.0
Summary: Hardware detection and classification utilities
Home-page: https://github.com/redhat-cip/hardware
Author: eNovance
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib/python3.7/dist-packages
Requires: ipaddress, Babel, pbr, netaddr, pandas, numpy, six, pexpect
Required-by:

root@debian:~# python -V
Python 3.7.3

hardware-detect cmd infiniband fail

I have this error on hardware-detect

  Info: detect_hpa : controller 0 : The specified controller does not have any physical drives on it.
  Cannot get device pause settings: Operation not supported
  modprobe: FATAL: Module ipmi_smb not found.
  Info: Probing ipmi_si failed
  Info: Probing ipmi_devintf failed
  Traceback (most recent call last):
    File "/usr/bin/hardware-detect", line 10, in <module>
      sys.exit(main())
    File "/usr/lib/python2.7/site-packages/hardware/detect.py", line 814, in main
      _main(options)
    File "/usr/lib/python2.7/site-packages/hardware/detect.py", line 776, in _main
      detect_infiniband(hrdw)
    File "/usr/lib/python2.7/site-packages/hardware/detect.py", line 367, in detect_infiniband
      ib_infos = ib.ib_global_info(card_type)
    File "/usr/lib/python2.7/site-packages/hardware/infiniband.py", line 40, in ib_global_info
      global_info = cmd('ibstat %s -s' % card_drv)
  TypeError: not all arguments converted during string formatting

I notice that old use of cmd return https://github.com/enovance/hardware/blame/65fe16757f22ade7a2739e0297989115c3f5794b/hardware/infiniband.py#L20

  >>> from commands import getoutput as cmd
  >>> cmd("echo foo")
  'foo'

But cmd changed and currently return a tuple https://github.com/enovance/hardware/blob/master/hardware/infiniband.py#L27 (ib_card_drv use cmd)

  >>> from hardware.detect_utils import cmd
  >>> cmd("echo foo")
  (0, 'foo\n')

Same on this lines :

Have you any idea of why this changes doesn't broke the unittest ?

Regards,
Florian

CONTRIBUTING.rst contains an incorrect link for the bugtracker

CONTRIBUTING.rst reads:

Bugs should be filed on Launchpad

However, the link yields 404