redhat-cip / hardware Goto Github PK
View Code? Open in Web Editor NEWPython module for hardware detection and classification
License: Apache License 2.0
Python module for hardware detection and classification
License: Apache License 2.0
Seems to have an issue with Python 3 during the HP SmartArray detection.
I'm running Debian 10, no issue with MegaCLI.
root@debian:~# hardware-detect
/bin/sh: 1: cli64: not found
Info: detect_areca: No controller found
Traceback (most recent call last):
File "/usr/local/bin/hardware-detect", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/hardware/detect.py", line 962, in main
detect_hpa(hrdw)
File "/usr/local/lib/python3.7/dist-packages/hardware/detect.py", line 82, in detect_hpa
if not cli.launch():
File "/usr/local/lib/python3.7/dist-packages/hardware/hpacucli.py", line 216, in launch
self.process.expect(PROMPT_REGEXP)
File "/usr/local/lib/python3.7/dist-packages/pexpect/spawnbase.py", line 341, in expect
timeout, searchwindowsize, async_)
File "/usr/local/lib/python3.7/dist-packages/pexpect/spawnbase.py", line 369, in expect_list
return exp.expect_loop(timeout)
File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 103, in expect_loop
idx = self.new_data(incoming)
File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 29, in new_data
index = searcher.search(window, len(data))
File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 293, in search
match = s.search(buffer, searchstart)
TypeError: cannot use a string pattern on a bytes-like object
root@debian:~# pip show hardware
Name: hardware
Version: 0.22.0
Summary: Hardware detection and classification utilities
Home-page: https://github.com/redhat-cip/hardware
Author: eNovance
Author-email: [email protected]
License: UNKNOWN
Location: /usr/local/lib/python3.7/dist-packages
Requires: ipaddress, Babel, pbr, netaddr, pandas, numpy, six, pexpect
Required-by:
root@debian:~# python -V
Python 3.7.3
The command used to get the ddr timings (ddr-timings-$ARCH) has been lost in time and it's not really supported anymore [1]
We could migrate the ddr timing detection mechanism to a common easily available tool like decode-dimms, included in the i2c-tools suite.
It provides a nice output that can be easily parsed, here's an example:
# decode-dimms version $Revision$
Memory Serial Presence Detect Decoder
By Philip Edelbrock, Christian Zuckschwerdt, Burkart Lingner,
Jean Delvare, Trent Piepho and others
Decoding EEPROM 8-0050 8-0051
Guessing DIMM is in bank 1 bank 2
---=== SPD EEPROM Information ===---
EEPROM CRC of bytes 0-125 OK (0x64DE)
# of bytes written to SDRAM EEPROM 384
Total number of bytes in EEPROM 512
Fundamental Memory type DDR4 SDRAM
SPD Revision 1.1
Module Type SO-DIMM
EEPROM CRC of bytes 128-253 OK (0x2355)
---=== Memory Characteristics ===---
Maximum module speed 2400 MHz (PC4-19200)
Size 16384 MB
Banks x Rows x Columns x Bits 16 x 16 x 10 x 64
SDRAM Device Width 8 bits
Ranks 2
Rank Mix Symmetrical
AA-RCD-RP-RAS (cycles) 17-17-17-39
Supported CAS Latencies 18T, 17T, 16T, 15T, 14T, 13T, 12T, 11T, 10T
---=== Timings at Standard Speeds ===---
AA-RCD-RP-RAS (cycles) as DDR4-2400 17-17-17-39
AA-RCD-RP-RAS (cycles) as DDR4-2133 15-15-15-35
AA-RCD-RP-RAS (cycles) as DDR4-1866 13-13-13-30
AA-RCD-RP-RAS (cycles) as DDR4-1600 11-11-11-26
---=== Timing Parameters ===---
Minimum Cycle Time (tCKmin) 0.833 ns
Maximum Cycle Time (tCKmax) 1.600 ns
Minimum CAS Latency Time (tAA) 13.750 ns
Minimum RAS to CAS Delay (tRCD) 13.750 ns
Minimum Row Precharge Delay (tRP) 13.750 ns
Minimum Active to Precharge Delay (tRAS) 32.000 ns
Minimum Active to Auto-Refresh Delay (tRC) 45.750 ns
Minimum Recovery Delay (tRFC1) 350.000 ns
Minimum Recovery Delay (tRFC2) 260.000 ns
Minimum Recovery Delay (tRFC4) 160.000 ns
Minimum Four Activate Window Delay (tFAW) 21.000 ns
Minimum Row Active to Row Active Delay (tRRD_S) 3.300 ns
Minimum Row Active to Row Active Delay (tRRD_L) 4.900 ns
Minimum CAS to CAS Delay (tCCD_L) 5.000 ns
Minimum Write Recovery Time (tWR) 15.000 ns
Minimum Write to Read Time (tWTR_S) 2.500 ns
Minimum Write to Read Time (tWTR_L) 7.500 ns
---=== Other Information ===---
Package Type Monolithic
Maximum Activate Count Unlimited
Post Package Repair One row per bank group
Soft PPR Supported
Module Nominal Voltage 1.2 V
Thermal Sensor No
---=== Physical Characteristics ===---
Module Height 30 mm
Module Thickness 2 mm front, 2 mm back
Module Reference Card E revision 1
---=== Manufacturer Data ===---
Module Manufacturer SK Hynix
DRAM Manufacturer SK Hynix
Manufacturing Location Code 0x01
Manufacturing Date 2018-W37
Assembly Serial Number 0x2D37BA35 0x2D37E675
Part Number HMA82GS6AFR8N-UH
Number of SDRAM DIMMs detected and decoded: 2
[1] https://github.com/redhat-cip/edeploy/blob/master/build/sources/timings.c
will you also detect and list GPUs?
there's a few more in the wild that need different tools to talk to them:
find them listed inside:
https://github.com/alexmyczko/autoexec.bat/blob/master/xymon/raidhw
Fails are E225 missing whitespace around operator, E303 too many blank lines and E501 line too long.
Should we include pep8 tests in the Travis CI test suite?
Here's a test case to demonstrate the failure:
def test_generate_only_tuple(self):
tup = ('bar', 'baz')
model = {
'foo': tup
}
result = generate.generate(model)
self.assertEqual(len(result), 1)
self.assertEqual(result[0]['foo'], tup)
The result will contain the model 16387064 times.
I'm in a process of using the output of python hardware to manage a 50K server farm.
One of the feature I'll need is to be able to split the output per parts to ease the conformity checking from the installed part vs a list of possible parts (initial + remplacement parts).
The current output is a single file, I'm wondering if we could output each component in a separate file but keep a single one for compat.
Any thoughts on this ?
Travis team announced in this post that their platform is changing model, and it will not be entirely free anymore.
They will still have a free tier that could be enough for this project, and they also announced in another post that open source testing will always be free, but we should still find alternatives, just in case.
For example Github actions can be used to run unit and linting tests for python, but we can explore other solutions as well.
The output of fio
parsed by the disk benchmark has changed around 2017: axboe/fio@d694a6a
The disk benchmark code in hardware-detect
fails to parse its output.
The time has come!
Or, well, it's coming very soon!
https://pythonclock.org/
We should start considering, or better actively doing something, about dropping Python 2.7 support.
First step should be removing testing support, then refactoring the code to remove compatibility.
I believe testing can be dropped now or very early after beginning of next year (00:01 January 1st sounds like a great time and the perfect way to celebrate the start of 2020).
We can wait a bit longer for the compatibility break, just in case bugs need to be fixed, and can be done in multiple steps.
Thoughts?
It looks like the version of lshw shipped with Ubuntu 20.04 (Focal) is not compatible with this library due to the way it outputs JSON now. It's wrapped as an array - which hardware lib doesn't expect.
[
{
"id" : "thegibson",
"class" : "system",
"claimed" : true,
"description" : "Computer",
"width" : 64,
"capabilities" : {
"smp" : "Symmetric Multi-Processing",
"vsyscall32" : "32-bit processes"
},
"children" : [
{
"id" : "core",
"class" : "bus",
"claimed" : true,
"description" : "Motherboard",
"physid" : "0",
"children" : [
{
"id" : "memory",
"class" : "memory",
"claimed" : true,
"description" : "System memory",
"physid" : "0",
"units" : "bytes",
"size" : 4294967296
},
{
"id" : "cpu",
"class" : "processor",
"claimed" : true,
"product" : "Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz",
"vendor" : "Intel Corp.",
"physid" : "1",
"businfo" : "cpu@0",
"width" : 64,
"capabilities" : {
...
This had a downstream effect of breaking Ironic Python Agent built against this/Ubuntu 20.04 (not that it's supported yet).
CONTRIBUTING.rst reads:
Bugs should be filed on Launchpad
However, the link yields 404
MEM benchmarking fails with error "a bytes-like object is required, not 'str'"
hardware-detect --benchmark mem
...
Memory Performance: 8 logical CPU to test (ETA: 350 seconds)
Benchmarking memory @1K from CPU 0 for 5 seconds (1 threads)
Traceback (most recent call last):
File "/usr/bin/hardware-detect", line 10, in <module>
sys.exit(main())
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 981, in main
bm_mem.mem_perf(hrdw)
File "/usr/lib/python3.6/site-packages/hardware/benchmark/mem.py", line 152, in mem_perf
block_size, 1, cpu_nb)
File "/usr/lib/python3.6/site-packages/hardware/benchmark/mem.py", line 91, in run_sysbench_memory_threaded
if "transferred" in line:
TypeError: a bytes-like object is required, not 'str'
Original report:
https://bugzilla.redhat.com/show_bug.cgi?id=1753653
Currently the allowed line length for tox tests is 160 characters, which is a bit too much according to standard PEP8 convention.
We should remove the max-line-length
option from tox.ini
and change the code accordingly.
It's always possible to have special cases and exclude checks to favour readability, for example using # noqa
Running hardware-detect with python3 fails with following error:
Traceback (most recent call last):
File "/bin/hardware-detect", line 10, in
sys.exit(main())
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 900, in main
_main(options)
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 829, in _main
detect_disks(hrdw)
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 323, in detect_disks
detect_utils.read_SMART(hw_lst, "/dev/%s" % name)
File "/usr/lib/python3.6/site-packages/hardware/detect_utils.py", line 353, in read_SMART
if (line.startswith("Device does not support SMART") or
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
Taking a look into the code, the same issue will be found in several places over detect_utils.py.
CPU benchmarking fails with error "a bytes-like object is required, not 'str'"
hardware-detect --benchmark cpu
[...]
Traceback (most recent call last):
File "/bin/hardware-detect", line 10, in <module>
sys.exit(main())
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 1022, in main
_main(options)
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 962, in _main
bm_cpu.cpu_perf(hrdw)
File "/usr/lib/python3.6/site-packages/hardware/benchmark/cpu.py", line 118, in cpu_perf
run_sysbench_cpu(hw_lst, max_time, 1, processor_num)
File "/usr/lib/python3.6/site-packages/hardware/benchmark/cpu.py", line 52, in run_sysbench_cpu
if "total number of events" in line:
TypeError: a bytes-like object is required, not 'str'
Original report:
https://bugzilla.redhat.com/show_bug.cgi?id=1709663
Meanwhile, I can run apt install python3-hardware
and then hardware-detect --human
but the output
comes with a few error messages that could be easily catched:
/bin/sh: 1: cli64: not found
Info: No Areca controller found
Cannot find megacli on the system
read_smart: Reading S.M.A.R.T information on /dev/sdb
read_smart: Reading S.M.A.R.T information on /dev/sdb with -d ata
read_smart: no device /dev/sdb
read_smart: Reading S.M.A.R.T information on /dev/sda
read_smart: Reading S.M.A.R.T information on /dev/sda with -d ata
read_smart: no device /dev/sda
modprobe: FATAL: Module ipmi_smb not found in directory /lib/modules/6.1.0-5-amd64
Info: Probing ipmi_si failed
Info: Probing ipmi_devintf failed
IANA PEN registry open failed: No such file or directory
Info: No Infiniband device found
IANA PEN registry open failed: No such file or directory
/bin/sh: 1: Syntax error: end of file unexpected
Unable to run hp-conrep:
[('hpa', 'slots', 'count', '2'),
The one with cli64: not found, Syntax error?, Unable to run...
Hi folks!
This is a formal proposal to give Riccardo Pittau aka @elfosardo commit rights. Riccardo is one of the most active (probably the most active) contributors in terms of both code and review.
What do you think? @tbreeds @ErwanAliasr1?
Hi,
Building python-hardware in fedora is taking quite long (~20-30 mins). While investigating on it i've found that directory:
hardware/cardiff/VMs
contains ~ 77MB (+18000 files) of .hw files apparently containing data.
Are those files used/useful in the package or is it something that may be cleaned?
Best regards,
Alfredo
As per issue #245 on edeploy, if lshw segaults, we are no more pushing information about NICs and some other components.
In the SpinalStack case, that's a big issue as it breaks upload.py which requires mac addr & so.
In the wrangling case, that will surely have impacts too.
I would suggest stopping everything and report a failure if we don't have any output from lshw.
What do you think ?
In fedora, it's requested to build python3 versions however latest release 0.18 does not support it. I've seen some commits related to python3, could we get a new release which supports it?
Currently logs from hardware are seen in discoverd logs as
Feb 03 12:05:25 install-server.etest ironic-discoverd[41142]: INFO:root:Reading state from /etc/edeploy/state
Feb 03 12:05:25 install-server.etest ironic-discoverd[41142]: INFO:root:testing compute
Feb 03 12:05:26 install-server.etest ironic-discoverd[41142]: INFO:root:testing control
It's not clear where these logs come from. Use logging.getLogger('name')
to get reasonably named logger.
Vendor ID isn't in the lscpu output:
[root@ibm-p9wr-13 ~]# LANG=en_US.UTF-8 lscpu -x
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 144
On-line CPU(s) mask: ffffffffffffffffffffffffffffffffffff
Thread(s) per core: 4
Core(s) per socket: 18
Socket(s): 2
NUMA node(s): 6
Model: 2.2 (pvr 004e 1202)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2300.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): ffffffffffffffffff
NUMA node8 CPU(s): ffffffffffffffffff000000000000000000
NUMA node252 CPU(s): 0
NUMA node253 CPU(s): 0
NUMA node254 CPU(s): 0
NUMA node255 CPU(s): 0
Which causes
[root@ibm-p9wr-13 ~]# hardware-detect
/bin/sh: cli64: command not found
Info: detect_areca: No controller found
Cannot find megacli on the system
read_smart: Reading S.M.A.R.T information on /dev/sdb
read_smart_ata: Found S.M.A.R.T information on /dev/sdb
read_smart: Reading S.M.A.R.T information on /dev/sda
read_smart_ata: Found S.M.A.R.T information on /dev/sda
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
connect: Connection refused
Failed to connect to lldpad - clif_open: Connection refused
Traceback (most recent call last):
File "/usr/bin/hardware-detect", line 10, in <module>
sys.exit(main())
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 997, in main
_main(options)
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 924, in _main
if not detect_system(hrdw):
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 666, in detect_system
get_cpus(hw_lst)
File "/usr/lib/python3.6/site-packages/hardware/detect.py", line 725, in get_cpus
processor), 'vendor', lscpu['Vendor ID']))
KeyError: 'Vendor ID'
[root@ibm-p9wr-13 ~]#
The cardfiff module brings two extremely heavy dependencies: numpy and pandas. At least on Red Hat systems, pandas ends up pulling X11 libraries. I've noticed that there are no cross-dependencies between hardware.cardiff and the rest of the library. What if we create a new project and move it there?
I have this error on hardware-detect
Info: detect_hpa : controller 0 : The specified controller does not have any physical drives on it.
Cannot get device pause settings: Operation not supported
modprobe: FATAL: Module ipmi_smb not found.
Info: Probing ipmi_si failed
Info: Probing ipmi_devintf failed
Traceback (most recent call last):
File "/usr/bin/hardware-detect", line 10, in <module>
sys.exit(main())
File "/usr/lib/python2.7/site-packages/hardware/detect.py", line 814, in main
_main(options)
File "/usr/lib/python2.7/site-packages/hardware/detect.py", line 776, in _main
detect_infiniband(hrdw)
File "/usr/lib/python2.7/site-packages/hardware/detect.py", line 367, in detect_infiniband
ib_infos = ib.ib_global_info(card_type)
File "/usr/lib/python2.7/site-packages/hardware/infiniband.py", line 40, in ib_global_info
global_info = cmd('ibstat %s -s' % card_drv)
TypeError: not all arguments converted during string formatting
I notice that old use of cmd return https://github.com/enovance/hardware/blame/65fe16757f22ade7a2739e0297989115c3f5794b/hardware/infiniband.py#L20
>>> from commands import getoutput as cmd
>>> cmd("echo foo")
'foo'
But cmd changed and currently return a tuple https://github.com/enovance/hardware/blob/master/hardware/infiniband.py#L27 (ib_card_drv use cmd)
>>> from hardware.detect_utils import cmd
>>> cmd("echo foo")
(0, 'foo\n')
Same on this lines :
Have you any idea of why this changes doesn't broke the unittest ?
Regards,
Florian
I'm using 2feb4b8
I'm seeing the following issue:
(hardware) [centos@TestBifrost inspect-hardware-results]$ hardware-cardiff -I ipmi -p 'results/extra_hardware_kef1i-a00*'
Traceback (most recent call last):
File "/home/centos/venvs/hardware/bin/hardware-cardiff", line 11, in <module>
sys.exit(main())
File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/cardiff.py", line 643, in main
analyze_data(global_params, pattern, ignore_list, detail)
File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/cardiff.py", line 202, in analyze_data
rampup_value, current_dir)
File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/cardiff.py", line 147, in compare_performance
detail, rampup_value, current_dir)
File "/home/centos/venvs/hardware/lib/python2.7/site-packages/hardware/cardiff/check.py", line 174, in network_perf
results[system] = Series(series, index=net)
File "/home/centos/venvs/hardware/lib/python2.7/site-packages/pandas/core/series.py", line 249, in __init__
.format(val=len(data), ind=len(index)))
ValueError: Length of passed values is 1, index implies 0
The error seems to arise due to a mismatch in length between the index argument and the series argument.
Hi @ErwanAliasr1 @fredericlepied,
Sorry for using issue tracker for question, but I'd like to (publicly) ask about the state of maintenance of this library. The thing is, we still rely on it in TripleO, but according to #42 (comment) it essentially not maintained, and pull requests have not been merged since June. The last release is from year 2016.
Do you need help with maintaining the project? I can offer at least basic help with sorting out trivial pull requests and bug reports, and making releases. If you wish so, we can discuss transferring it to the OpenStack git/gerrit infrastructure.
Cheers,
Dmitry
/cc @tbreeds
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.