Git Product home page Git Product logo

openhtf's Introduction

DISCLAIMER: This is not an official Google product.

OpenHTF

The open-source hardware testing framework.

Build Status Coverage Status Percentage of issues still open Average time to resolve an issue

Overview

OpenHTF is a Python library that provides a set of convenient abstractions designed to remove as much boilerplate as possible from hardware test setup and execution, so test engineers can focus primarily on test logic. It aspires to do so in a lightweight and minimalistic fashion. It is general enough to be useful in a variety of hardware testing scenarios, from the lab bench to the manufacturing floor.

Installing OpenHTF

NOTE: We recommend using virtualenv to create an isolated Python environments for your projects, so as to protect system-wide Python packages the OS depends upon. The installation instructions assume you've already created a virtualenv and activated it if you wish to do so.

Option 1: Installing via 'pip' (recommended)

The most straightforward way to get the openhtf Python package into your Python environment is simply to install it via pip. This will install the most recent production release.

pip install openhtf

Option 2: Installing from Source

If you want to install from source instead (for example, if you want some new feature that hasn't made it to the production release yet), you can download the source code via git or other means, and install the openhtf package into your Python environment using the standard setup.py script.

python setup.py install

Using OpenHTF

The fastest way to get started is to take a look in the examples/ directory, where you'll find sample test scripts and plugs. In addition, many of OpenHTF's modules are fairly well documented inline through the use of docstrings.

Note: some of the examples/ require protocol buffer code to be generated via python setup.py build command. This requires protocol buffer compiler library to be installed (additional instructions).

Nomenclature

OpenHTF uses certain nomenclature internally for several of its core concepts. Some of the more important terms are listed here for clarity.

DUT (Device Under Test)

DUT refers to an individual piece of hardware being evaluated, exercised, or tested.

Test

The top-level abstraction that OpenHTF deals with is the test. A test is just a series of steps performed on/with a DUT, usually along with some data-gathering or measurement steps. In the OpenHTF paradigm, tests are expressed as regular Python programs (.py files) that import and instantiate the 'Test' class from the openhtf module. That way test code is as straightforward as possible to read and write. This also provides for the flexibility to do anything in a test that can normally be done in Python. Superficially, what distinguishes an OpenHTF test from any other Python program is that the OpenHTF test imports the openhtf package, instantiates the Test class, and calls its Execute() function. From there, OpenHTF manages the setup, execution, and teardown of the test, keeps track of anything gathered, and provides a pass/fail result.

At times it may be necessary to disambiguate between different common readings of the word test. In such scenarios we use the following more precise terms:

  • test run - A single start-to-finish execution of a specific test.
  • test recipe - A test definition that may be executed multiple times, each time as a distinct test run.
  • test script - A .py file that contains a test recipe.

Station

Stations capture the notion that a given test ran at some point and may run again. It loosely reflects the idea of physical test stations that process multiple DUTs over time. OpenHTF writes a breadcrumb to the filesystem (in a directory that can be set using the --rundir flag) each time a test runs, and all tests that have the same name are considered to be of the same station. This way the web frontend can display a consolidated list of known tests as a list of stations.

Phase

OpenHTF tests are broken down into logical blocks called phases. Phases are no more than normal Python callables (usually functions) combined with the needed metadata. Writing an OpenHTF test is just a matter of writing a bunch of phase functions and specifying the order in which they should be executed.

Measurement

OpenHTF gathers data about a DUT in the form of measurements. Usually, measurements are declared along with a specification that describes what constitutes a "passing" value. If OpenHTF finishes the test run and one or more measurements were out of that spec, the result of the whole test run will be considered a fail.

Attachment

Sometimes may want to capture additional data that is more complex or free-form than a measurement. An attachment can link arbitrary binary data to a test record, along with an optional MIME type.

Plug

The essence of an OpenHTF test is to interact with a DUT to exercise it in various ways and observe the result. Sometimes this is done by communicating directly with the DUT, and other times it's done by communicating with a piece of test equipment to which the DUT is attached in some way. A plug is a piece of code written to enable OpenHTF to interact with a particular type of hardware, whether that be a DUT itself or a piece of test equipment. OpenHTF comes packaged with a growing collection of useful plugs, but supports the creation of custom plugs as well.

openhtf's People

Contributors

amyxchen avatar arsharma1 avatar cclauss avatar cricdecyan avatar danlipsitt avatar dbhatman avatar dependabot[bot] avatar dieppedalle avatar fahhem avatar glados-verma avatar grybmadsci avatar iharshit15 avatar jamesmtsloan avatar jettisonjoe avatar johnlunney avatar kdsudac avatar kehrazy avatar kenadia avatar kunaljdoshi avatar lalten avatar maddychan avatar rcfie avatar rchen152 avatar sam-bristow avatar shijunliu-google avatar spudmn avatar tan01 avatar tcm0116 avatar trishnaguha avatar wallacbe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

openhtf's Issues

Log a warning on setting a measurement to 'None'.

This might indicate an 'ERROR' outcome in some scenarios. For instance, if we set a measurement using the output of some hardware plug. Logging a warning in this scenario gives us a breadcrumb in case we don't validate the measurement in this way.

Add pass/fail/limit display to new frontend.

Ideally this should include:

  • On-the-fly validation as measurements are set.
  • Overall pass/fail display when a testrun concludes.
  • History of a few past testruns on this station.

measurement.py _AssertValidKey error

With the latest code:

commit bace48a
Merge: 256b4e0 91dacd6
Author: Joe Ethier [email protected]
Date: Tue Jan 5 13:03:44 2016 -0800
Merge pull request #86 from fahhem/tox
Use tox to test multiple environments

Following the measurement.py example to decorate a Measurement:

@measures(Measurement(
'number_widgets').InRange(5, 10).Doc(
'''This phase parameter tracks the number of widgets.'''))

I would get a repeated list of the following error:
File ".../openhtf/openhtf/util/measurements.py", line 315, in getattr
self._AssertValidKey(name)
File ".../openhtf/openhtf/util/measurements.py", line 300, in _AssertValidKey
if name not in self._measurements:
RuntimeError: maximum recursion depth exceeded

Make OpenHTF truly cross-platform.

Right now we're using a modules that are *nix-specific in a couple of places. Notably 'inotify' and 'select'. Users can currently work around this using VMs, but we should try to work natively on Linux, OSX, and Windows if possible.

AttributeError - test_state.running_phase.attachments.update(self.attachments)

See AttributeError running HTF. Also seen with example:

Provide a DUT ID in order to start the test.
test
What's the widget type?
asdf

2015-12-05 00:00:31,850 - ERROR - Failed to execute test.
Traceback (most recent call last):
File "/usr/local/openhtf/openhtf/exe/init.py", line 138, in _ThreadProc
self._ExecuteTest(phase_executor)
File "/usr/local/openhtf/openhtf/exe/init.py", line 164, in _ExecuteTest
for phase_result in phase_executor.ExecutePhases():
File "/usr/local/openhtf/openhtf/exe/phasemanager.py", line 150, in ExecutePhases
result = self._ExecuteOnePhase(self._test_state.pending_phases[0])
File "/usr/local/openhtf/openhtf/exe/phasemanager.py", line 180, in _ExecuteOnePhase
result_wrapper.SetResult(phase_thread.JoinOrDie())
File "/usr/local/venv/local/lib/python2.7/site-packages/contextlib2.py", line 56, in exit
next(self.gen)
File "/usr/local/openhtf/openhtf/exe/phase_data.py", line 154, in RecordPhaseTiming
test_state.running_phase.attachments.update(self.attachments)
AttributeError: 'NoneType' object has no attribute 'update'
ERROR:root:Thread raised an exception: Thread-1
Traceback (most recent call last):
File "/usr/local/openhtf/openhtf/util/threads.py", line 51, in run
self._ThreadProc()
File "/usr/local/openhtf/openhtf/exe/init.py", line 138, in _ThreadProc
self._ExecuteTest(phase_executor)
File "/usr/local/openhtf/openhtf/exe/init.py", line 164, in _ExecuteTest
for phase_result in phase_executor.ExecutePhases():
File "/usr/local/openhtf/openhtf/exe/phasemanager.py", line 150, in ExecutePhases
result = self._ExecuteOnePhase(self._test_state.pending_phases[0])
File "/usr/local/openhtf/openhtf/exe/phasemanager.py", line 180, in _ExecuteOnePhase
result_wrapper.SetResult(phase_thread.JoinOrDie())
File "/usr/local/venv/local/lib/python2.7/site-packages/contextlib2.py", line 56, in exit
next(self.gen)
File "/usr/local/openhtf/openhtf/exe/phase_data.py", line 154, in RecordPhaseTiming
test_state.running_phase.attachments.update(self.attachments)
AttributeError: 'NoneType' object has no attribute 'update'
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in bootstrap_inner
self.run()
File "/usr/local/openhtf/openhtf/util/threads.py", line 51, in run
self._ThreadProc()
File "/usr/local/openhtf/openhtf/exe/__init
.py", line 138, in _ThreadProc
self._ExecuteTest(phase_executor)
File "/usr/local/openhtf/openhtf/exe/init.py", line 164, in _ExecuteTest
for phase_result in phase_executor.ExecutePhases():
File "/usr/local/openhtf/openhtf/exe/phasemanager.py", line 150, in ExecutePhases
result = self._ExecuteOnePhase(self._test_state.pending_phases[0])
File "/usr/local/openhtf/openhtf/exe/phasemanager.py", line 180, in _ExecuteOnePhase
result_wrapper.SetResult(phase_thread.JoinOrDie())
File "/usr/local/venv/local/lib/python2.7/site-packages/contextlib2.py", line 56, in exit
next(self.gen)
File "/usr/local/openhtf/openhtf/exe/phase_data.py", line 154, in RecordPhaseTiming
test_state.running_phase.attachments.update(self.attachments)
AttributeError: 'NoneType' object has no attribute 'update'

Add ability to launch without config file

Add ability to launch without config file. Add ability to pass a required configs through command line.

--config: The OpenHTF configuration file for this tester
(default: '/usr/local/openhtf_client/config/clientfoo.yaml')
--config_value: Allows specifying a configuration key=value on the command line. The format should be --config_value key=value. This value
will override any existing config value at config load time and will be a string;
repeat this option to specify a list of values
(default: '[]')

Device GetSerial() is returning an empty string measurement

Using usb AdbPlug, GetSerial() call is returning empty string measurement. Expected is Device serial number string value. (This is without Cambrionix unit. Device directly connected to host machine on USB port).

@openhtf.TestPhase(timeout_s=360)
@plug(adb=usb.AdbPlug)
@measures(Measurement('Serialnum'))
def DeviceSerialNum(test, adb):
print (type(adb.GetSerial()))
test.measurements.Serialnum = adb.GetSerial()
print 'getserial done'

Remove support for optional measurements.

All declared measurements should be required measurements. If a measurement was unable to be taken, we should probably end the test with the "ERROR" outcome.

...At least, that's one line of reasoning we need to consider.

Eliminate the TestPhaseResult class from phasemanager.py.

I'm changing this from an inline TODO to an issue in our tracker...

# TODO(jethier): Do we really need this to be a tuple?  All we do is check if
# phase_result is an instance of BaseException and set raised_exception based
# on that.  Why not just save whatever we would store in phase_result and then
# do the subclass check when we care?  It's annoying to see
# phase_result.phase_result everywhere.
class TestPhaseResult(collections.namedtuple(
    'TestPhaseResult', ['phase_result', 'raised_exception'])):
  """Result of a phase, and whether it raised an exception or not."""

TestState conflates test state and test outcome

Noticed this when handling output to testrun proto, TestState uses an enum:
State = Enum(
'RUNNING', 'ERROR', 'TIMEOUT', 'ABORTED', 'WAITING', 'FAIL', 'PASS',
'CREATED'
)

This represents not only the test state, but also the outcome (pass/fail) - in face the test state is copied into the 'outcome' field of the test record. This could potentially result in really weird 'outcome' values if something goes wonky with the state of the TestState object ('RUNNING', 'CREATED', etc). Also, 'FAIL', and 'PASS' aren't really analogous to 'CREATED' and 'RUNNING'. I think TestState should have 'COMPLETED' as a state, and the outcome should be tracked separately.

phase_data raises "RuntimeError: Validate must only be called once" with loop=True

To reproduce, add loop=True to test.Execute in hello_world.py and loop it twice:

sudo python hello_world.py --config hello_world.yaml
Provide a DUT ID in order to start the test.
test
What's the widget type?
hi
Provide a DUT ID in order to start the test.
test
INFO:hello_world.py:Hello World!
What's the widget type?
hi
INFO:hello_world.py:Example says: Example Plug Did Stuff!
2015-12-05 15:51:42,619 - ERROR - Failed to execute test.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/init.py", line 138, in _ThreadProc
self._ExecuteTest(phase_executor)
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/init.py", line 164, in _ExecuteTest
for phase_result in phase_executor.ExecutePhases():
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phasemanager.py", line 150, in ExecutePhases
result = self._ExecuteOnePhase(self._test_state.pending_phases[0])
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phasemanager.py", line 180, in _ExecuteOnePhase
result_wrapper.SetResult(phase_thread.JoinOrDie())
File "build/bdist.linux-x86_64/egg/contextlib2.py", line 56, in exit
next(self.gen)
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phase_data.py", line 147, in RecordPhaseTiming
for name, measurement in measurement_map.iteritems()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phase_data.py", line 147, in
for name, measurement in measurement_map.iteritems()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/util/measurements.py", line 152, in Validate
raise RuntimeError('Validate must only be called once')
RuntimeError: Validate must only be called once
ERROR:root:Thread raised an exception: Thread-1
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/util/threads.py", line 51, in run
self._ThreadProc()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/init.py", line 138, in _ThreadProc
self._ExecuteTest(phase_executor)
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/init.py", line 164, in _ExecuteTest
for phase_result in phase_executor.ExecutePhases():
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phasemanager.py", line 150, in ExecutePhases
result = self._ExecuteOnePhase(self._test_state.pending_phases[0])
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phasemanager.py", line 180, in _ExecuteOnePhase
result_wrapper.SetResult(phase_thread.JoinOrDie())
File "build/bdist.linux-x86_64/egg/contextlib2.py", line 56, in exit
next(self.gen)
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phase_data.py", line 147, in RecordPhaseTiming
for name, measurement in measurement_map.iteritems()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phase_data.py", line 147, in
for name, measurement in measurement_map.iteritems()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/util/measurements.py", line 152, in Validate
raise RuntimeError('Validate must only be called once')
RuntimeError: Validate must only be called once
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/util/threads.py", line 51, in run
self._ThreadProc()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/__init
.py", line 138, in _ThreadProc
self._ExecuteTest(phase_executor)
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/init.py", line 164, in _ExecuteTest
for phase_result in phase_executor.ExecutePhases():
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phasemanager.py", line 150, in ExecutePhases
result = self._ExecuteOnePhase(self._test_state.pending_phases[0])
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phasemanager.py", line 180, in _ExecuteOnePhase
result_wrapper.SetResult(phase_thread.JoinOrDie())
File "build/bdist.linux-x86_64/egg/contextlib2.py", line 56, in exit
next(self.gen)
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phase_data.py", line 147, in RecordPhaseTiming
for name, measurement in measurement_map.iteritems()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/exe/phase_data.py", line 147, in
for name, measurement in measurement_map.iteritems()
File "/usr/local/lib/python2.7/dist-packages/openhtf-1.0-py2.7.egg/openhtf/util/measurements.py", line 152, in Validate
raise RuntimeError('Validate must only be called once')
RuntimeError: Validate must only be called once

Audit failure code output

Right now, all we really get is test-level pass/fail. The ability exists to add details, but the framework doesn't do it. It would be nice to get phase-specific info added automatically.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.