Testing ideas for LeBauer projects

Basic ideas for testing Python code:

Coders are responsible for writing their own tests
Tests should use known inputs and verify that expected outputs are created
Tests should actively probe all possible errors to ensure the program fails gracefully
Pytest is a decent framework to use

The point of testing is to improve code quality and reliability. Effective use of other modules and tools can help:

argparse to validate arguments and produce documentation
pylint and flake8 to suggest improvements and find errors
mypy and type hints for static type checking

Hello, World

The first example will use the "Hello, World!" example from Tiny Python Projects. A PDF of the chapter is included in the "hello" directory, but only the finished "hello.py" program is included.

The most basic principle here is that the program is run with known inputs and is tested for expected outputs. Testing which, for instance, simply runs a function or a program but does not bother to verify that the output is correct is not effective.

To begin, the program uses the standard "argparse" module to validate the arguments and produce documentation:

$ ./hello.py -h
usage: hello.py [-h] [-n NAME]

Say hello

optional arguments:
  -h, --help            show this help message and exit
  -n NAME, --name NAME  Name to greet

While most of the Python code I’ve encountered already uses "argparse," I’ve found instances where relying on this module more (e.g., setting "default" values or using the type=argparse.FileType) could improve programs. For examples of some basic "argparse" programs, please see the included "argparse.pdf" chapter and the code here:

https://github.com/kyclark/tiny_python_projects/tree/master/appendix_argparse

The "hello.py" program, for example, has a --name option. Being an option, it has a default value so that the program can run without any arguments:

$ ./hello.py
Hello, World!

And this default can be overridden:

$ ./hello.py --name Universe
Hello, Universe!

Here is the entire source code for the program:

#!/usr/bin/env python3
"""
Author:  Ken Youens-Clark <[email protected]>
Purpose: Say hello
"""

import argparse


# --------------------------------------------------
def get_args():
    """Get the command-line arguments"""

    parser = argparse.ArgumentParser(description='Say hello')
    parser.add_argument('-n', '--name', default='World', help='Name to greet')
    return parser.parse_args()


# --------------------------------------------------
def main():
    """Make a jazz noise here"""

    args = get_args()
    print('Hello, ' + args.name + '!')


# --------------------------------------------------
if __name__ == '__main__':
    main()

The "hello_test.py" is so named so that "pytest" can find the file and run the functions inside it with names staring with "test_":

$ pytest -v
============================= test session starts ==============================
...

hello_test.py::test_exists PASSED                                        [ 25%]
hello_test.py::test_usage PASSED                                         [ 50%]
hello_test.py::test_default PASSED                                       [ 75%]
hello_test.py::test_input PASSED                                         [100%]

============================== 4 passed in 0.32s ===============================

This test file is a basic integration test in that it runs the program as the user would run it. There are no functions in such a simple program to warrant unit tests as they would essentially duplicate the integration test.

Here is the source code for the tests:

#!/usr/bin/env python3
"""tests for hello.py"""

import os
from subprocess import getstatusoutput

prg = './hello.py' (1)


# --------------------------------------------------
def test_exists(): (2)
    """exists"""

    assert os.path.isfile(prg) (3)


# --------------------------------------------------
def test_usage():
    """usage"""

    for flag in ['-h', '--help']: (4)
        rv, out = getstatusoutput(f'{prg} {flag}') (5)
        assert rv == 0 (6)
        assert out.lower().startswith('usage') (7)


# --------------------------------------------------
def test_default():
    """Says 'Hello, World!' by default"""

    rv, out = getstatusoutput(prg) (8)
    assert rv == 0 (9)
    assert out.strip() == 'Hello, World!' (10)


# --------------------------------------------------
def test_input():
    """test for input"""

    for val in ['Universe', 'Multiverse']: (11)
        for option in ['-n', '--name']: (12)
            rv, out = getstatusoutput(f'{prg} {option} {val}') (13)
            assert rv == 0 (14)
            assert out.strip() == f'Hello, {val}!' (15)

prg (program) declared as a global as it is used throughout the program
I usually verify that the expected program exists. I usually run "pytest -xv" where the "-x" flag will stop at the first failing test. Here, if the program doesn’t exist, there’s no point in going further.
The assert statement is the heart of the test. It will throw an exception if the given argument is not True. The os.path.isfile() function is useful for detecting if files exists, e.g., input or output files.
Here we run the program with both "-h" and "--help" flags to be sure the program will produce a "usage" statement.
The subprocess.getstatusoutput() function will execute a command via the OS and return the exit code of the process along with the captured STDOUT/STDERR.
All programs that exit normally should have a return value of 0. A non-zero exit value should always be used to indicate the program encountered a problem or failed to finish normally. Printing usage is a normal request, not an error, so the program ought to return 0 (for "0 errors).
Here we verify that the output (STDOUT) from the program begins with the word "usage." This is not verifying that the entire help documentation is correct, only that the program appears to be well-behaved. Often testing is limited to spot checks and is not exhaustive.
Run the program with no arguments. This is always a good early test. If the program expects arguments, try breaking it or providing too few.
This program has only options and so can run normally with no arguments; therefore the exit code should be 0.
The output should be "Hello, World!" It’s crucial to point out that we test both the exit code and the output of the program are expected values!
When providing input testing values, it’s vital to try more than one. Here we will try to greet two different strings.
Additionally we should verify that the program recognizes both the short and long option names.
We run the program using a constructed command line with the flag name and the option value.
Again the exit code should be 0 as this is valid input.
The output is again verified to be what is expected.

This program is rather simple, and so it’s difficult to try to break it. Still, the above points to basic principles of using "pytest" to positively asserting that the program works as expected.

Word Finder

Let us examine a slightly more complicated example that can highlight the use of both unit and integration tests and also involve effective use of "argparse" and type hinting. The "finder.py" program will find words of a given --len length in one or more given --file arguments:

$ ./finder.py -h
usage: finder.py [-h] [-l int] FILE [FILE ...]

Find words in a file of given length

positional arguments:
  FILE               Input file(s)

optional arguments:
  -h, --help         show this help message and exit
  -l int, --len int  Length of words to find (default: 3)

The user interface will enforce many requirements for the program such as requiring the input file(s):

$ ./finder.py
usage: finder.py [-h] [-l int] FILE [FILE ...]
finder.py: error: the following arguments are required: FILE

Checking that the input files are valid:

$ ./finder.py blargh
usage: finder.py [-h] [-l int] FILE [FILE ...]
finder.py: error: argument FILE: can't open 'blargh': \
[Errno 2] No such file or directory: 'blargh'

Ensuring that the --len argument is greater than 0:

$ ./finder.py -l -4 tests/fox.txt
usage: finder.py [-h] [-l int] FILE [FILE ...]
finder.py: error: --len "-4" must be > 0

This is all handled by either directly by "argparse" or manually during the processing of the arguments:

import argparse
import re
import sys
from itertools import starmap
from functools import partial
from typing import TextIO, NamedTuple, List


class Args(NamedTuple): (1)
    file: List[TextIO]
    length: int


# --------------------------------------------------
def get_args() -> Args: (2)
    """Get command-line arguments"""

    parser = argparse.ArgumentParser(
        description='Find words in a file of given length',
        formatter_class=argparse.ArgumentDefaultsHelpFormatter)

    parser.add_argument('file',
                        help='Input file(s)',
                        metavar='FILE',
                        type=argparse.FileType('rt'), (3)
                        nargs='+') (4)

    parser.add_argument('-l',
                        '--len',
                        help='Length of words to find',
                        metavar='int',
                        type=int,
                        default=3) (5)

    args = parser.parse_args()

    if args.len < 0: (6)
        parser.error(f'--len "{args.len}" must be > 0')

    return Args(args.file, args.len) (7)

This is a class that represents the arguments to the program.
The return value from the function is annotated with the Args class so that mypy can use the type information to validate any code that uses the args.
Any "file" argument must be a readable text file.
We require one or more inputs.
This is an option, so we set a reasonable default value.
Use the "parser.error()" function to manually throw an error.
Return a fully typed, read-only object that represents the arguments.

To find all the words 3 characters long in one file:

$ ./finder.py tests/fox.txt
  1: The
  2: fox
  3: the
  4: dog

To find all the words 5 characters in more than one file:

$ ./finder.py -l 5 tests/*.txt
  1: quick
  2: brown
  3: jumps
  4: human
  5: bands
  6: which
  7: among
  8: earth
  9: equal
 10: which
 11: which
 12: impel

The rest of the program is rather simple:

# --------------------------------------------------
def main() -> None:
    """Make a jazz noise here"""

    args = get_args() (1)

    if words := find_words(args.length, args.file): (2)
        print('\n'.join(
            starmap(lambda i, w: f'{i:3}: {w}', enumerate(words, 1))))
    else:
        sys.exit(f'Found no words of length {args.length}!') (3)


# --------------------------------------------------
def find_words(word_length: int, fhs: List[TextIO]) -> List[str]: (4)
    """Find words in a given text of a certain length"""

    words: List[str] = []
    clean = partial(re.sub, '[^a-zA-Z]', '')
    accept = lambda word: len(word) == word_length

    for fh in fhs:
        for line in fh:
            words.extend(filter(accept, map(clean, line.split())))

    return words

Because of the return type annotation on the "get_args()" function, mypy knows that "args" is of the type Args.
Note the := syntax new to Python 3.8 that allows assignment and evaluation in one step.
A decision to return an error code when no words are found using "sys.exit()".
The type annotations on this signature are complex but useful. Note that List[str] is more informative than the primitive list.

It’s important to note that the code inside "find_words()" could have be placed inside the "main()" function, but then we would not be able to write a unit test. All the unit and integration tests along with test input files live in the "tests" directory:

$ ls tests/
finder_test.py*		fox.txt
finder_unit_test.py	preamble.txt

There is just one function with a unit test in "tests/finder_unit_test.py":

import io
from finder import find_words


# --------------------------------------------------
def test_find_words() -> None:
    """Test find_words"""

    text = lambda: [io.StringIO('The quick brown fox jumps over the lazy dog.')]
    assert find_words(1, text()) == [] (1)
    assert find_words(2, text()) == []
    assert find_words(3, text()) == ['The', 'fox', 'the', 'dog'] (2)
    assert find_words(4, text()) == ['over', 'lazy']
    assert find_words(5, text()) == ['quick', 'brown', 'jumps']
    assert find_words(6, text()) == []

Run the test with parameters we know will return nothing.
Run the test with parameters we know will return something.

In both cases, the test uses known inputs and checks that expected outputs are returned. This should be the bare minimum for any sort of testing.

Note that this function does not raise an exception. If you need to test a function that does raise an exception under certain conditions, see https://docs.pytest.org/en/stable/assert.html#assertions-about-expected-exceptions.

So here is how you should run the tests:

$ python3 -m pytest -xv

Note that the "Makefile" has a "test" target that allows you to type "make test" as a shortcut:

$ make test
python3 -m pytest -xv
============================= test session starts ==============================
...

tests/finder_test.py::test_exists PASSED                                 [ 10%]
tests/finder_test.py::test_usage PASSED                                  [ 20%]
tests/finder_test.py::test_no_args PASSED                                [ 30%]
tests/finder_test.py::test_bad_length PASSED                             [ 40%]
tests/finder_test.py::test_bad_file PASSED                               [ 50%]
tests/finder_test.py::test_fox_3 PASSED                                  [ 60%]
tests/finder_test.py::test_preamble_10 PASSED                            [ 70%]
tests/finder_test.py::test_fox_and_preamble_5 PASSED                     [ 80%]
tests/finder_test.py::test_dies_none_found PASSED                        [ 90%]
tests/finder_unit_test.py::test_find_words PASSED                        [100%]

============================== 10 passed in 0.46s ==============================

As noted in our discussions, I found it a little difficult to get the relative import of the "finder.py" code when the unit test was moved to the "tests" directory when I simply used the pytest command:

"Running pytest with pytest […] instead of python -m pytest […] yields nearly equivalent behaviour, except that the latter will add the current directory to sys.path, which is standard python behavior." — https://docs.pytest.org/en/stable/pythonpath.html#pytest-vs-python-m-pytest

If you would like test coverage information, you can install the coverage module run the following:

$ coverage run -m pytest -xv

The output will be the same as above, but there should now be a ".coverage" directory. You could, for instance, run the "report" action to see how well the tests are covering:

$ coverage report
Name                        Stmts   Miss  Cover
-----------------------------------------------
finder.py                      33     12    64%
tests/finder_test.py           60      0   100%
tests/finder_unit_test.py      10      0   100%
-----------------------------------------------
TOTAL                         103     12    88%

The integration tests in "tests/finder_test.py" start off with basic assertions such as the "finder.py" exists, will create a "usage" statement when asked. Note that tests are run by "pytest" in the order in which they are found in the source code, so the order of function definition is important!

def test_exists():
    """exists"""

    assert os.path.isfile(prg)


def test_usage():
    """usage"""

    for flag in ['-h', '--help']:
        rv, out = getstatusoutput(f'{prg} {flag}')
        assert rv == 0 (1)
        assert out.lower().startswith('usage')

Asking for the usage is not an error, so the return code should be 0.

The next tests start all give bad input to the program and check for error codes. It is crucial to try to break the program and verify that it fails gracefully!

For instance, when run with no arguments:

def test_no_args():
    """Dies on no arguments"""

    rv, out = getstatusoutput(prg)
    assert rv != 0 (1)
    assert out.lower().startswith('usage')

The program correctly reports a non-zero status and also prints a "usage."

The next test gives a bad "--len" value (a negative number):

def test_bad_length():
    """Dies on bad length"""

    bad = random.choice(range(-10, 0))
    rv, out = getstatusoutput(f'{prg} -l {bad} {fox}')
    assert rv != 0
    assert out.lower().startswith('usage')
    assert re.search(f'--len "{bad}" must be > 0', out) (1)

The program returns an error code, prints a "usage" along with a helpful error message that indicates exactly to the user the name of the argument and the offending value.

The next test gives a bad file argument:

def test_bad_file():
    """Dies on bad file"""

    bad = random_string() (1)
    rv, out = getstatusoutput(f'{prg} -l 3 {bad}')
    assert rv != 0
    assert out.lower().startswith('usage')
    assert re.search(f"No such file or directory: '{bad}'", out) (2)

We will use a randomly generated string as the "file" name.
Again, the program errors out and prints useful error messages/help.

Now that we know the program will reject all bad inputs, we can start testing with good inputs:

def test_fox_3():
    """test runs ok"""

    assert os.path.isfile(fox)
    rv, out = getstatusoutput(f'{prg} {fox}') (1)
    assert rv == 0
    lines = list(map(str.strip, out.splitlines()))
    assert len(lines) == 4 (2)
    assert lines == ['1: The', '2: fox', '3: the', '4: dog'] (3)

Run using the known default value for length.
We know there should be 4 lines of output.
Verify the words are correct.

The next test uses a different input file and exercises the long name for the length option:

def test_preamble_10():
    """test runs ok"""

    assert os.path.isfile(preamble)
    rv, out = getstatusoutput(f'{prg} --len 10 {preamble}')
    assert rv == 0
    lines = list(map(str.strip, out.splitlines()))
    assert len(lines) == 1
    assert lines == ['1: separation']

The next test uses multiple input files to ensure the positional arguments are correctly handled:

def test_fox_and_preamble_5():
    """test runs ok"""

    assert os.path.isfile(preamble)
    rv, out = getstatusoutput(f'{prg} --len 10 {preamble} {fox}') (1)
    assert rv == 0
    lines = list(map(str.strip, out.splitlines()))
    assert len(lines) == 1
    assert lines == ['1: separation']

Two positional arguments.

The last test ensures that the program returns an error code when no words can be found. I would not normally consider this to be a error, but for demonstrations purposes I added this test so as to highlight the use, for instance, of sys.exit() in the code to handle this:

def test_dies_none_found():
    """returns error when no words found"""

    rv, out = getstatusoutput(f'{prg} --len 20 {preamble} {fox}')
    assert rv != 0 (1)
    assert out == 'Found no words of length 20!' (2)

Not finding any words is considered an error.
Check that the error message is correct.

Author

Ken Youens-Clark <[email protected]>

kyclark / lebauer-testing Goto Github PK

lebauer-testing's Introduction

Testing ideas for LeBauer projects

Hello, World

Word Finder

Author

lebauer-testing's People

Stargazers

Watchers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent