jelleas / checkpy Goto Github PK

A Python tool for running tests on Python source files. Intended to be used by students whom are taking courses in the Minor Programming at the UvA.

License: MIT License

Python 100.00%

checkpy's People

Contributors

Stargazers

Watchers

Forkers

stgm python3pkg programmeer tomkooij

checkpy's Issues

Require (data) files

Check for existing data files (specified in tests), error if not present?
Try download if possible?

t.passed decorator does not chain

@t.test(0):
def first(test):
    pass

@t.passed(first) 
@t.test(1)
def second(test):
    pass

@t.passed(second) 
@t.test(2)
def second(test):
    pass

You'll receive:
AttributeError: 'NoneType' object has no attribute 'hasPassed'

V2 Reusing tests: generating tests with factories

TL:DR;

testAppleDef = helpers.assertDefFactory(function="simulate_apple", parameters=["x", "dt"])

Problem: checks with descriptive error messages, hints and detailed asserts are both lengthy and hard to re-use. Take this small example:

from checkpy import *

@test()
def testAppleDef():
    """Defines the function `simulate_apple()`"""
    assert "simulate_apple" in static.getFunctionDefinitions(),\
        "make sure the function is defined with the correct name"

    assert getFunction("simulate_apple").parameters == ["x", "dt"],\
        "make sure the function has the correct parameters"

Re-using (copy-pasting) this check would require choosing a new test name, a new description and carefully replacing the function name "simulate_apple()" in multiple places. Plus all the inherent drawbacks of duplicating code.

Solution: a factory-style approach to generating tests. In the same file or in a separate file (for instance helpers.py):

import checkpy
from uuid import uuid4

def assertDefFactory(
    function: str,
    parameters: Tuple[Any]=(),
    timeout: int=10,
) -> checkpy.tests.Test:
    """Generates a test to assert a function definition is present with set parameters."""
    def foo():
        assert function in checkpy.static.getFunctionDefinitions(),\ 
            "make sure the function is defined with the correct name"

        assert checkpy.getFunction(function).parameters == list(parameters),\
            "make sure the function has the correct parameters"
 
    # Give the test function a unique name
    # Normally this would be the test's function name, but we don't have that here
    foo.__name__ == str(uuid4())

    # Define the description separately here
    # A formatted docstring  (f"""""") is only evaluated when the function is called
    # And... that might not happen if the test depends on others.
    foo.__doc__ = f"Defines the function `{function}()`"

    # Call the test decorator (checkpy.test()), then pass the test function foo
    return checkpy.test(timeout=timeout)(foo)

Back in the tests file replace the original test (testAppleDef) with:

import helpers

testAppleDef = helpers.assertDefFactory(function="simulate_apple", parameters=["x", "dt"])

In case helpers.py is defined in another directory, be sure to add that directory to sys.path. For instance:

import pathlib
import sys

sys.path.insert(0, str(pathlib.Path(__file__).parent))
import helpers

What about dependencies?

# Call `passed` or `failed` directly
testAppleDef = passed(someOtherTest)(helpers.assertDefFactory(function="simulate_apple", parameters=["x", "dt"]))

# Other checks can depend on generated checks like normal
@passed(testAppleDef)
def testAppleFunc():
     ...

What about custom hints and checks?

Probably best to add the option for a callback to the factory. For instance:

def assertDefFactory(
    ...
    before: Callable[[], None]=lambda: None,
) -> checkpy.tests.Test:
    """Generates a test to assert a function definition is present with set parameters."""
    def foo():
        before()
        ...

Then use it like so:

testAppleDef = helpers.assertDefFactory(
    function="simulate_apple",
    parameters=["x", "dt"],
    before=lambda: helpers.checkForNotAllowedCode("break")
)

Loops?

Sure.

for name, params in [("foo", ["x"]), ("bar", ["x", "y"])]:
    globals()[f"testDef{name.capitalize()}"] =\
        helpers.assertDefFactory(function=name, parameters=params)

More examples?

def assertFuncFactory(
    function: str,
    input: Union[Tuple[Any], Dict[str, Any]],
    output: Any,
    outputType: type=Any,
    hint: Callable[[Any], None]=lambda output: None,
    timeout: int=10,
) -> checkpy.tests.Test:
    """Generates a test to assert the outcome of a function call."""
    def foo():
        f = checkpy.getFunction(function)

        if isinstance(input, dict):
            inputStr = ", ".join([f"{k}={v}" for k, v in input.items()])
        else:
            inputStr = ", ".join(str(v) for v in input)

        if isinstance(input, dict):
            realOutput = f(**input)
        else:
            realOutput = f(*input)

        assert realOutput == checkpy.Type(outputType),\
            "make sure the function returns the correct type"

        hint(realOutput)

        assert realOutput == output,\
            f"Did not expect output {realOutput} (with input {inputStr})"

    foo.__name__ == str(uuid4())
    parameterStr = ", ".join(str(p) for p in input)
    foo.__doc__ = f"Testing {function}({parameterStr})"
    return checkpy.test(timeout=timeout)(foo)

def appleHint(output):
    t, v = output
    if v == approx(4.52, abs=0.1) or t == approx(159.47, abs=1):
        assert False, "Did you mix up the order of the return values?"

    if v == approx(44.3, abs=0.3):
        assert False, "Did you forget to convert to km/h?"

testApple = passed(testAppleDef, timeout=90, hide=False)(
    helpers.assertFuncFactory(
        function="simulate_apple",
        input=(100, 0.01),
        output=(approx(159.47, abs=0.1), approx(4.52, abs=0.1)),
        outputType=Tuple[float, float],
        hint=appleHint
    )
)

print Testing: file.py even if file is missing

That makes testing a module with missing submissions easier.

V2 matplotlib (& numpy)

Matplotlib has multiple blocking functions that often require disabling, and there is no easy way to do it.

New api:

monkeypatch.patchMatplotlib() # tries to set Agg as backend and disables blocking functions
monkeypatch.patchNumpy() # don't throw floating point warnings, throw errors instead

auto test download

utf-8 decoding error on binary files in tests repo

v2.0.10 downloads all files, not just .py files, but assumes everything is a text file....

V2 What should "from checkpy import *" import?

Current thinking:

__all__ = [
	"test",
    "failed",
    "passed",
    "outputOf",
    "only",
    "include",
    "exclude",
    "require",
    "file"
]

Should probably include getFunction too.

Catch print statements within functions

Get rid of path.Folder, path.File

Current version is just confusing

Write tests for failed and passed decorators in tests!

set .isTestCreator to True

declarative.class_

Test classes:

from checkpy import * 

Lexicon = (declarative.class_("Lexicon")
    .params("word_length")
    .method("get_word").params().returnType(str)
)

Hangman = (declarative.class_("Hangman")
    .params("word", "number_of_guesses")
    .method("guess").params("letter").returnType(bool)
    .method("number_of_guesses").params().returnType(int)
    .method("won").params().returnType(bool)
)

Then for checks:

testGuess = test()(Hangman
    .init("hello", 5)
    .method("number_of_guesses").call().returns(5)
    .method("guess").call("a").returns(False)
    .method("number_of_guesses").call().returns(4)
    .method("guess").call("e").returns(True)
    .method("number_of_guesses").call().returns(4)    
)

testWon = passed(testGuess)(Hangman
    .init("a", 1)
    .method("won").call().returns(False)
    .method("guess").call("a").returns(True)
    .method("won").call().returns(True)
)

Mixing with @literal ?

globals_ = {"Hangman": Hangman, "Lexicon": Lexicon}


@literal(globals=globals_)
def testGuess():
     """number of guesses decreases after incorrect guess"""
     hangman = Hangman("hello", 5)
     assert hangman.number_of_guesses() == 5
     assert not hangman.guess("a")
     assert hangman.number_of_guesses() == 4
     assert hangman.guess("e")
     assert hangman.number_of_guesses() == 4

@passed(testGuess)
@literal(globals=globals_)
def testWon():
     """won() returns True after guessing the word"""
     hangman = Hangman("a", 1)
     assert not hangman.won()
     assert hangman.guess("a")
     assert hangman.won()

isinstance(Hangman, declarative.class_("Hangman")) should return True.
any attribute access on declarative.class_("Hangman") should be passed on to Hangman
globals should be inserted before import *, then overwritten after to ensure globals are set on import and not overwritten by the student. Like so:
```
globalsCopy = deepcopy(globals)
exec("from Hangman import *", globals=globals)
globals.update(globalsCopy)
```
name mangling for methods defined on declarative.class_("Hangman") to prevent accidental calls from the student's code???

V2 Replace entities.path with pathlib.Path

Keep track of changes in global()

Use argv in caching

cache.py tuple() not hashable

I ran into some problems running checkpy because the following lines failed:

CheckPy/checkpy/caches.py

Lines 47 to 49 in 67f36f1

 key = args + tuple(values) + tuple(sys.argv) 

 if key not in localCache:

It specifically failed when passing a list to a function that is wrapped by this cache.

A tuple is not guaranteed to be hashable, it fails when the tuple contains an unhashable type:

([]) in {}  # TypeError: unhashable type: 'list'

Solution: use str() to create hashable versions of any object (with __str___)

Perhaps I'll create a PR sometime :-)

tinydb 4.1 removed table.purge_tables

assertlib.fileContainsFunctionDefinitions considers functions in comments as defined

Resolve -register path

Immediately resolve the path to an absolute path. Otherwise checkpy.testPath becomes relative

Whoops! tests and storage fall under entities

V2 File management

With the following api:

only("hello.py")     # only hello.py will be present
include("*.csv")     # also include all csv files
exclude("foo.csv", "bar.csv")   # don't include foo.csv and bar.csv
require("qux.zip")   # make sure qux.zip exists and then include qux.zip

Calling any of these functions will create a sandbox (a temporary directory) and copy all included files over. Each subsequent call will not create another sandbox, but will move any newly included or excluded files from or to the sandbox.

The api functions can be called in the test module and inside each test. A call in the module will create one sandbox for all tests to use. Useful for any compiling or pipeline-ish operation. For example:

from checkpy import *
import os

only("foo.py")

@test()
def testFoo1():
    """Foo 1"""
    with open('pretend_result_of_foo.csv", "w") as f:
        pass

@passed(testFoo, hide=False)  # @passed is not necessary, but probably the right thing to do here
def testFoo2():
    """Foo 2"""
    assert "pretend_result_of_foo.csv" in os.listdir()

Calling an api function inside a test will create a new sandbox for the test alone. It can only copy files from the module level sandbox if that also exists. test-level calls are useful for ensuring a clean slate for each test and for controlling files per test. For example:

from checkpy import *
import os

only("foo.py", "bar.py")

@test()
def testFoo():
    """Foo works"""
    only("foo.py")  # cannot include "qux.py" here because of module level sandbox
    assert os.listdir() == ["foo.py"]

literal tests

@literal()
def testGuess():
     """number of guesses decreases after incorrect guess"""
     hangman = Hangman("hello", 5)
     assert hangman.number_of_guesses() == 5
     assert not hangman.guess("a")
     assert hangman.number_of_guesses() == 4
     assert hangman.guess("e")
     assert hangman.number_of_guesses() == 4

produces the following output on AssertionError:

This check failed. Run the following code in the terminal to see why:
$ python3
>>> from hangman import *
>>> hangman = Hangman("hello", 5)
>>> assert hangman.number_of_guesses() == 5
>>> assert not hangman.guess("a")
>>> assert hangman.number_of_guesses() == 4
>>> assert hangman.guess("e")
>>> assert hangman.number_of_guesses() == 4

New test design

Going from this :

@t.test(0)
def exactHello1(test):
	test.test = lambda : assertlib.exact(lib.outputOf(test.fileName), "Hello, world!\n")
	test.description = lambda : "prints \"Hello, world!\""

to this:

@test()
def exactlyHello():
	'''prints "Hello, world!"'''
	return outputOf() == "Hello, world!\n"

While still supporting all previous forms of customization:

@test()
def exactlyHello(test):
	'''prints "Hello, world!"'''
        test.timeout = 3
        test.fail = "So close!"
	return outputOf() == "Hello, world!\n"

Some thoughts:

Lambdas are rarely necessary and should be optional.
test.fileName shouldn't be everywhere. Lets make it the default argument for things like outputOf.
test.test should be optional. The current approach is error-prone (accidentally calling student code outside) and verbose (especially for multi statement test functions).
Test order should be implicit by default.

@reset decorater for clearing cache between tests

useful for stateful programs

V2 pytest.approx

Approximately checking numbers is hard, it requires multiple lines of code and is easy to mess up. assertlib.numberOnLine used to address this, but produces no helpful feedback and is hard to use with assert statement.

Instead let's use pytest.approx:

assertlib.numberOnLine(1, line, deviation=0.5)

=>

numbers = []
for item in line.split():
    try:
        numbers.append(float(item))
    except ValueError:
        pass

assert approx(1, abs=0.5) in numbers
# produces a message like:
# assert 1 ± 5.0e-01 in [42.0, 12.0]

Sandboxing to prevent students from using wrong data files

for instance:

import checkpy.tests as t
import checkpy.lib as lib
import checkpy.assertlib as assertlib
from checkpy.entities.path import userDirectory

def sandbox():
    lib.require("data.file", source="https://data.com/data.file")
    lib.require("module.py", source=userDirectory)

@t.test(0)
def testingSomething(test):
    test.test = lambda : True
    test.description = lambda : "foo"

	key = args + tuple(values) + tuple(sys.argv)

	if key not in localCache: