jelleas / checkpy Goto Github PK
View Code? Open in Web Editor NEWA Python tool for running tests on Python source files. Intended to be used by students whom are taking courses in the Minor Programming at the UvA.
License: MIT License
A Python tool for running tests on Python source files. Intended to be used by students whom are taking courses in the Minor Programming at the UvA.
License: MIT License
Check for existing data files (specified in tests), error if not present?
Try download if possible?
@t.test(0):
def first(test):
pass
@t.passed(first)
@t.test(1)
def second(test):
pass
@t.passed(second)
@t.test(2)
def second(test):
pass
You'll receive:
AttributeError: 'NoneType' object has no attribute 'hasPassed'
TL:DR;
testAppleDef = helpers.assertDefFactory(function="simulate_apple", parameters=["x", "dt"])
Problem: checks with descriptive error messages, hints and detailed asserts are both lengthy and hard to re-use. Take this small example:
from checkpy import *
@test()
def testAppleDef():
"""Defines the function `simulate_apple()`"""
assert "simulate_apple" in static.getFunctionDefinitions(),\
"make sure the function is defined with the correct name"
assert getFunction("simulate_apple").parameters == ["x", "dt"],\
"make sure the function has the correct parameters"
Re-using (copy-pasting) this check would require choosing a new test name, a new description and carefully replacing the function name "simulate_apple()" in multiple places. Plus all the inherent drawbacks of duplicating code.
Solution: a factory-style approach to generating tests. In the same file or in a separate file (for instance helpers.py
):
import checkpy
from uuid import uuid4
def assertDefFactory(
function: str,
parameters: Tuple[Any]=(),
timeout: int=10,
) -> checkpy.tests.Test:
"""Generates a test to assert a function definition is present with set parameters."""
def foo():
assert function in checkpy.static.getFunctionDefinitions(),\
"make sure the function is defined with the correct name"
assert checkpy.getFunction(function).parameters == list(parameters),\
"make sure the function has the correct parameters"
# Give the test function a unique name
# Normally this would be the test's function name, but we don't have that here
foo.__name__ == str(uuid4())
# Define the description separately here
# A formatted docstring (f"""""") is only evaluated when the function is called
# And... that might not happen if the test depends on others.
foo.__doc__ = f"Defines the function `{function}()`"
# Call the test decorator (checkpy.test()), then pass the test function foo
return checkpy.test(timeout=timeout)(foo)
Back in the tests file replace the original test (testAppleDef
) with:
import helpers
testAppleDef = helpers.assertDefFactory(function="simulate_apple", parameters=["x", "dt"])
In case helpers.py
is defined in another directory, be sure to add that directory to sys.path
. For instance:
import pathlib
import sys
sys.path.insert(0, str(pathlib.Path(__file__).parent))
import helpers
# Call `passed` or `failed` directly
testAppleDef = passed(someOtherTest)(helpers.assertDefFactory(function="simulate_apple", parameters=["x", "dt"]))
# Other checks can depend on generated checks like normal
@passed(testAppleDef)
def testAppleFunc():
...
Probably best to add the option for a callback to the factory. For instance:
def assertDefFactory(
...
before: Callable[[], None]=lambda: None,
) -> checkpy.tests.Test:
"""Generates a test to assert a function definition is present with set parameters."""
def foo():
before()
...
Then use it like so:
testAppleDef = helpers.assertDefFactory(
function="simulate_apple",
parameters=["x", "dt"],
before=lambda: helpers.checkForNotAllowedCode("break")
)
Sure.
for name, params in [("foo", ["x"]), ("bar", ["x", "y"])]:
globals()[f"testDef{name.capitalize()}"] =\
helpers.assertDefFactory(function=name, parameters=params)
def assertFuncFactory(
function: str,
input: Union[Tuple[Any], Dict[str, Any]],
output: Any,
outputType: type=Any,
hint: Callable[[Any], None]=lambda output: None,
timeout: int=10,
) -> checkpy.tests.Test:
"""Generates a test to assert the outcome of a function call."""
def foo():
f = checkpy.getFunction(function)
if isinstance(input, dict):
inputStr = ", ".join([f"{k}={v}" for k, v in input.items()])
else:
inputStr = ", ".join(str(v) for v in input)
if isinstance(input, dict):
realOutput = f(**input)
else:
realOutput = f(*input)
assert realOutput == checkpy.Type(outputType),\
"make sure the function returns the correct type"
hint(realOutput)
assert realOutput == output,\
f"Did not expect output {realOutput} (with input {inputStr})"
foo.__name__ == str(uuid4())
parameterStr = ", ".join(str(p) for p in input)
foo.__doc__ = f"Testing {function}({parameterStr})"
return checkpy.test(timeout=timeout)(foo)
def appleHint(output):
t, v = output
if v == approx(4.52, abs=0.1) or t == approx(159.47, abs=1):
assert False, "Did you mix up the order of the return values?"
if v == approx(44.3, abs=0.3):
assert False, "Did you forget to convert to km/h?"
testApple = passed(testAppleDef, timeout=90, hide=False)(
helpers.assertFuncFactory(
function="simulate_apple",
input=(100, 0.01),
output=(approx(159.47, abs=0.1), approx(4.52, abs=0.1)),
outputType=Tuple[float, float],
hint=appleHint
)
)
That makes testing a module with missing submissions easier.
Matplotlib has multiple blocking functions that often require disabling, and there is no easy way to do it.
New api:
monkeypatch.patchMatplotlib() # tries to set Agg as backend and disables blocking functions
monkeypatch.patchNumpy() # don't throw floating point warnings, throw errors instead
v2.0.10 downloads all files, not just .py files, but assumes everything is a text file....
Current thinking:
__all__ = [
"test",
"failed",
"passed",
"outputOf",
"only",
"include",
"exclude",
"require",
"file"
]
Should probably include getFunction
too.
Current version is just confusing
set .isTestCreator to True
Test classes:
from checkpy import *
Lexicon = (declarative.class_("Lexicon")
.params("word_length")
.method("get_word").params().returnType(str)
)
Hangman = (declarative.class_("Hangman")
.params("word", "number_of_guesses")
.method("guess").params("letter").returnType(bool)
.method("number_of_guesses").params().returnType(int)
.method("won").params().returnType(bool)
)
Then for checks:
testGuess = test()(Hangman
.init("hello", 5)
.method("number_of_guesses").call().returns(5)
.method("guess").call("a").returns(False)
.method("number_of_guesses").call().returns(4)
.method("guess").call("e").returns(True)
.method("number_of_guesses").call().returns(4)
)
testWon = passed(testGuess)(Hangman
.init("a", 1)
.method("won").call().returns(False)
.method("guess").call("a").returns(True)
.method("won").call().returns(True)
)
Mixing with @literal
?
globals_ = {"Hangman": Hangman, "Lexicon": Lexicon}
@literal(globals=globals_)
def testGuess():
"""number of guesses decreases after incorrect guess"""
hangman = Hangman("hello", 5)
assert hangman.number_of_guesses() == 5
assert not hangman.guess("a")
assert hangman.number_of_guesses() == 4
assert hangman.guess("e")
assert hangman.number_of_guesses() == 4
@passed(testGuess)
@literal(globals=globals_)
def testWon():
"""won() returns True after guessing the word"""
hangman = Hangman("a", 1)
assert not hangman.won()
assert hangman.guess("a")
assert hangman.won()
isinstance(Hangman, declarative.class_("Hangman"))
should return True.
any attribute access on declarative.class_("Hangman")
should be passed on to Hangman
globals should be inserted before import *
, then overwritten after to ensure globals are set on import and not overwritten by the student. Like so:
globalsCopy = deepcopy(globals)
exec("from Hangman import *", globals=globals)
globals.update(globalsCopy)
name mangling for methods defined on declarative.class_("Hangman")
to prevent accidental calls from the student's code???
I ran into some problems running checkpy
because the following lines failed:
Lines 47 to 49 in 67f36f1
It specifically failed when passing a list to a function that is wrapped by this cache.
A tuple
is not guaranteed to be hashable, it fails when the tuple
contains an unhashable type:
([]) in {} # TypeError: unhashable type: 'list'
Solution: use str()
to create hashable versions of any object (with __str___
)
Perhaps I'll create a PR sometime :-)
Immediately resolve the path to an absolute path. Otherwise checkpy.testPath becomes relative
With the following api:
only("hello.py") # only hello.py will be present
include("*.csv") # also include all csv files
exclude("foo.csv", "bar.csv") # don't include foo.csv and bar.csv
require("qux.zip") # make sure qux.zip exists and then include qux.zip
Calling any of these functions will create a sandbox (a temporary directory) and copy all included files over. Each subsequent call will not create another sandbox, but will move any newly included or excluded files from or to the sandbox.
The api functions can be called in the test module and inside each test. A call in the module will create one sandbox for all tests to use. Useful for any compiling or pipeline-ish operation. For example:
from checkpy import *
import os
only("foo.py")
@test()
def testFoo1():
"""Foo 1"""
with open('pretend_result_of_foo.csv", "w") as f:
pass
@passed(testFoo, hide=False) # @passed is not necessary, but probably the right thing to do here
def testFoo2():
"""Foo 2"""
assert "pretend_result_of_foo.csv" in os.listdir()
Calling an api function inside a test will create a new sandbox for the test alone. It can only copy files from the module level sandbox if that also exists. test-level calls are useful for ensuring a clean slate for each test and for controlling files per test. For example:
from checkpy import *
import os
only("foo.py", "bar.py")
@test()
def testFoo():
"""Foo works"""
only("foo.py") # cannot include "qux.py" here because of module level sandbox
assert os.listdir() == ["foo.py"]
@literal()
def testGuess():
"""number of guesses decreases after incorrect guess"""
hangman = Hangman("hello", 5)
assert hangman.number_of_guesses() == 5
assert not hangman.guess("a")
assert hangman.number_of_guesses() == 4
assert hangman.guess("e")
assert hangman.number_of_guesses() == 4
produces the following output on AssertionError:
This check failed. Run the following code in the terminal to see why:
$ python3
>>> from hangman import *
>>> hangman = Hangman("hello", 5)
>>> assert hangman.number_of_guesses() == 5
>>> assert not hangman.guess("a")
>>> assert hangman.number_of_guesses() == 4
>>> assert hangman.guess("e")
>>> assert hangman.number_of_guesses() == 4
Going from this :
@t.test(0)
def exactHello1(test):
test.test = lambda : assertlib.exact(lib.outputOf(test.fileName), "Hello, world!\n")
test.description = lambda : "prints \"Hello, world!\""
to this:
@test()
def exactlyHello():
'''prints "Hello, world!"'''
return outputOf() == "Hello, world!\n"
While still supporting all previous forms of customization:
@test()
def exactlyHello(test):
'''prints "Hello, world!"'''
test.timeout = 3
test.fail = "So close!"
return outputOf() == "Hello, world!\n"
Some thoughts:
outputOf
.test.test
should be optional. The current approach is error-prone (accidentally calling student code outside) and verbose (especially for multi statement test functions).useful for stateful programs
Approximately checking numbers is hard, it requires multiple lines of code and is easy to mess up. assertlib.numberOnLine
used to address this, but produces no helpful feedback and is hard to use with assert
statement.
Instead let's use pytest.approx
:
assertlib.numberOnLine(1, line, deviation=0.5)
=>
numbers = []
for item in line.split():
try:
numbers.append(float(item))
except ValueError:
pass
assert approx(1, abs=0.5) in numbers
# produces a message like:
# assert 1 ± 5.0e-01 in [42.0, 12.0]
for instance:
import checkpy.tests as t
import checkpy.lib as lib
import checkpy.assertlib as assertlib
from checkpy.entities.path import userDirectory
def sandbox():
lib.require("data.file", source="https://data.com/data.file")
lib.require("module.py", source=userDirectory)
@t.test(0)
def testingSomething(test):
test.test = lambda : True
test.description = lambda : "foo"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.