There's a few ways of how import filters may be bypassed such as monkey patching autograde or manipulating special variables in notebook state. Create respective test cases and prevention stategies:

• introduce default import filters: autograde and multiprocessing (see #24)
• regex search code cells for usage of special variables

Run some sort of code inspection that checks for library imports and compares them to a white list. This is not a trivial task as there are tons of edge cases.

• import foo classic import, easy to catch
• import foo as bar also not too hard
• from foo import bar again, doable
• x='foo';__import__(x) things are getting trickier
• import fnord.foo as notfoo import via other, allowed libraries
• ...

Python is a dynamic language and therefore allows a lot of dirty stuff to be done. Anyway, there should be a basic mechanism that prevents 99% of people to commit fraud.

The current state is, that the after all grading is finished we have to notify students of their grades and the reasons for that grade.

I wrote a small selenium script that gets the job done, but it might be worth exploring the API of moodle directly.

Hash all files in working directory and keep them only when changed during notebook execution.

It would be great if the filename of the archive is included in the summary.csv. This makes it much easier to identify which archive belongs to which student. Otherwise one has to search through all the archives to find the matching one.
I have already implemented that in a fork of mine: https://github.com/Feelx234/autograde
This fork also includes some other changes which makes it unsuitable for a direkt PR.

Requirements should be an attribute of the NotebookTest class and are installed when execute is called. This may look like it was done here.

The hooks are registered just like test methods (different decorator of course) and invoked before and after execution.

There's misunderstandings of what autograde can do and what not (see #8 (comment)). We should add a section in the readme/reports that clarifies the intentions of this project.

Right now, timeout on code execution is achieved by exploiting pythons debug features. This comes with a significant performance loss while it is still possible to be bypassed.

To enhance isolation and overall performance, although some initial overhead will be introduced, a process based approach should be considered.

I propose to mime the signature of exec, wrapping exec for cases where no timeout is specified and falling back on a separate process otherwise. For transferring state, I consider multiprocessing.Manager. However, feasibility has to be assessed.

autograde started as a unit testing like tool for Jupyter notebooks but evolved to be something more than that.

With it's audit mode, autograde got enabled for simi automatic workflows. So far, that includes:

• update / comment test results
• review markdown comments
• review plots

To get the most out of the tool, its workflow has to be considered when designing tasks and tests.

Here, I'd like to collect common task scenarios and ideas of how to formulate and test them without losing too many degrees of freedom. Ideally, this results in a guide with best practices and code snippets for both, tasks and tests.

It would be great if one can disable the notebook comparison which takes forever when using autograde summary. I would even suggest that it is disabled by default.
A dirty hack is already implemented in my fork: https://github.com/Feelx234/autograde

• ensure overlay-fs is used (increased build time dramatically)
• add :z flag to mounts to solve permission problems

Handle plotting appears to be a bit tricky and will focus on matplotlibonly, at least in first place.

• Inject inital code that ensures matplotlib is aware of running in a headless environment.
• Inject a code snipped after each cell, that checks whether something was potted, safes current state and resets figure eventually

In autograde/cli/test.py the subprocess is executed once using.call (no backend) and once using.run (with backend)

.call works fine
But when using.run with backend podman I get an error on Ubuntu complaining about the :Z argument not being recognized by podman. I think is related to the thing that the arguments are already surrounded with " which are then passed into .run

backend=podman works fine if I just replace line 68 with line 66 and remove the :Z

It would be great if the summary.csv also contains the grade per task, it currently only shows the total grade.

I have already made that change PR coming.

Scenario:
While writing your checker_script, you might want to see the output of the test directly without recreating the entire archive.
Also creating the archive multiple times is problematic as a FileExistsError is raised when the same notebook is created multiple times for the same (dummy) Team.
I would suggest providing a function like this:

def grade_notebook_no_archive(self, nb_path, target_dir=None, context=None):
    target_dir = target_dir or os.getcwd()    # prepare notebook
    with open(nb_path, mode='rb') as f:
        nb_data = f.read()    nb_hash = blake2b(nb_data, digest_size=4).hexdigest()    # prepare context and execute notebook
    with open('code.py', mode='wt') as c:
        # actual notebook execution
        try:
            logger.debug('execute notebook')
            state = exec_notebook(
                io.StringIO(nb_data.decode('utf-8')),
                file=c,
                ignore_errors=True,
                cell_timeout=self._cell_timeout
            )        except ValueError:
            state = {}    # infer meta information
    group = state.get('team_members', {})    if not group:
        logger.warning(f'Couldn\'t find valid information about team members in "{nb_path}"')    # execute tests
    logger.debug('execute tests')
    results, summary = self.apply_tests(state)
    enriched_results = dict(
        autograde_version=autograde.__version__,
        orig_file=str(nb_path),
        checksum=dict(
            md5sum=md5(nb_data).hexdigest(),
            sha256sum=sha256(nb_data).hexdigest(),
            blake2bsum=nb_hash
        ),
        team_members=group,
        test_cases=list(map(str, self._cases)),
        results=results,
        summary=summary,
    )
    return enriched_results

Note that this function still outputs the code.py file.
Then in your python test script you can do

if __name__ == '__main__':
    target = Path("my_solutions.ipynb")
    result = grade_notebook_no_archive(nbt, nb_path = target.absolute())
    for val in result['results'].values():
        print("<<<<<  ", val['target'])
        print(val['message'])

Having such a file would also writing tests for autograde much easier as you can isolate the test_result from the archive.

The container image builds on continuumio/anaconda3 which does not support Python 3.8 and is too bloated.

I recently figured out, that capture_output's functionality is offered by contextlib.redirect_stdout and contextlib.redirect_stderr from the standard library!

As we are not (yet) able to fully automatically grade textual answers of students we need a semi automatic solution.

After automatic tests have been run, it would be great to have a user interface that allows you to quickly grade students text submission by hand.

Minimal requirements are:

1. Show the answers that students gave (image or text)
2. Allow a grading person to specify comments and a grade
3. Write those back into e.g. the report but also somewhere else, such that if the archive gets overwritten by rerunning the tests, the comments and grades persist
4. allow the option to later revisit that comment and grade

Bonus requirements:

• Let a grader specify expected points to be mentioned ("Erwartungshorizont"), with possible grading schemes (e.g. 0.5 points for mentioning A, 0.5 points for mentioning B ...) or sth like mention at least 2 out of 4 each gives 0.5 point up to a max of 1. Also nice: feedback to students like: You forgot to mention A) -> - 0.5 points
• A way to check that all textual/image answers have been graded

A web interface would be great for that purpose. Preferably allow grading of all submissions for one task and then the next task rather than all tasks for one submission an then the next submission.

Possible issues:

• rendering rst into html?

As no automatic testing will be perfect, it would be great to have a unified way that allows you to easily, consistently and persistently update the grade of submissions.

Hi,

as the title suggests, it would be helpful to have the option to create test cases, which have no effect on the final grading and do not have a score. For example, I would like to give the students feedback if their list of group members has the correct format and if it contains enough participants, without giving home assignment points for this sanity check. Visual feedback with colors (red/green) and text (comments, stdout) should be preserved.

Correct me if I am wrong, but at the current stage it is only possible to create test cases with "np.nan" as maximal score. But this implies a NaN score for the final sum of points at the end of the report.

best,
Lukas

Scenario: a student is asked to implement a function for computing a e.g. a mean/median/variance of some data points from scratch without using library code. As this can be done without any side effects, an isolated view on that function without it having access to the notebook state would be handy for testing purposes (and should be easier to implement as import restrictions, see #2).

I found sth weird when multiprocessing is used with autograde. This is not a major concern just sth too keep in mind when moving forward.

Lets say a notebook contains roughly the following:

from multiprocessing import Pool

class Multiplier:
   def __init__(self, val):
       self.val = val
  def mult(self, x):
       return x *  self.val

def mult_by3_multiprocessing(x_list):
       cls = Multiplier(3)
       pool = Pool(2)
       _out = pool.map(cls.mult,x_list)
       pool.close()
      return _out

Now assume we want to test the mult_by3... function.
When calling the mult_by3 function there will be an error in the evaluation of the test complaining about some pickling errors.
Problem is, that pickle can't find the Multiplier class because it is defined in the state_dict of the notebook rather than the state_dict of the test.py. For context internally objects are shipped to other processed using pickle when using multiprocessing.

Just something to keep in mind

The current version of autograde is not working for me on Python 3.8.5:

However, I checked out at commit 470d3b4 and it is working fine.

I also noticed (as minor issue - but maybe it is even intended?) that the --target directory for results is not initialized automatically: