Unfortunately, there are simulation codes whose input files are not portable (e.g. because they require the absolute path to some data directory... no idea why).

For such cases, the current hash function will not work on CI since even when the data directory is provided, the absolute path to it will differ.

One might argue that it is really the simulation code that should be rewritten here, but unfortunately that is not always an option.
For such cases, it would be useful to be able to pass a custom hash function to the mock_code_factory fixture.

As we've done for aiida-core etc, it feels overkill to bother with gitflow for this package, and easier to delopy new releases without

@ltalirz @dev-zero?

• test that with --testing-config-action=generate, after tests, the template is written
• test that with --mock-regenerate-test-data, test data is actually rewritten (so far tests only that it still runs)

This will allow automatic migrations of the configuration and output directories.

The version number should be w.r.t. changes in the config and/or output format, not corresponding to the package version.

It would be useful to add tools to figure out why a particular calculation was not matched. Ideally, we could find some measure of "closeness" for the calculation, and show which is the closest.

In aiida-fleur we would like to place the executables to generate mock code test data relative to the location of the .aiida-testing-config.yml. Currently this is only possible with some hacks (See
https://github.com/JuDFTteam/aiida-fleur/blob/develop/.github/workflows/ci.yml#L136
for an example)
Here the paths are modified in the CI to be absolute

sed -i "s/\./${GITHUB_WORKSPACE//\//\\/}\/tests/g" .aiida-testing-config.yml
./run_all_cov.sh --local-exe-hdf5

We usually don't add compiled fleur executables to a global bin folder, since there are multiple executables with different configurations on the same machine very often.

Will merge #47 since it is needed by aiida-lsmo, but one should add

• a test of a run withmpi=True
• if possible, an exception/warning, if a number of processes >1 is detected (or implement a solution for it)

For tests where only a single calculation is run, it might make sense to "label" the output instead of / in addition to the hashing match.

I am not familiar with the genesis of the aiida-testing, but it is obviously highly generic and does not convey what functionality the plugin adds on top of what already exists in aiida-core.

Some thoughts: The plugin focuses on

• tests of processes
• tests that integrate AiiDA and the simulation software
• tests that go from setting up calculation inputs through to the final result of the calculation/workflow

Name candidates could therefore be:

1. aiida-process-tests
2. aiida-integration-tests
3. aiida-e2e-tests (e2e=end-to-end)

I would probably vote for option 3

CI should run:

• tests
• pre-commit
• doc build (depends on #2)

The usual approach to versioning of codes in AiiDA is, at the moment, to encode the version both in the label and in the description of the code.

It seems to me, the documentation of mock_code should include an example of a versioned code (besides diff).
The easiest thing would simply be to follow this pattern - it might lead some people to point a code "cp2k-5.1" to an actual cp2k-6.1 binary on their system (because that's what they happen to have installed) but I still think it's still better than to have no versioning.
This is necessary for tests against multiple versions of a code, and I think we should therefore already start with good habits.

Small detail: Currently, the mock-code factory only allows setting the label, not the description of a code.
Also, both label and description can be set directly via the constructor rather than like this

P.S. Currently, the kwarg for the entry point is called entry_point in the factory and input_plugin_name in the code.
One could decide to harmonize this.

First, we should collect cases where the current method fails, to get a better view of the requirements.

Add an introductory documentation for 'export_cache' in the user guide.

see #57 (comment) for how to do it

Currently, the .aiida-testing-config.yml config file is searched for in the CWD and all its parent directories. The recommended place to put it for an AiiDA plugin would be at the project root, so that it is always found regardless of where in the project the tests are executed.

Question to plugin developers: Does this work with your use case / plugin structure?

We should add a command-line flag to wipe the output directory before running the tests.

In the case of "uniquely labelled" content, maybe there should be an option to only regenerate it if the test was actually run.

Also useful would be a way to delete only outputs that were never touched during the test run.

To test that each submodule works also in isolation, add pytest runs with the "exclude plugin" option, e.g.

pytest -p no:aiida_testing.mock_code tests/export_cache

Feature request, nice to have.

It would be very nice if one could allow for bash regs in the ignorefile list.
example

ignore_files=['cdn**'] # to ignore files with names cdn01, cdn02, cdn23, ...

(if a code produces an arbitray number of files with certain naming pattern)

Otherwise thanks for your work! Currently everything is working nicely on linux. On mac I had an issue, which I did not yet managed to track down yet.

As reported by @DanielMarchand in #36, the install can fail if fastentrypoints cannot be found.

This can be fixed by un-vendoring fastentrypoints, and introducing the [build-system] info in a pyproject.toml. While we're at it, we can also consider switching from setup.json with a "complex" setup.py, to setup.cfg with a minimal setup.py.

Example setup.py: https://github.com/Z2PackDev/TBmodels/blob/dev/setup.py
Here we actually fail gracefully if fastentrypoints isn't present. I think this is for the purpose of running python setup.py {sdist,bdist_wheel}. Because pip isn't involved in that, the build isolation + environment setup doesn't work as expected.

Example pyproject.toml: https://github.com/Z2PackDev/TBmodels/blob/dev/pyproject.toml

Both these files can be used pretty much unaltered AFAICT, only setup.cfg would need adapting.

See #36 for additional discussion.

The current implementation only stores the outputs of a calculation, and not the inputs. We should add support (maybe flag-enabled) to also keep the inputs.

Add an introductory user guide for the mock_code submodule in the user guide.

Here is an example of a traceback resulting from missing test data on CI.

The error displayed is that some node that was expected from the calculation was not created, but this could have many reasons, which makes it unnecessarily hard to debug.

Of course, the calculation class itself could be more clever about parsing the outputs but ideally we (=aiida-testing) would be able to communicate to pytest that the mock code did encounter an input it did not yet know.

I'm not quite sure about the best way to accomplish this.
One could perhaps try to monkey-patch the calculation class of the input plugin when setting up the mock code Code instance, such that it does some extra check after the original _prepare_for_submission but perhaps there is also a less invasive way?

Any ideas @greschd ?

The current code for the ignore_files parameter allows ignoring files only based on their filename, not based on their path in the the run directory

However, I have a use case where a code creates a bunch of folders and I simply want to ignore the content of some of these - e.g. I'd like to ignore 'VTK/*'

An ignore_paths approach seems generally more powerful to me, so I propose to add this.

Add a "sanity check" session-scoped fixture (with autouse) that checks that the aiida-mock-codes executable works. For example, it could fail when pkg_resources detects an incompatibility in the requirements.

This probably also needs some change in the aiida-mock-codes executable itself, giving it a --sanity-check option or similar.

The current CI tests all install the package with pip install -e .. This doesn't fully test the packaging, because it roughly corresponds to adding the directory to sys.path. If there is a mistake in the manifest file or listed packages to install, it will not be caught.

We should add a job that installs with pip install ., without -e.

Prompted by a regression introduced in #38, which is fixed by #40.

The code is currently tested with (and designed for) a direct, non-MPI calculation only. We should test if it works also with schedulers and MPI.

Currently, the .aiida-testing-config.yml config file is committed to the repository. In general (and for plugins), this kind of defeats the purpose of the config file: It contains the parts that are different on different systems, and can not be hardcoded in the tests.

A partial solution is to just recommend not committing .aiida-testing-config.yml, however it would be good to make all fixtures run (if all caches are present) also without a config file, for the purposes of CI. Having to create a special "CI config file" instead would be quite cumbersome.

It is a bit painful right now to test for code that fails in unusual ways. Current caching assumes a kind of one-to-one mapping between inputs and outputs. What we would like is to test multiple fail conditions, e.g., due to node failure during computation, but all share the same input parameters. The only way that this can be done now is to manually copy cached files and modify neutral parameters, e.g., title in pw.x. @greschd @ltalirz