Semi-Automatic & Fully-Automatic Correctness Tests Needed about openems HOT 3 OPEN

During my optimization effort, I'm using a simple field dump test for correctness verification. Just find a simulation example, and turn the field dump on (for throughout testing I probably need to dump the entire simulation domain, but for now I'm dumping a 2D plane only as I'm using a PCB example). Then I compare the VTK output of both the upstream reference run and my own fork.

Since my code is not supposed to touch the floating-point arithmetic, only memory access, I'm checking floating-point for exact equivalence, which is okay if you're running the same code on the same system...

import os
import pyvista


def check_shape(dataset_a, dataset_b):
    if dataset_a.n_cells != dataset_b.n_cells:
        return False
    elif dataset_a.n_points != dataset_b.n_points:
        return False
    elif dataset_a.bounds != dataset_b.bounds:
        return False
    else:
        return True 


def check_timestep(upstream_dataset, dev_dataset):
    if not check_shape(upstream_dataset, dev_dataset):
        print("dataset do not have the same shape!")
        exit(1)

    points = 0 
    bad_points = 0 
    for i, val_i in enumerate(upstream_dataset.point_data['E-Field']):
        for j, val_j in enumerate(val_i):
            if upstream_dataset.point_data['E-Field'][i][j] != dev_dataset.point_data['E-Field'][i][j]:
                bad_points += 1
            points += 1

    if bad_points > 0:
        print("%d points in the dataset are not equal!" % bad_points)

    print("... %d points" % points)

files = list(os.scandir("./upstream/sim/dump_0/"))
files.sort(key=lambda file: file.name)
for dataset in files:
    print("checking...", dataset.name)

    upstream_dataset = pyvista.read(dataset.path)
    dev_dataset = pyvista.read("./dev/sim/dump_0/" + dataset.name)
    check_timestep(upstream_dataset, dev_dataset)

from openems.

I'm thinking of taking a shot at this. Any preference on python or octave?
I'm thinking of running the examples from the tutorial and saving the results as a reference. Then making a script that will run the examples and compare them to the saved reference. I guess there should be some adjustable tolerance on the what count as an identical result.

from openems.

Any preference on python or octave?

I'm personally relying heavily on Python, so that would be much more preferable for me, but that's probably more of a question for folks making engine/performance improvements etc. who wish to test performance updates. I.e., if they've already tested it in Python or Octave and validated the results are virtually the same, I probably wouldn't worry too much about re-running an extensive set of tests.

I guess there should be some adjustable tolerance on the what count as an identical result.

I agree, that had also been my thought process a while back.

from openems.