Test writing process improvements about aria-at HOT 4 OPEN

Will authoring tests in Excel be the long term solution? If not, it may be prudent to start work on an alternative sooner rather than later.

from aria-at.

@mfairchild365 commented:

Will authoring tests in Excel be the long term solution? If not, it may be prudent to start work on an alternative sooner rather than later.

I've been giving this some thought. At first, I thought Excel might be a good solution for at least the medium term, but after having gone through the process of reviewing test plans, I really think not.

Writing test plans is hard, even harder to get right than I anticipated. There is going to be lots and lots and lots of revising, even after we have significant experience writing them. I am anticipating that we will get quite a bit of feedback on the plans. So, if we had to rely on Excel imports to do all that work, it would get very time consuming.

The number of times I wished to myself, "Dang, where is the edit button for this test" is already pretty significant.

Our current test plan review pages are pretty nice. I can easily imagine an "Add Test" button next to the H1 at the top, and an "Edit test" button next to each H2. In short, I think we need to invest in a test composer/editor.

In the meantime, @spectranaut, can you give @jongund some requirements for modifying his script so that we can at least make importing reasonably efficient in the near term? That would enable us to keep moving forward with test writing until we have the infrastructure in place to build a test composer and keep you focused on building that infra.

from aria-at.

Ok, advice for @jongund about the script:

1. The scripts Jon wrote should be moved to scripts/ directory so they can be used for generating more tests.

   a. This will involved updating the scripts relative paths, specifically, where the test files and commands.json files are written to. Probably this should be a command line variable that tells you the directory name for the design pattern (like "checkbox" or "menubar-editor"). Then, if the exported CSV files are always named the same thing and kept in the data/ directory you won't have to provide them as a command line variable (because you can just look for a file of that name).

   b. You could just have one script, pass it the directory for the example, and it could look for all the appropriate files and create all the appropriate files. It would remove one step of work!

2. The following data validation should be done for the tests:

   a. The commands.json file should be loaded by the script that produces test files, and it should make sure that every "task" has a corresponding entry in commands.json (also cross referencing the "applies to" field).

   b. There should be data validation on the string values provided for "applies to" field.

   c. There should be data validation on the string value for "mode"

   d. There should be data validation on the key strings -- the ones that correspond to key or key combinations in tests/resources/keys.mjs. This file could probably be consumed and read by the python script for this data validation.

3. All tests should be delete by the script before they are re-written. This is necessary because if a file is deleted from the excel/CSV file, it needs to be deleted from the directory.

from aria-at.

@spectranaut

Thank you for the feedback.
Valarie I will work on theses changes.

from aria-at.