This plugin should probably be registered by an interface's test-suite manually (and entry-point removed). There is an API for this, so we just need to create a public-facing factory which will generate a plugin object which can be registered. See this comment for additional motivation.

Currently the code smashes all of the possible types together in one large cartesian product.

We can probably get 90% of the testing power by splitting method registrations by type. So for each type-class, we would have a single-param test for the "happy path", and then some sample of combinations from #4 for different combinations.

A non-exhaustive list of edge cases I am used to dealing with interfaces are listed below:

1. An action that takes only parameters
2. An action with parameter predicates
3. An action which generates multiple outputs
4. Visualizations from only parameters (usually metadata)
5. Metadata....
6. Multiple metadata inputs
7. Lists/Sets of parameters
8. Unions of inputs
9. Unions of primitives
10. Unions of Lists/Sets
11. Lists/Sets of Unions
12. TypeMap and TypeMatch

I think we can hit this if we have the following registrations:

• Exhaustive list of single-param methods (metadata is a param). So we would have an action for each candidate type (N).
  • covers: 1, 2, 5
• A sampled combination of 2-3 params and inputs (<<N)
  • covers: nothing unique, but should catch the same weirdness that the cartesian product would catch
• A few manually defined multiple output registrations
  • covers: 3, expects that output logic in an interface is independent from inputs (has always been true so far...)
• A few manually defined visualizers
  • covers: 4, expects that the input logic for visualizers are handled the same in methods (has always been true so far...)
• A method which takes all metadata params at once.
  • covers: 6
• Using the exhaustive list of single-params, wrap them in a List and Set (2N)
  • covers: 7
• A few manually defined methods/visualizers with Union-ed inputs
  • covers: 8, unions of inputs are simpler because they are always the interface mechanism (artifact)
• Unions of each pairwise combination of parameters (N x N)
  • covers: 9
• A sampled combination of 3 params in Union (<<NxN)
  • covers: 9
• Unions of each pairwise combination of Set/List-ed parameters (N x N)
  • covers: 10
• Lists and Sets of the sampled Unions (2(<<NxN))
  • covers: 11

... I am ignoring TypeMap for the moment.

I think a nice way to approach this would be to turn each of these into generators which can be consumed by other generators, allowing us to do wrapping and products as needed using the same overall interface. This also prevents any of the combinatorics from occurring before requested as generators are lazily evaluated.

Also, I kind of expect a reasonable split would be testing 1-6,8 as a particular registration mode, 7 as a more advanced one, and 9-11 as a even more advanced one. So a good start would be getting 1-6,8 in shape and then moving on to 7, etc.

See #5 for particular details about this.

It would be good to allow registrars of this plugin to choose how many test-cases are desired. This would allow quick sanity checking and iteration without spending time exhaustively running each permutation.

This could be accomplished using reservoir sampling to decide when to use a given iteration of the product.

See #5 for additional details, but essentially we want to cover the branches in this qiime2 utility.

... no suggestions at this time.