braindump of suggestions about tmt HOT 9 CLOSED

If we could make tmt a library, that the runner could use, that would be the best

from tmt.

Thanks for the feedback. Good idea to separate listing info and executing tests. What about introducing a testset subcommand which could cover metadata investigation but could be also combined with test execution? It could look like this:

List available testsets:

tmt testset --list

Include gathered details about individual test steps in the output (basically what the tool does now):

tmt testset --list --steps
or
tmt testset --list --verbose

Filter available test sets:

tmt testset --list --filter REGEXP

Use together with test execution (e.g. select only test sets relevant for pull requests):

tmt testset --filter ^/ci/test/pull-request
tmt prepare execute testset --filter ^/ci/test/pull-request

I think the test execution should be the primary use case for the command line so I'd suggest to make the syntax for that super-short (preferrably without the extra run command).

tmt
tmt execute
tmt prepare execute
tmt provision prepare execute

What do you think?

from tmt.

Regarding the podman dependency and containerized runner: Agreed to have the support from the start but I'd propose to enable super-minimal-dependencies as well, meaning for those use cases when you just want to execute tests on a prepared vm/localhost without fetching any container/distro images.

from tmt.

@psss in regards to the first comment, it would also solve the situation :)

In regards to second, this will mean that we will need to add additional rpm dependencies to tmt, like ansible (for how: ansible), standard-test-roles (for how: sti, once we use it). By using a container this can be avoided. But I get your idea. The only thing that bugs me is how to solve both use cases without implementing the same functionality in the runner and in tmt.

from tmt.

So we do not have a duplicate functionality. That would mean abstracting away the command run (in case of localhost it would run directly commands on localhost, in case of the runner remotly via ssh). What do you think @psss? Also it would make sense to speak about this with @happz if there is no issue moving this forward like this,

from tmt.

Maybe the problem with the various dependencies we could just simply solve by having the various backends as subpackages in the future (tmt, tmt-sti, tmt-beakerlib, etc.)

from tmt.

@psss and also ... so for interaction with the CI systems runs we will have another tool? that is what you are suggesting?

from tmt.

Yeah, I like very much the idea of a common library. The major functionality should be implemented in the python module, the command line interface should be very simple/basic/dumb. If there si need for extra requirements I would agree the subpackage approach could be a nice way. That would allow minimal installation of python3-tmt for integration with CI, tmt for simple command line with only basic test execution on localhost and additional plugins such as tmt-sti or tmt-beakerlib for more.

from tmt.

I believe we've covered raised questions in the comment above and in person. Closing. Feel free to reopen if anything essential still needs to be addressed.

from tmt.