Documented API to query the testmon database for information about pytest-testmon HOT 8 OPEN

A colleague just came up with this exact idea independently and asked me about it like 10 minutes ago btw, so there's something here :P

ping @OakNinja

from pytest-testmon.

.testmondata is an sqlite database which is self-explanatory. Did you explore the database? (Update: As I was corrected the database structure is quite difficult to understand and it needs to be studied along the python code. It's also changing from version to version.) The big thing which is missing in the data is line numbers (only block checksums are stored, so that the data is relatively stable when facing unrelated changes of the file)

It could be relatively easy to add checksums of the lines to the database.

However the UI is the real challenge. Have a look here please: https://nedbatchelder.com/blog/201612/who_tests_what.html

@boxed So I would say your request is outside of the scope of testmon. Should we rephrase this issue to request storing line numbers or line checksums, or even whole lines in the sqlite database? Or should we close it as "wontfix" :/

Thanks for you feedback and usage of testmon.

from pytest-testmon.

Well, last time I had a long discussion with you you changed the database schema radically, so I don't consider it a stable API :P

I tried figuring out from the code how it all works but I didn't get it. Just exposing out something of the internal API to make the above usable would be pretty awesome I think.

from pytest-testmon.

Oh, and if we can query on the tests that touch a line we can create a coverage report with this data which would be pretty cool.

from pytest-testmon.

Yeah - this would be so awesome!

from pytest-testmon.

@boxed @OakNinja It seems the efforts to implement this in coverage.py stalled https://bitbucket.org/ned/coveragepy/issues/170/show-who-tests-what :(

As I wrote earlier I think the real challenge is the UI. How often would you care to write sql queries to find out which tests executed a specific line?
Is there any other report worth extracting? Do you think it would be practical to write sql for that?

With testmon you can just break the line and see what depends on it by executing pytest --testmon :)

from pytest-testmon.

A note for anyone reading this thread now - "Who tests what" was implemented in Coverage v5: nedbat/coveragepy#170 / https://nedbatchelder.com/blog/201810/who_tests_what_is_here.html.

.testmondata is an sqlite database which is self-explanatory.

For what it's worth, as a new user experimenting with this tool (and evaluating other tools that try to solve this problem), it's quite hard to evaluate what testmon is actually doing with the data in the database, short of running pytest --testmon and trying to observe how it selects tests. And while I do know SQL, I don't find the DB schema to be self-explanatory :). I'm sure it makes sense when you know the internals.

smother is another tool (not currently maintained, alas) that does similar things, and it has the commands smother diff and smother lookup that are really helpful for figuring out what the test runner would do:

https://smother.readthedocs.io/en/latest/reporting.htm

Perhaps the smother commands could provide some inspiration for queries that users might need; some commands that I found useful were:

smother diff -- diff the codebase and figure out what lines (chunks in testmon) have changed, and output the tests that are going to touch the changed code.

smother lookup src/path/myfile.py -- show me all the tests that touch this file. Useful for planning changes, e.g. if I'm preparing to refactor a given file, or exploring the codebase. This is probably now covered by the "who tests what" from coverage. However I suspect testmon is deriving that coverage information orthogonally to covereage? If so, being able to see what testmon thinks is covering a file is more useful than what coverage thinks, if they diverge.

from pytest-testmon.

Documented and stable API never happened and is not a priority because I'm still (occasionally) working on different versions of the underlying algorithm and data structures and I don't feel the need for the requested functionality.

Also Coverage v5 now completely covers the original request.

Wouldn't static analysis within an IDE be better to explore dependencies between methods? As I said originally I don't see many people using the data without a decent UI story.

from pytest-testmon.