Gradle checks still fail after following installation instructions about ort HOT 5 CLOSED

The point of raising this repeatedly is this: if I need a compliance tool, and basic self-tests fail, I do not immediately understand whether I can use the tool to check compliance on my own software, or whether it will skip important details.

Unfortunately, that's not a good start.

from ort.

Thanks for the report. However, I'm not sure I can follow your argumentation here. Running these tests is a tool for developers to check against regressions in code changes they make. No pure user of a compliance tool would ever run them. So why do you insist it's crucial to make a developer tool work for the end user? I basically was already asking the same question over here, but unfortunately you did not reply.

from ort.

Here, I'll give it to you as a user story triplet:

As a developer or development manager,
In order to ensure compliance,
I want to know that my compliance check tool works.

from ort.

But running developer tests in order to check that the tool works for users is bogus. Sure, functional tests try to resemble what a user would do, but you can't say anything about the outcome of the tests unless you put ultimate trust into the people who wrote the tests, and believe that they took great care of reasonable coverage, to test the right things, and in the end not only have hard-coded a "BUILD SUCCESSFUL" console message. And if you put ultimate trust into these people anyway, you can just as well just run the tool and try it out (yay!).

Also, if due to your missing tools say 70% of the tests would be skipped, what confidence does that give you if the remaining tests pass? Are you now 30% certain the tool works? Or 100% certain it works for your use-case? What is you use-case? Is it covered by tests at all?

I'm sorry, but to me your argumentation is broken and artificial. The only way to check if the compliance tool works for you is to try it out, and not to run its developer checks.

from ort.

But - again, speaking as a developer or development manager - if I evaluate tools, they have to fulfil certain criteria before I even try them out:

1. They need to be sufficiently documented (you have some gaps there)
2. They need to self-test correctly (that's currently not the case)
3. They're actively maintained (seems to be the case)
4. etc.

Only after these checks are passed, do I bother trying them out or looking at the code base. These are very basic "smells" that I don't want to deal with in anything my business relies on. And these criteria actually follow an order as well:

1. Testing and compliance tools are judged the harshest.
2. Compilers.
3. Basic frameworks and essential libraries.
4. Optional libraries can fail a few criteria because they're easily replaced.

This is roughly in line with how damaging a failure of the tool is to the business.

I'm not even saying you're wrong per se, all you arguments are true. I just question the perspective. From the perspective of someone evaluating the reliability of tools in order to base their business interests on them, your tool currently fails pre-selection, and doesn't even qualify for being tried out.

You don't have to agree, and can feel free to close this issue, of course. I'm not invested into this more than trying to help a friend here.

from ort.