question: Comparing models with multiple runs about ranx HOT 3 CLOSED

Hi and thanks for the kind words! :)

That's a tricky question.
I actually don't have a scientific answer, but your idea sounds reasonable to me.

Right now ranx does not support what you say out of the box but it can probably be achieved using the "low level" function you find in the source code.

An alternative could be to consider every variation of your model separately, run the comparison and then do something like "3 out of 4 model variations significantly improved over the baseline(s) so we can say that, generally, the model is better".

It would be safer to say that only if all your model variations improved over the baseline(s) to be honest...

You should talk with someone more experienced than me to find a scientific answer to your problem.

If you do, please, keep me posted!

Elias

from ranx.

Thank you for the quick response!

Surprisingly haven't been able to find any discussions on this subject specifically, but have no doubt that they must exist :).
Your alternative is definitely the safest approach, but testing all combinations for a large number of models and runs can become pretty computationally expensive.

If I come across the scientific answer to this I will definitely update you.

Best regards,

Tiago

from ranx.

Closing. Feel free to re-open if needed.

from ranx.