feature request: save Report.comparisons as JSON about ranx HOT 13 CLOSED

Hi, an export option for the Report class is already on my to-do list! :)

I will come back with a proposal so that we can discuss it before I implement the functionality.

from ranx.

I added Report.to_dict and Report.save.
I updated ranx on PyPi with these new features.

Closing.

from ranx.

Fixed in 0.1.10. Sorry for the inconvenience.

from ranx.

Hey, sorry for the delay.

This is my proposal for the Report.to_dict function (I can add a Report.save_as_json function for convenience too):

{
    # metrics and model_names allows to read the report without
    # inspecting the json to discover the used metrics and
    # the compared models
    "metrics": ["metric_1", "metric_2", ...],
    "model_names": ["model_1", "model_2", ...],
    #
    "model_1": {
        "scores": {
            "metric_1": ...,
            "metric_2": ...,
            ...
        },
        "comparisons": {
            "model_2": {
                "metric_1": ...,  # p-value
                "metric_2": ...,  # p-value
                ...
            },
            ...
        },
        "win_tie_loss": {
            "model_2": {
                "W": ...,
                "T": ...,
                "L": ...,
            },
            ...
        },
    },
    ...
}

Let me know what you think. :)

from ranx.

Looks great (and there was not so much delay 😅)!

from ranx.

I’m getting a "TypeError: Object of type int64 is not JSON serializable" which is probably coming from numba or numpy

from ranx.

Yeah, I know about that issue. I will look into it soon.

As a workaround, you can call report.to_dict() and save the dictionary as a JSON by yourself with the exact same code I wrote for the report.save function.

That issue it's kinda weird.

from ranx.

don’t you need to convert int64 to int in to_dict?

from ranx.

for example in transformers they use:

def denumpify_detensorize(metrics):
    """
    Recursively calls `.item()` on the element of the dictionary passed
    """
    if isinstance(metrics, (list, tuple)):
        return type(metrics)(denumpify_detensorize(m) for m in metrics)
    elif isinstance(metrics, dict):
        return type(metrics)({k: denumpify_detensorize(v) for k, v in metrics.items()})
    elif isinstance(metrics, np.generic):
        return metrics.item()
    elif is_torch_available() and isinstance(metrics, torch.Tensor) and metrics.numel() == 1:
        return metrics.item()
    return metrics

from ranx.

this fixes it but you probably want to deal with it some other way? If not I can open a PR PaulLerner@7e2218d

from ranx.

I will look into it soon.

from ranx.

I’m still getting TypeError: Object of type int64 is not JSON serializable

from ranx.

oops, looks like I was on the wrong branch, sorry about that

from ranx.