Predictor (inference) method for BERT-based classifier about openframing HOT 5 CLOSED

Yes. What would the alternative be? Just the metrics? (I called them statistics in the API, but metrics is a better term).

from openframing.

I think we should have a metrics method (for the case where we're labeling an unlabeled set, for example) and a metrics + statistics method that calls the metrics method (for the case where we're labeling a validation or test set and want to see how it performs). Does that make sense? So we'd want an inference-only endpoint for sure

from openframing.

Can you clarify what you mean by statistics and what you mean by metrics?

What I mean by metrics is exactly the same thing as what I mean by statistics(f1, precision, recall, accuracy).

We do have an endpoint for running inference on an unlabeled set, but currently, there is no API endpoint for calculating any metrics/statistics on an unlabeled set, as we wouldn't have any "gold" labels by definition, since it's unlabeled dataset.

from openframing.

The wording at the beginning is terrible - my bad.

Basically, I was thinking that once the model exists and is trained, getting predictions from it (1 task we'll need) is a relatively self-contained task, as is getting the performance of those predictions (another task we'll need).

So we'll want 2 functions in the end (which encompasses this issue as well):

• one for inference only; returns the results
• one that does inference & computes how well the model does; returns (optionally?) the results and definitely the performance metrics.

Does that all make sense to you?

from openframing.

Ah ok. Yes, that makes sense.

from openframing.