Comments (5)
Yes. What would the alternative be? Just the metrics? (I called them statistics in the API, but metrics is a better term).
from openframing.
I think we should have a metrics method (for the case where we're labeling an unlabeled set, for example) and a metrics + statistics method that calls the metrics method (for the case where we're labeling a validation or test set and want to see how it performs). Does that make sense? So we'd want an inference-only endpoint for sure
from openframing.
Can you clarify what you mean by statistics and what you mean by metrics?
What I mean by metrics is exactly the same thing as what I mean by statistics(f1, precision, recall, accuracy).
We do have an endpoint for running inference on an unlabeled set, but currently, there is no API endpoint for calculating any metrics/statistics on an unlabeled set, as we wouldn't have any "gold" labels by definition, since it's unlabeled dataset.
from openframing.
The wording at the beginning is terrible - my bad.
Basically, I was thinking that once the model exists and is trained, getting predictions from it (1 task we'll need) is a relatively self-contained task, as is getting the performance of those predictions (another task we'll need).
So we'll want 2 functions in the end (which encompasses this issue as well):
- one for inference only; returns the results
- one that does inference & computes how well the model does; returns (optionally?) the results and definitely the performance metrics.
Does that all make sense to you?
from openframing.
Ah ok. Yes, that makes sense.
from openframing.
Related Issues (20)
- Provide example datasets in the framing and LDA pages so the user could test the functionalities without having their own data HOT 2
- Add 'queued for training' to the training statuses HOT 2
- Embedd the video tutorial in the 'Tutorials' tab of our Webpage
- Echo back to the user the error messages that the backend responds with HOT 1
- LDA topic name assignment doesn't work HOT 3
- Create additional pretrained classifiers HOT 8
- What are the minimum system specs required for AWS EC2 instance? Then Downgrade to it. HOT 2
- Change http://www.openframing.org to https://www.openframing.org HOT 1
- Let the user get a link in the email when training a classifier is done, and then upload an unlabeled set HOT 12
- Let the emails sent for training a classifier or topic model include metrics HOT 1
- Adding the project tags, descriptions, and Website to the 'About' section in GitHub HOT 1
- Add a one liner in HTML to explain to the user that they need two columns in their data namely Example, Label
- Provide instruction for the user in a line and let them know they should have only one column in their data
- Show a message to the user on the landing page of openframing.org when the system is down or under reconstruction
- Cannot input "/" in the label HOT 1
- When sharing the openframing.org in a social media (e.g. Facebook), the thumbnail is not good HOT 3
- Don't use 'gun violence' for 'other' policy issue when we ask the user to use 'our dataset' HOT 5
- Add an example to visualize framing in news HOT 6
- Document error messages in the backend API
- support for multiple languages for BERT and LDA HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openframing.