Home repository
nlpsandbox / participation-dashboard Goto Github PK
View Code? Open in Web Editor NEWA flexdashboard that reports participation metrics for the NLP Sandbox
License: Apache License 2.0
A flexdashboard that reports participation metrics for the NLP Sandbox
License: Apache License 2.0
Share the dashboards with Bradley (MCW) and Hongfang (Mayo Clinic) and ask them what they think of them. Regarding the participation dashboard generated for each data site, ask them whether there are additional metrics that they would like to include, for example for the grant reports.
This ticket depends on the completion of the following tasks:
Take the following plot as example:
As time passes, the point of the current best-performing tool gets pushed and compressed to the left side of the plot. At some point, an observer may easily overlook the point of the best-performing tool if it is on the far left side of the plot.
Idea:
Flexdashboard icons; https://rstudio.github.io/flexdashboard/articles/using.html#icon-sets
docker run -e SYNAPSE_AUTH_TOKEN=... nlpsandbox/participation-dashboard:0.2.0 Rscript render_markdown.R --source_table_synapse_id syn23633030 --destination_folder_synapse_id syn26127246
Unable to find image 'nlpsandbox/participation-dashboard:0.2.0' locally
0.2.0: Pulling from nlpsandbox/participation-dashboard
16ec32c2132b: Already exists
43156c469c75: Pull complete
ff9235179179: Pull complete
150c98294b3b: Pull complete
562c1c9e2b91: Pull complete
31d357ac90ff: Pull complete
45c622a91246: Pull complete
ddea88b3b92c: Pull complete
1e24733cf252: Pull complete
a70478df8afc: Pull complete
Digest: sha256:c7ab5fee1e8cf780667311f67c1464dcc5f925fe56cfdee6b11b69d55a73a9d9
Status: Downloaded newer image for nlpsandbox/participation-dashboard:0.2.0
Welcome, Thomas Yu!NULL
processing file: participation-dashboard.Rmd
|... | 4%
ordinary text without R code
|...... | 8%
label: setup (with options)
List of 1
$ include: logi FALSE
|........ | 12%
ordinary text without R code
|........... | 16%
label: unnamed-chunk-1 (with options)
List of 2
$ message: logi FALSE
$ include: logi FALSE
Rows: 165 Columns: 64
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (27): ROW_ETAG, name, status, dockerrepositoryname, dockerdigest, orgSag...
dbl (37): ROW_ID, ROW_VERSION, id, createdOn, createdBy, modifiedOn, evaluat...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
|.............. | 20%
inline R code fragments
|................. | 24%
label: unnamed-chunk-2
|.................... | 28%
ordinary text without R code
|...................... | 32%
label: unnamed-chunk-3
|......................... | 36%
ordinary text without R code
|............................ | 40%
label: unnamed-chunk-4
|............................... | 44%
ordinary text without R code
|.................................. | 48%
label: unnamed-chunk-5
|.................................... | 52%
ordinary text without R code
|....................................... | 56%
label: unnamed-chunk-6
|.......................................... | 60%
ordinary text without R code
|............................................. | 64%
label: unnamed-chunk-7
|................................................ | 68%
ordinary text without R code
|.................................................. | 72%
label: unnamed-chunk-8
|..................................................... | 76%
ordinary text without R code
|........................................................ | 80%
label: unnamed-chunk-9
Quitting from lines 153-191 (participation-dashboard.Rmd)
Error in loadNamespace(x) : there is no package called 'plotly'
Calls: <Anonymous> ... loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
In addition: Warning message:
Problem with `mutate()` column `datetime`.
ℹ `datetime = lubridate::as_datetime(.data$createdOn/1000, origin = "1970-01-01")`.
ℹ One or more parsing issues, see `problems()` for details
Execution halted
Report the number of unique submitter IDs (they can be user ID or team ID) who made at least one submission during a given month. Report for the past 12 months. The motivation is to follow the evolution of the number of users who actively uses the NLP Sandbox over time (instead of just looking at the number of registered users).
Coming back to the idea of having our first dashboard rendered with different (sub)set of data. At this point I'm looking for two rendering of the dashboard:
The best way to select the data to use is probably at the level of the submission queues. As a reminder, for each NLP Sandbox task, we have an entrypoint, public-facing submission queue. We then have an internal queue for each data site that supports this task (not all data sites support all the tasks). Taking the example of the Date Annotation Task:
Below are all the entrypoint queue IDs that can be used for the overall participation dashboard:
Note: The current overall dashboard taken as input all the queues and report 6 tasks open. However the sixth task, covid-19 annotation, is not open yet. Filtering using submission queue will solve this issue.
How to identify the ID of the submissions (rows) to keep in the table Main Admin View
from which participation data are obtained:
Main Admin View
(syn22277124) where evaluationid
takes a value in the set of queue IDs listed above.MCW internal queue ID:
How to identify the ID of the submissions (rows) to keep in the table Main Admin View
from which participation data are obtained:
MCW
(syn22277124) where evaluationid
takes a value in the set of queue IDs listed above. From these rows, create a set of unique submission ID from the column name
called A
.Main Admin View
(syn22277124), keep all the rows where id
takes a value in A
.The title of the dashboards rendered should be:
Consider adding a (JSON) configuration file to the code base to customize the rendering of the participation dashboard, and soon other dashboards. At this point, for the participation dashboard, the configuration file should define the submission queue to use to filter the data and the title of the dashboard.
Show the value of the number of registered user now. This value correspond to the number of Synapse users in the Synapse team NLP Sandbox Users (currently 16 users)
Depends on #2
Report on the number of submissions made over time to visualize whether the adoption of the NLP Sandbox increase.
Suggestion:
The motivation is that we should have at least one submission per month ^^. In this early phase of the project, we don't get a submission every week.
performance-dashboard_contact_annotation.html
=> performance-dashboard-contact-annotation.html
Andrew Lamb
from the headerNLP Sandbox Contact Annotators
(adapt for other annotators)Overview
boxThe NLP Sandbox decomposed the PHI annotation task into smaller, modular tasks like the date annotation task, person name annotation task, etc. One of the motivation is to enable tool developers to identify where their time would be best invested by looking at the leaderboard of each task. For example, if there is multiple solutions with a near perfect score for the date annotation task but no satisfying solution yet for the person name annotation task, this will indicates to the developer that their time would be best spent working on a new solution for the person name annotation task.
Instead of visiting all the leaderboards in order to obtain this information, we could compile this information in a small dashboard made of tiles (square or rectangle), one for each task. Each tile should include the following information:
Therefore, just by looking at the color of the tiles, one would be able to identify the challenging tasks for which no satisfying solution has been submitted yet (red-orange colors).
@andrewelamb We can discuss offline how to get the above information if needed.
Note that for each of the current task, we report a score for two datasets, so we can either:
Also since we report more than one performance metric, we would need to select one.
And to complicate further, we have task like the Location annotation task that report scores for two variants of the task. :)
Dashboards:
Report the number/ratio of submissions that are successful. A submission is either failing because of a bug in the benchmarking infrastructure or because of a mistake/bug in the submitted tools. It would be nice to be able to distinguish the two as only the first type of errors inform on the stability of the infrastructure. Yet, we can indirectly contribute to reduce the second type of errors with good documentation. Anyway, this metric will be an incentive for us to improve the system/documentation.
Suggested layout:
status = ACCEPTED
(always based on the Main Admin View unless specified)SYNAPSE_TOKEN
.For each task, draw a plot similar to this one from paperswithcode. In this plot, the performance of each submissions is represented by a dot.
The blue line represents the evolution of the best submission.
It would be great to include this plot on the top of the leaderboard page of each task. Therefore, individual HTML file should be rendered for each task.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.