There are many options (both open-source and paid) for deploying Shiny apps. We will first explore RStudio Connect, then move on to others such as shinyproxy if necessary

• Have a call with RStudio sales team

Determine 6 sets of questions to use in the app ... One extra in case we have more time

There should be a more definitive break between rounds - now it is just a pop-up window 'wait for your instructor'. Instead we could have a code.

The App disconnects from the server when an "e" is typed in the range question answer box without an number afterwards.
e.g 1e Disconnection, 1e10 works fine (scientific notation for numbers).

Names like "Van der Wal" are not getting recorded correctly. We should trimws(FirstName) and trimws(LastName) before writing out to {pins}.

Would be nice to store the pins/results for several workshops separately and not on our R Studio Connect Pin Board.
This should avoid messing up the pins from different workshops and the need of sharing R Studio Connect Login data with everyone running a calibration workshop.

By gathering all pins in a general collection we can later make use of this to see how people progress.
We could, for example, see how men and women (gender perspective) answer the questions and how wide the ranges are.

Wondering if translating the app to another language will impact the reading of the results in the admin app? We will test it out quickly now but will probably need to revisit this.

Following the literature on 'structured expert judgement' cf. 'Procedures Guide for Structured Expert Judgement in Accident Consequence Modeling' we could find a way to weight the inputs from users, based on their qualifications:

• how well they did in the calibration
• how much of an expert they are for the particular question they are answering
• etc.

RPDprcgd.pdf

We could consider gamification. Wits & Wagers might be a good template for making this a competition between people who are getting calibrated.

https://boardgamegeek.com/boardgame/20100/wits-wagers

Suggestion from our new colleague from University of Amsterdam @kbryk96

i.e. participants with the actual value in their range are "winners":
if multiple participants have the true value within the range the 'winners' are the ones with the best upper and lower bounds

Additional game option:

if the many have the actual value in the range compute Breir or other score for best

Set the URL as an environment variable (or something else)

• Should everyone receive the questions in the same order?
• Should all of the binary "TRUE/FALSE" questions be asked first, then the "Numeric Range" questions? Or should they be interspersed?

In the "Binary Results" and "Range Results" tables in the UI, we would like to have a way for users to view the actual question text (not just the question number) so that they can remember what the question was that they got correct/incorrect.

An approach to accomplishing this could be formatting the "Question" column in the table such that, when values in that column are hovered over, the question text appears.

We removed the decimals. Now 70% is 70 and not 0.70 or '70%'. The formatting should be uniform throughout all questions now.

We should prevent the submission of responses to "Range" type questions when the user has entered a value for "Lower 90" that is greater than the value they entered for "Upper 90"

Replace the current "xyz" text labels by a reference to a tab (or something else)

Large numeric values in {reactable} tables may display in scientific notation

Would be good to have a QR Code access the App for in person workshops. Typing the Link seems to be very challenging sometimes.

Create a new branch with the English version and range only - then - send it to colleagues to try to break it!

On the "Range" questions where there are > 10 questions in a group, the {reactable} table goes reverts back to the first 10 results after each question.

Let's see if there is a way that we can show the "current" page.

After the table (see issue #32) is ready

The hyperlinks in the "Truth" column of the interactive tables seem to be repeating the same 10 URLs across groups

Issue is due to this line in the app... needs to be re-factored

This would happen by generating a new set of questions from the input table for the model or models that the expert is expected to give feedback on

• Data should be stored seperately for each session and for each expert
• we should pre-assign names so that we know who answered (this with the app itself so that we keep track of the calibrated-ness of each participant)
• we should probably specify question numbers according to experts and experts should be able to skip some questions so that we are not asking the wrong people e.g. the nutritionist will not necessarily know about markets and the market expert will not necessarily know about nutrition contents.

In some cases the App does not react on the "Submit" button in the control field. You can still go back to the question and change the answer or look at the personal results but sumitting doesn't work.
This occoured during a calibration training with participants using differnet browsers. It is not dependet on the question as it occoured for differen questions.

Put together a collection of alternative programs for the training so that our users can choose one that works for their schedule. Currently we make a new one each time we do a training.

If a name including the letters ä, ö, ü or ß is used as user name the app crashs after the last question of the first question set (Group_1).

The selection of questions currently happens with an offline process:

• select questions from the google sheet
• send numbers to @ChristineSchmitz
• write in global.R with proper annotation {}[],, etc
• push to git
• update app in RStudio (or wait for automatic update)

Wish: The process could be simply in the admin app; for each calibration workshop, the admin should be able to identify the questions they want to use, and that is it. It should be possible to check them off on a list and these should be the questions used.

Maybe, in addition, the admin should be allowed to make more questions (not important but would be cool).

The letsencrypt (now certbot) is not working with the following erors:

1. The website is down and has error 'NET::ERR_CERT_DATE_INVALID'

2. The command certbot renew --dry-run on our Linux server has the error

Attempting to parse the version 2.6.0 renewal configuration file found at /etc/letsencrypt/renewal/agtools.app.conf with version 0.40.0 of Certbot. This might not work.
Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The requested nginx plugin does not appear to be installed
Attempting to renew cert (agtools.app) from /etc/letsencrypt/renewal/agtools.app.conf produced an unexpected error: The requested nginx plugin does not appear to be installed. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/agtools.app/fullchain.pem (failure)

The issue really seems to be in the Attempting to parse ... renewal configuration part.
Checks are ok:

• nginx and certbot are fully up to date (i.e. sudo apt update is all good)
• But ice cream does not have any bones? Or isn't them wasn't?

Add videos for each section explaining / providing a recap and preparing calibrate-ees for the next rounds (videos on biases, equivalent bet, premortem, unreasonably wide distribution etc.) ... These can be embedded in the pop up and be available in the notes for each section. We have done this for one video - as a kind of overview and a test - in the calibrator admin app already in app.R:

On app launch, display a pop-up modal welcoming the admin user

shiny::modalDialog(
title = "Welcome, Admin!",
shiny::HTML(
glue::glue(
"<iframe width='560' height='315'",
"src='https://www.youtube.com/embed/7P2YI9-smfU'",
"title='YouTube video player' frameborder='0' allow='accelerometer;",
"autoplay; clipboard-write; encrypted-media; gyroscope;",
"picture-in-picture' allowfullscreen></iframe>",
.sep = " "

Ultimately this should lower the load that we put on the trainer.

Add a window to select the App language (interface and questions) before entering the user name (e.g Flags to click on). --> one app for all languages instead of different git branches.

Should we allow negative values in Lower90 and Upper90 numeric inputs? Currently they are not allowed.

The given number for lower and upper bound are 0. When you start typing in your estimate the 0 belongs in the field. Would be easier if the field is empty when you start typing.

This could happen with

• storing each session's results in a new folder (not on the main page of the RStudio Connect / Posit)
• pre-assigning a username for each participant so that we have a clear way to keep track of them

Should we have an offline version of this?
Each has a spreadsheet etc. as it was (worked ok in the past already)
Just in case there needs to be an offline training

Once RStudio Connect is up and running, to get the app deployed we will have to:

• publish the app to the Connect server
• create & publish a {pins} test board and pin
• configure any necessary authentication
• configure URL for accessing the Calibration app directly

Question 3 the Brier should not be 1.

Code for Brier calculation:
https://github.com/CWWhitney/calibration_app/blob/Different_languages_binary_only/R/error.R

Currently, users who get disconnected have to re-launch the app and start over from the very beginning. At a minimum, we need to grab the users {pins} board data (if it exists), and then start them at the most recent group they were working on (for which data has not yet been written out to their individual pin.

Long term, we need to work on resolving the connection issues during these high-concurrency periods.

After the last question of each round (Group) the result table directly disappears. The result of the last question should be show before the not to next round window appears.

This is not the same as #16 which belongs to the question within one round of questions.

We want the ability to display the associated text from the "Comments" field when hovering over the questions "Source" hyperlink in the binary & range tables.

We currently haven't decided on the appropriate database to use to store the questions that get loaded into the app. IMO, the database needs to have easy functionality for data entry. I propose using Google Sheets as an option, in tandem with the {googlesheets4} package. Of course, there are other options that can be considered.

• determine database to use
• develop R code to replace data load before app UI code

Language and question type (binary/range) selection window at the beginning.

One App would make it much easier to maintain it, but wo should keep in mind that long if else conditions make the App unstable.

Once the RStudio Connect license has been obtained, will need to do the following to get Connect ready for apps:

• Install Connect on University server
• Configure HTTPS & URL for connecting to server

Getting "bad credentials" error. May need to whitelist the URL of RStudio Connect inside the Google account security. Creating an app password and using that in the config file on RStudio Connect has not resolved the issue.

Good news!
We finally collected together the questions we have piled up over the years (not as many as I suspected).
These are now online for us to fiddle with:
https://docs.google.com/spreadsheets/d/1yTboPXmDMF43YmjsuEH7bbPwcEj4fPfBD68rNWrOPSI/edit#gid=0

Because the "range" question data (the true answers) will be read in from the Google Sheet as type numeric, we need to find a way to convert percentage responses & answers cleanly in the UI (so that users can enter "70" instead of "0.70" when trying to respond with 70%)

To ensure that we are able to conduct a full calibration training with binary and range all the time. We want to make an Version with only binary as well.
(And at same time, checking if binary alone works as good as range alone)

At the beginning of the README, I would like to provide some more context as to what this app is intended to accomplish, and how Bonn sees it being used