I am trying to fix the compilation issue introduced by the including the image ForkingPaths.png.

There were a few typos in the code, but now that those are fixed it still doesn't compile and the culprit is the papaja format. Do we need to use this format? Or can we just switch to a cleaner/simpler format? If we can, I am happy to change this (it's a quick thing), so we have better control over the settings and output.

(What was the journal we were thinking of submitting too again? Maybe they have a latex template and we can just use that as the tex template for knitr).

This will be extremely useful (I believe) for working through the functions in this package:

https://besjournals.onlinelibrary.wiley.com/doi/10.1111/j.2041-210X.2012.00224.x

Leave your comments here.

• Bastiaansen et al., 2019 Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology
• Boehm et al., 2018 Estimating across-trial variability parameters of the Diffusion Decision Model: Expert advice and recommendations
• Botvinik-Nezer et al., 2020 Variability in the analysis of a single neuroimaging dataset by many teams
• Breznau et al., 2020 The Crowdsourced Replication Initiative: Investigating Immigration and Social Policy Preferences
• Dulith et al., 2019 The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models
  Functional Imaging Analysis Contest 2006
• Fillard 2011 Quantitative evaluation of 10 tractography algorithms on a realistic diffusion MR phantom
• Maier-Hein 2017 The challenge of mapping the human connectome based on diffusion tractography
• Salganik et al., 2020 Measuring the predictability of life outcomes with a scientific mass collaboration
• Schweinsberg et al., 2020 Radical dispersion in estimates when independent scientists operationalize and test the same hypothesis with the same data
• Silberzahn et al., 2018 Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
• Starns et al., 2019 Assessing theoretical conclusions with blinded inference to investigate a potential inference crisis
• van Dongen et al., 2019 Multiple Perspectives on Inference for Two Simple Statistical Scenarios

Other stuff:

• ecology repo: https://osf.io/e5xhm/
• another list: https://docs.google.com/document/d/12XlCX0UWKLH1RJ9NsSciWxXn4DD0oh4slqd_TmKjp0A/edit

Action items

• Continue working on surveys (@jvcasillas) (https://github.com/jvcasillas/many_analyses/issues/1#issuecomment-771687908)
• Include information about German phonetics/phonology, consider linking to IPA chart in German
• For logo, possibly make forking paths same width as word, consider font boldness, make plural
• Look at landing page again, focus on text (everybody)

Discuss

• Meta-analytical model.
  • Prior for the group-level standard deviation: HalfCauchy(0, 1) might be too wide.
• Considering making a many_analyses.github.io website.

Put your ideas in this issue.

I have pushed changes from my and Timo's branches into master. Now it's a good time to merge from master into trdev.

Some notes for us to discuss later

Dependent variable

• (1) confirmed hypothesis vs. null
• (2) standardized effect magnitude (will be difficult with GAMs etc.)
• (3) uncertainty? (some standardized measure of certainty?)

Predictors (don't have to use them all later on, but things that would be interesting)

• career stage (grad student, postdoc, faculty)
• self assessed expertise in inferential stats / acoustic analysis
• familiarity with phenomenon
• estimated quality of analysis by external reviewers
• preregged? (binary)
• did analyst go back to acoustic analysis after starting statistical analysis? (binary)
• Numbers of DVs, corrected for multiple testing?
• Random effect specifications? (I think the data allow for random slopes for subjects only, but items are a relevant random intercept variable)

--> many predictors will make collinearity an issue (need to check Tomaschek et al. 2018)

Sample

• Sample size justification (sample = # of analysis teams)? (might not be relevant if we avoid testing hypotheses explicitly; could easily take some of the variability observed in the other papers and run a power simulation tho)

Model

• Bayesian multilevel analysis (logistic link for 1, gaussian link for 2)
• DV ~ Co-variates + (1 | teams)

I pushed changes from my and Timo's branch into the master branch, so this is a good time to merge from master into jcdev.

Action items

• Upload PDFs of surveys to OSF > Questionnaires > pdfs [@jvcasillas].

Discuss

• Experimental procedure timeline.

  • Maybe split the timeline in two so that we can interweave speaker and confederate into a single line.
  • Or find other alternatives to make the timeline clearer.
  • Keep and make clear.

• Sample size justification.

  • We might leave it for now as is or hedge it like "we will aim at as many teams as possible, although we estimate a necessary minimum of 12" or the like. Or even we can take it out and just state that we will make sure to maximise number of teams or something.

• Ask about ask about semantic atypicality.

• Make new component on OSF.

• Create consent form.

• Timo will send us the presentation file of the video and we can have a look at the script to see if/what we can simplify a few parts.
• Read through docs/ExperimentalDetails to see if we can make it simpler and check experimental procedure against what is in the presentation video.
  • Maybe write a shorter Experimental Details doc? and link to extra info on OFS or the like.
• Research Question: move focus from "referring expression" (i.e. the green banana) to "utterance"? We don't want to bias the analysts to focus on the noun only.
• Check Joseph's google forms and docs (links here #1).

Possible phases of completing the RR. Feel free to add, remove, edit, etc.

• create RR template in Rmd
• outline sections
• determine writing responsibilities
• complete draft
• Abstract.
• Cover letter.

Let's collect our ideas for exploration here:

Interesting covariates:

• quality (from reviewers)
• posthoc changes to acoustic analysis = either binary or count
• self-reported RDF exploitation = count
• acoustic dimension (f0, int, dur, ...) = categorical
• temporal window (segment, syllable, word, phrase) = categorical
• Uniqueness of model specification or acoustic analysis via Soerensen = continuous
• Conservativeness of model (number of random effect parameters) = count/continuous
• exclusion of data = binary
• demographics of analysis teams
  • experience (maybe time from PhD? can be negative) = continuous
  • initial belief in effect = continuous
• frequentist vs Bayesian?
  ...

Possible exploration ways

• logistic regression on dichotomous inferential conclusion (there is an effect vs. nope)
• random forest classification yielding a relative ranking of predictor
• visual map, e.g. uniqueness of acoustic measurement against model conservativeness against effect size

• make list of possible (RR compatible) and desirable venues
• decide on venue to submit

Action items

• RR [@stefanocoretta]
  • Add data description.
  • Update peer-review section.
  • Update links.
  • Edit simulation.
• Update OSF [@troettge @jvcasillas]
  • READMEs.
• Update per-review Goggle form: use 0-100 scale in all Qs [@jvcasillas]

Discuss

• Peer review (#44).
• Questionnaires.
• Clean-up Gdrive.

Action items

• RR
  • Edit section on reviewers' ratings.
  • Simulate ratings and update appendix.
  • Upload data to OSF and WIki.
  • F.A.Q.?
  • Detailed workflow (Timo)

Discuss

• Peer-review survey.
  • Rating for acoustic and statistical analysis: separate or together?
  • For the other criteria text or numeric scale?
• Divergent transitions in simulation of factors model.

logo proposal. dunno... see here.

🌌 - Project status overview.

Action items

• RR [@stefanocoretta]
  • Update peer-review section.
  • Update links.
  • Add comments on parts that defo need editing.
  • Edit simulations.Rmd.
• Update peer-review survey (see #44 comments).
• Update OSF [@troettge @jvcasillas]
  • READMEs. Also see #6.
• Clean unused files on GDrive [@jvcasillas]

Discuss

• Comments in RR draft.
• Original "Blue bananas" study details to be included in the RR (as in related links/publications/other).

• analysis team
• reported effect sizes: effects reported by the analysis teams.
• standardised (refitted) model: refit of analysis team's model in Bayesian.
• standardised effect size: effects as returned by the standardised model.
• Bayesian random-effects meta-analysis and meta-analytic(al) model
• analytic(al) and demographic predictors (formerly factors).

First attempt, but I'm struggling with the we design. Ill ask Liz later. Have a look and tell me what you think. There is still a lot of text, I'd love to reduce this by half or so. The buttons are not yet clickable. I envision leading to the agreement survey and to an email address if there are questions.

@jvcasillas @stefanocoretta

• https://www.reddit.com/r/statistics/comments/fouwp9/q_effect_size_calculation_for_brms_delta_t/
• https://www.barelysignificant.com/phd_thesis/appendix-brms.html#inference-and-conclusions
• https://www.researchgate.net/publication/250185702_Effect_Sizes_in_Cluster-Randomized_Designs

AGENDA for meeting at 20th Jan:

[] Discuss landing page (Timo)
[] Discuss analyst journey and experience (Timo)

I am done for now. I won't make any more changes until after our meeting tomorrow. 🙂

Qs regarding dataset:

All materials are right now on a privat OSF. Relevant files are

• the sound files (duh). Pretty big file folder (30files à 100+MB)
• relevant textgrids that we generated according to our prereg (see Q below)
• experimental files for two experiments: 1. A norming study to norm the typicality of modifier plus noun (e.g. blue banana) and 2. the production experiment. For 1., there is a github repository from a student of Michael who programmed that one.
  For 2. we have images and trial lists in the OSF repo and I'm sure Simon has the code for the browser-based experiment somewhere.

I think we should make all of this available to the analyst. It would be potentially helpful to make a screen recording of both the norming study and the actual experiment so that analyses teams can watch them and get a feel for the experiment.

Okay my Qs:

• How should we go about storing the dataset. OSF strikes me eventually as the better option.
• How should we go about the dataset per se. Maybe we should offer the analyses teams a text grid that at least labels relevant utterances and their experimental condition? What do you think?

List of similar papers:

• https://docs.google.com/document/d/12XlCX0UWKLH1RJ9NsSciWxXn4DD0oh4slqd_TmKjp0A/edit
• Bastiaansen et al., 2019 Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology
• Boehm et al., 2018 Estimating across-trial variability parameters of the Diffusion Decision Model: Expert advice and recommendations
• Botvinik-Nezer et al., 2020 Variability in the analysis of a single neuroimaging dataset by many teams
• Breznau et al., 2020 The Crowdsourced Replication Initiative: Investigating Immigration and Social Policy Preferences
• Dulith et al., 2019 The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models
  Functional Imaging Analysis Contest 2006
• Fillard 2011 Quantitative evaluation of 10 tractography algorithms on a realistic diffusion MR phantom
• Maier-Hein 2017 The challenge of mapping the human connectome based on diffusion tractography
• Salganik et al., 2020 Measuring the predictability of life outcomes with a scientific mass collaboration
• Schweinsberg et al., 2020 Radical dispersion in estimates when independent scientists operationalize and test the same hypothesis with the same data
• Silberzahn et al., 2018 Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
• Starns et al., 2019 Assessing theoretical conclusions with blinded inference to investigate a potential inference crisis
• van Dongen et al., 2019 Multiple Perspectives on Inference for Two Simple Statistical Scenarios

Other stuff:

• ecology repo: https://osf.io/e5xhm/

@jvcasillas @troettge I need your input on these items (we can discuss on Wed).

• 1.2 Crowdsourcing alternative analyses
  • "Sum-up of Neuroscience Cognitive modeling Clinical Predictive models"
  • I am not sure what I should add here.
• 2.4.1 Descriptive statistics
  • "We anticipate that the majority of statistical analyses will be expressible as a (generalized) linear regression model." ADD FORMULA
  • Is this just the general glm formula?
  • 
• 2.4.2 Meta-analytical estimation
  • Which predictors are we gonna include in the meta-analytical model?
• Is a CC-BY license for the data ok?

(linking the meeting minutes: #9)

Ok, this one should work. I made some light edits to the RR and the experimental design .Rmd files. I also put each sentence on its own line so that it is easier to track changes moving forward. The RR also includes a second bib file for references (separate from the one papaja generates automatically for R packages). I've only added two refs so far, but I wanted to get it started.

Action items

• Continue with RR updates.
• Create org, website (@jvcasillas).
• Import issues from old repo.

To discuss

• many_analyses ported to many-speech-analyses org (issues not imported)
  • Imported issues with the GitHub REST API.
• (B4SS workshop: learnB4SS/learnB4SS#16).
• RR
  • Reviewers' ratings: mean + SD.
  • Move formulas to appendix.

We need to decide licensing for data code, teams code/data, etc.

Licensing for the production study data depends on the consent and on what you guys (of that project) prefer.

@jvcasillas @troettge

bib updates (about 90% done)

Merge changes after OSF repo restructuring.

Just a ping that I restored the master branch with work that was removed. If you have changes in your local repo, just to be on the safe side, stash them, and then do git pull --force. Then you can pop the stash and continue working.

@troettge @jvcasillas

Ok, this bad boy has moved to https://github.com/many-speech-analyses/many_analyses

Adjust workflows accordingly. I won't delete this repo until after we talk this week.

@stefanocoretta @troettge

Pros and cons

Action items

• RR [@stefanocoretta]
  • Edit based on last meeting's discussion.
  • Update links.
  • Edit simulations.Rmd.
  • References: parker et al (eco), summerfield, charles et al [@troettge @jvcasillas].
  • Info on consent form [@troettge].
• ForkinPaths.png [@troettge]
  • Connect panels in A.
• Upload .svg versions of images. [@troettge]
• Update peer-review survey (see #44 comments).
• Update OSF [@troettge] #6
  • Upload wav file to Data > production > audio.
• Clean unused files on GDrive [@jvcasillas]
• Prep list of terms to be discussed [@stefanocoretta]. #51

Discuss

• Audio data. #6
  • eudat.eu
  • We will upload the audio data when the time comes to share them with the teams. In case reviewers will request to listen to them, we will share them directly.
• Terms for models, demographic, analytic/al. See #51
  • What we have is fine, but we need to think of something better for "demographic" (this includes research expertise and prior belief). I will also make sure that there is consistency of use of analytic(al).
• Sample size justification.
  • We might leave it for now as is or hedge it like "we will aim at as many teams as possible, although we estimate a necessary minimum of 12" or the like. Or even we can take it out and just state that we will make sure to maximise number of teams or something.
• Ask teams if they know about papers.
  • Decide on when and where to ask.
  • At the end of the analyses and peer review.
• PDFs of surveys instead of links to them?
  • We will upload pdf versions of the surveys to OSF and link to those in the RR.
• Original "Blue bananas" study details to be included in the RR (as in related links/publications/other).
  • @troettge I'd like to be more explicit about the fact that the data comes from a prev study and was not designed for this one specifically. Is there any info that would help doing this?

I am done for now. I won't make any more changes until after our meeting tomorrow. 🙂

Action items

• Write summary of prior meta studies (@troettge #31).
• Check missing references in RR (@troettge).
• Check https://journals.sagepub.com/home/amp for tex template (@jvcasillas).
  • Guidelines.
    • Description
    • Note: Can't find LaTeX template (JVC)
• Landing page, copy text to gdoc (@troettge ).
• Analysts journey (all).
  • Think about how to divide in steps (conceptually and visually).
  • Think about possible icons.
• Continue RR draft (@stefanocoretta).
  • Edited the factor analysis section (still WIP).
• Continue OSF prepping.
  • Prepared TextGrids.

Discuss

• Timo will work on the landing page.
• Structure of Reviewers questionnaire.
  • @jvcasillas will do it. (ecoRR reviewer https://osf.io/4t36u/) (initial draft is in googledrive: many_analyses_peer_review_survey)
• Check transcription of objects and colours.

As it currently stands, there is a mismatch between the text in the RR and the survey. But a few general questions:

• Does the rating of the overall analysis include both the measurement and the statistical parts?
• Are all ratings supposed to be a number between 0-100? Or are all ratings except the one of the overall analysis supposed to be text?
• How are we gonna use the text responses?

@troettge @jvcasillas

• Data dictionary.
• Update Data component Wiki. @jvcasillas @troettge
• Create an updated version of the trial lists to include:
  • typicality of AN combo (categorical)
  • mean typicality rating of AN combo
  • sd of typicality rating of AN combo
  • trial number
  • sentence
  • english glosses
• Make an infographic of the data sources. @troettge are we doing this? @troettge
• Update OSF with osf/manage-osf.Rmd. @jvcasillas @troettge
  • That uses osfr. You need to create a PAT and add it in a file called .secrets in your home folder with the syntax
    Sys.setenv(OSF_PAT = "<your-pat-here>")
  • Create missing Readme.Rmd file.
  • Upload Readmes and data to OFS by running/rendering manage-osf.Rmd.

All css and image links are still pointing at the simplpoints website

Action items

• Port questionnaires to Google Forms (@jvcasillas, #8).
• Continue work on RR draft (@stefanocoretta).
  • Made several edits, mostly in Step 4 section (still WIP).
• Edit ExperimentalDetails.
  • Restructuring of sections.
  • Reformulation and extension.
• Edit forking paths figure (@troettge).
• Prepare .bib file for RR (@jvcasillas).
• Prepare graphics for analysis workflow (all) to be given to teams as aid (#26).
• Update OSF Data component (@stefanocoretta).
  • Started prepping of TextGrids.

Discuss

• RR
  • 💡Idea: would be nice to use a measurement-error model also in the analysis-factors analysis, so that we don't just use a point-estimate of the deviation from the meta-analytical mean.
    • GO.
  • How will we enter the demographic factors in the factors analysis since we have a score per individual rather than team?
    • Averaging scores.
    • Scale 1-100 belief.
  • Help with summarising neuroscience, cognitive modeling, clinical, predictive models in RR.
    • Timo will do it @troettge.
  • Discuss missing/ambiguous refs in RR
  • Use of papaja in RR draft (#28).
• Discuss landing page (Timo)
  • A few things should be highlighted: co-authorship, improving standards.
• Discuss analyst journey and experience (Timo)
  • In RR and landing page.
• Materials
  • Info on German determiner and adjective ending for prepping TextGrids.
• Project management
  • GH project Meetings.
  • GH Actions (assign label meetings and project Meetings automatically if issue title contains "Minutes").

@troettge Timo, could you let me know how the typicality categories were created? I just need to know if the mean or the median (or else) was calculated for each Adj/N pair, so I can include that in the trial lists (now it has mean, sd and median).

Current best option is Timo's German production data (OFS repo).

Include in comments other possible datasets.

Action items

• Landing page @troettge: #35.
• Reviewers questionnaire @jvcasillas: draft in gdrive.
• Check transcription of objects and colours.
• Continue RR drafting @stefanocoretta.
• OSF prepping @stefanocoretta: #16.
  • Corrected TextGrids IPA transcription (@troettge would be good if you could check here and here).
  • Added extended trial lists (with additional columns).
  • Added typicality ratings from the norming study, including a summary of it. [NOTE: there were encoding issues in the original file, for Moehre and Waescheklammer, which I fixed]

Discuss

• R scripts in scripts/r/.
• survey edits (https://github.com/jvcasillas/many_analyses/issues/1#issuecomment-771687908)

Points to discuss on the questionnaire for the collection of the analytical approach:

1. S et al have included a box for pasting a script (if any). Do we want that or do we want to collect scripts differently?
2. S et al have a question about their research questions and ask how likely the teams think that soccer referees tend to give more red cards to dark skinned players (and there's a Likert scale from very unlikely to very likely)? How could our question about prosodic variation be formulated?

See here for list (not exhaustive) of materials of we'll need.

List of necessary materials

• OSF repo (Timo).
  • Main repo.
  • Data component. #6
• Project timeline/Gannt chart (Joseph) (added to figs here; via https://github.com/jvcasillas/many_analyses/commit/2ca00a15ffa103e25ddef1093391ecd4c3c79e50 and https://github.com/jvcasillas/many_analyses/commit/722dc05b9495ee51e44f23d5a7b2eca8fac0767f)

STEP 1: recruitment

• Project description: draft / landing page (Timo)
• Experimental details.
• Recruitment form: (example) (Joseph draft)
• Intake form: (example) (Stefano draft)
• Consent form https://osf.io/xyp68/ (Joseph draft)
• Registration email (see Silberzahn Suppl (SS), p. 53): sent at recruitment. (Joseph draft)

STEP 2: primary data analyses

• Dataset:
  • Curate dataset (@stefanocoretta).
  • Data dictionary (variables description) (@stefanocoretta).
  • Detailed description of experiment (plus norming study): here.
  • Screen Recording of experiment (plus norming study)
• Analytical approach collection email (SS p. 53). (Stefano)
• Analytical approach collection questionnaire (SS p. 54). (Stefano, #8)
• Familiarity with analytical approach questionnaire (SS p. 72). (Joseph draft)

STEP 3: peer reviews

• Peer review survey of final analytical choices (SS p. 74): draft.

STEP 4: evaluate variation