Many have asked for a way to reset the simulation - which is currently not possible

https://rpsychologist.com/d3/ci/

I asked students from a PhD course to look at the p-value app and they were very confused by the p-hacking tool. I am sharing their feedback in the hope it helps with the development. I think the confusion stems from trying to achieve two separate demonstrations.

1. the problem of selective reporting: if only 'statistically significant' studies get published, the average effect size of those studies is biased upwards. This bias is more acute if the sample size per group is low (parameter 1) or if Cohen's d is close to zero (parameter 2).
2. The p-hacking tool (a misnomer?) is used to investigate the effect of sequential testing under which data are added until the study achieves statistical significance. In this instance, the bias is due to lack of acknowledgement of the stopping rule. The p-value will decrease as sample size increases, but the average effect size would be weighted by sample size (I arguably did not check if it was the case).

• My students were confused by the reported sample size of the studies for p-hacking (which is that of the largest study): it wasn't clear what it meant.
• The values of Effect size (true) and Effect size (sim) were rounded to one digits, while Cohen's d can be modified in increments of 0.01.
• With a large number of draws and large number of samples, the app lags.

The current control card is just a placeholder to implement the app logic.

The UI/UX is especially bad on mobile.

This is a really handy resource.
My concern is that it gives the impression that the correlation coefficient is tied to the gradient, since the gradient becomes steeper the higher the correlation you enter.
I think this distinction is confusing for students, myself included.
Might it be possible to have the correlation vary but the gradient remain the same?

The Cohen's d in the Common Language Explanation doesn't seem to update. It remains .8, regardless of changes to d.

Hey! So I teach (University) biology and science generally and I LOVE your visualizations, particularly the NHST one. I use them in every class, often with assignments for students to play around or figure something out.
Here is my 'issue': students need a lot of guidance to interpret the graph, in part because the axes are not defined or explained. I guess it depends on what audience you are aiming at. But in biology/most sciences z values are not common knowledge, certainly at the student level, and frankly the undergrads still struggle interpreting any graph.
My suggestion is to (a) label the axes, even briefly; (b) add 1-2 sentences in the explanation about what the axes show; and (c) generally stating a few things more explicitly, such as that Cohen's d is a measure of effect size but is not simply the difference between the means (since it's a z score).

I am thinking of making a short video lecture explaining all this and using your visualization - would you like me to share it with you if I do?

Finally I'd like to add, just for fun, that I believe NHST is indispensable to science (why: the short version is that because lacking a clear yes/no answer from a test, people will always bend the conclusion to their will)... but that's a different can of worms ;-) . I certainly agree with you that too few people actually understand it and that this is a problem.

Again THANKS SO MUCH for creating these, they are awesome.
Anna

Here's a video showing the broken thing (and a workaround):
https://youtu.be/20N-O8r0GnM
I'm using Chrome Version 105.0.5195.102 (Official Build) (arm64) on MacOS Monterey 12.3.1

I think the url is wrong?

Under the Questions header on your About page, "wan't" should read "want".

More examples could be added to the ES preset drop-down. Preferably, effects that are illustrative and robust.

Would need this info:

  {
    preset: "small",
    label: "Small", // label displayed
    M0: 100, // Mean first group
    muZeroLabel: "Control", // Label first group
    M1: 103, // Mean second group
    muOneLabel: "Treatment", // Label second group
    SD: 15, // SD
    d: 0.2, // Cohen's d
    xLabel: "Outcome" // x-axis label
  },

The behavior of gatsby-theme-rpsych-viz/src/components/SettingsInput.js is unintuitive. Currently any new values need to be submitted by hitting enter - which many users miss.

Possible solutions:

• auto-submit any changes. This behavior was used previously, but it was also annoying.
• auto-submit but debounce while users type
• add label under the field to "hit enter to submit" if there's unsubmitted changes
• add a clear submit button that will submit all fields, and clearly show if there's unsubmitted fields

see
https://twitter.com/Birkelbayes/status/1367404567365906433

Clicking share should give the correct URL even if embedded at another domain

https://twitter.com/ConstantPieters/status/1367776314179076097

Great project, love it!

Just a minor issue I wanted to report :
The user interface disappears when the number of observation is decreased below 1

I have an R version 4.2.2 on a MACOS. I try to download the package ("powerlmm") bu the answer is "Warning in install.packages :
package ‘powerlmm’ is not available for this version of R

A version of this package for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages"
I tried with the code :devtools::install_github("rpsychologist/powerlmm") but this also was uneffective.
Can you help me (step by step) ? thank you very much

Hej,

Blev lite förvirrad när jag läste detta:

och det är en 92.1% chans att en slumpmässigt utvald person från "kontroll"-gruppen kommer ha en högre poäng än en slumpmässigt utvald person från "behandling"-gruppen (probability of superiority). För att få ett mer positivt utfall i "kontroll"-gruppen jämfört med "behandling"-gruppen behöver vi behandla 1.5 personer i genomsnitt.

Det ska väl vara tvärt om?

Hi
When Cohen d =0, there is an explanation like that: "Moreover, in order to have one more favourable outcome in the "treatment" group compared to the "control" group, we need to treat 16.5 people on average. " it is seen as NNT= 16.5 on the graph. Then again, we calculate the number of people in the experimental group, ex. 20+6.1. This is confusing, at least mine :).
Is 16.5 a raw number or a percentage, because the numbers you use in the whole process are percentages, not raw points.
Thank you

ARR=(Control event rate)− (Experimental event rate)
ARR=0.2−0.3651=−0.1651
NNT=1/ARR
NNT=1/−0.1651=−6.1

Currently the "rescale" button will scale the axes to +/- 4 SD around the means. It should be possible to set the limits manually in the settings drawer.

I try to use the Cohens'd visualizer, but I can't put my actual means in the settings: I'm forced to center around the value of 100.
Is this normal? I don't understand how it works.

Under the "about the visualization" heading on the interactive Bayesian inference page, "which I use here as to compute Bayes factor" should read "which I use here to compute the Bayes factors" or similar.

When I adjust the cohen's d using only the arrows, it does not apply to the random experiments' outcome.
When I adjust it using the slider, it works as expected

vs.

Anyone is welcome to help translate the contents of https://rpsychologist.com/cohend

See docs/translations.md for more info on how to translate the contents.

Ongoing

• Portuguese-Brazil #17
• German #13
• Vietnamese #19

Finished

• Swedish
• French #16
• Spanish #15
• Norwegian #14
• Uzbek #50

Minor issues

• Settings panel: color labels might overflow, add word-break?
• Sort languages alphabetically in header menu?

Hi Kristoffer

Your Cohen's D interpretability machine/visualiser is great - I'm trying to use it to make the results of my meta-analysis more understandable. The explanations of formulas used are really handy, too. I've two questions which I'll post as separate issues, the second of which is:

Would it be possible to direct users to source code (preferably R but if it wasn't written in R no worries) for generating the text in the 'A Common Language Section' more generally? I can't find it easily here and this would be really handy especially for generating common language explanations for data outside of the ranges that your tool currently accepts (e.g. dealing with log-scale data)?

Thanks!
Matt

Hi Kristoffer

Your Cohen's D interpretability machine/visualiser is great - I'm trying to use it to make the results of my meta-analysis more understandable. The explanations of formulas used are really handy, too. I've two questions which I'll post as separate issues, the first of which is:

Could you please write up an explanation of how the last sentence ('This means that if there are 100 people in each group, and we assume that 20 people have favorable outcomes in the "control" group, then 20 + 12.9 people in the "treatment" group will have favorable outcomes.') is computed please?

Thanks!
Matt

Great project, I use it all the time!

Steps to reproduce:

• load https://rpsychologist.com/correlation/
• click Settings then the Anscombosaurus preset
• Everything looks fine
• Now change to the "Small" (r=0.1) preset

Results: r and r2 are incorrect as shown here:

It looks like there is a point in the upper left of the data set which did not get reset?

First, I love the visualizations you've made. Thanks for sharing them.

Shouldn't the example below be labeled as "inferior"? The confidence interval is fully to the negative side. https://rpsychologist.com/d3/equivalence/

A similar thing happens on the superior side.

I think in the first paragraph, this snippet
"as unintuitive and as misunderstood p-values"
should read
"as unintuitive and as misunderstood as p-values"

(would have created a pull request but did not find the text in the repo)