Hi Joshua,

I am in the process of analysing your package and I am currently writing a blog in which I will explore the functionalities further.

I am using the package distill to write my blog. In doing so, I come across peculiar behaviour of the function check_hmc_diagnostics( ) when I knit my rmarkdown file. The results themselves are not readable in the html file. I do get the titles of the outputs and I also see the results in the console, but they are not printed in the html file. Is that something specific to the implementation of the function or does it have more to do with distill do you think?

Hi Joshua,

First of all, a huge thanks for this work you've done. I'm setting my own first steps in Stan and Bayesian modelling and packages like this are a great resource to learn.

In my research we've focussed on the use of the BTL model for comparative judgement in education. Now I want to implement the same model making use of your package in Stan. I will sketch the basic structure of the data. We had for instance 40 judges comparing short texts written by students. They only had to judge which of both texts met the criteria best. So I guess this resembles the unidimensional model you've described. Judges all had 20 judgements of random composed pairs.

The data is at comparison level, with following variables:

• textA the text presented left side of the screen
• textB the text presented right side of the screen
• textChoosen the text that 'wins' the comparison
• judge the judge that made the decision

Now I'm a bit puzzled how I have to feed this type of data into your framework. I hope this question is not 'too simple' and I hope this is the place where to ask these kind of questions.

Kind regards,

Sven

Hi Joshua

Me again. I have another question.

In the literature on Comparative Judgement in education scholars are used to report the Scale Separation Reliability. The formula is:

SSR = G^2/ (1+G^2) with
G = (True SD) / RMSE

More info on SSR at https://www.cambridgeassessment.org.uk/Images/232694-investigating-the-reliability-of-adaptive-comparative-judgment.pdf

Given that we can estimate the model with pcFactorStan it seems possible to also estimate the SSR (and a posterior for SSR). From each draw we can get the estimated True SD and the RMSE (although I would not know how to get that last one). Would it be possible to integrate this calculation in the generated quantities code block in the Stan code? What would be needed there? I think this is more efficient than doing it afterwards in R. But I lack sufficient skills in Stan to figure out the Stan code here.

Thx to consider helping me out here.

Sven

Currently, the model is parameterized in a way that the variance of objects approaches 1.0 for each item. That's the point of calibrateItems. However, I suspect this step is unnecessary. When I get time, I'd like to try to reparameterize the model such that the item discrimination is fixed to the standard normal cumulative function and object variance is allowed to vary.

Just so I don't forget, I decided on the current parameterization while struggling to get the thresholds to sample smoothly. The key to the thresholds was to parameterize them as a proportion. It seems like it is feasible to adjust the prior on the thresholds to cope with any object variance.