duolingo / halflife-regression Goto Github PK

I am trying to understand the Python implementation with a view to port it to another language, but can't make sense of some variables.

• hlwt at line 37.
• dlp_dw at line 86.
• dlh_dw at line 87.
• slp at line 124.
• slh at line 125.

Could someone please explain what the acronyms for the above mean?

If I understood correctly, I see that pp in the result is the predicted probability of forgetting a word. So if pp is low, I should recommend a word associated with the pp as a user might have forgotten the word already. But from the result, I dont see associated word with pp. Then how can I recommend words to users?
And again, is my saying correct? (words with low pp values should be recommended)

Finally, I checked roughly the result and compared p (ground truth) and pp (prediction) and seemed to find that p and pp are quite not matching when they should. For example, when p is 0.0001 pp is 0.9999 and when p is 0.9999, pp is 0.0001. I saw this case quite often when I was skimming. But MAE shows it is very similar as your paper which means training is not broken. Is it normal that p and app are not so matching often? Should I still recommend repeating lexeme according to the pp values?

I have tried to reproduce your figure of 0.175 MAE for the fixed p-bar of 0.859. On neither the test set, nor the training set can I get this value. Could you detail for me how you have calculated it?

For context I am seeing if I can reproduce some of your results, or try various different modules on the same set of data. Firstly, I would like to reproduce your results as a baseline, but this one has eluded me.

Thanks.

Hi. First of all, thanks for making those codes public. While I was running the codes and data on my laptop, I found the experiment results are a little bit strange and counterintuitive. Though the recall error rate is quite similar to the results in paper, some other feature values really confused me:

1. I used the preset features for hlr: 'right' and 'wrong'. But when I read the model weights for these two features, it showed that both are negative (I have run the codes many times to confirm this). That means under the same circumstances, the more right answers you got from past history, the smaller half life you got, which means the more easily you would forget this word??

else: # fv.append((intern('right'), right)) # fv.append((intern('wrong'), wrong)) fv.append((intern('right'), math.sqrt(1+right))) fv.append((intern('wrong'), math.sqrt(1+wrong)))

2. When I used the model created in the experiment to predict all the half life of 13 million data, it shows that over 90% of them have a half life longer than 120 days? Does that mean averagely people need more than 4 months to likely forget a foreign word? This seems a little counterintuitive for me.

Thanks for making the great app even better and for uploading the code!

I am working on extending the model to allow each lexeme to have more features and have some minor doubts about the SGD implementation in the code:

1. The learning rate for the hlr case seems to differ from the method given in the appendix of the paper by a factor of 1 / (1 + inst.p). Is there any particular reason for that?

2. Usually, SGD iterations are repeated until some convergence criteria is met. In the code, however, it seems that only one pass is made on the complete data-set. Was that because empirically you observe that the results converge by then?

Thanks!

I have noticed the evaluation results for the half-life regression model are slightly different each time I run the code. So I was wondering how the evaluation results presented in the paper are determined.

Are the results given in table 2 of the paper based on one run or are they based on multiple runs (using the best result or the average or something else)?

Thanks!

@sueyhan previously opened a ticket for this, but then closed it without getting a response.

These are the model weights I get for training without lexical features (python experiment.py -l settles.acl16.learning_traces.13m.csv.gz):

wrong -0.2245
right -0.0125
bias 7.5365

I do not see how it can be correct that the right feature has a negative weight. This will cause the half life to get shorter as a user gets more correct answers, and therefore the model will predict a lower and lower probability of the user getting correct answers.

How can this be correct?

I am trying to run the experiment file, but I get the following error with the dataset that you have provided:

iterator should return strings, not bytes (did you open the file in text mode?) at line 217.

I am using python 3.7.6 and executing experiment.py with the following arguments:
python3 experiment.py data/settles.acl16.learning_traces.13m.csv.gz