Paraphrase prompts' format not compatible with the sample from ROME paper about memit HOT 3 OPEN

I do not know the exact details of this, but I can give you some insight. As you may know, when inserting the knowledge of prompts both methods, ROME and MEMIT, add some "noise" by adding previous tokens to the subject. They claim that:

Because the state will vary depending on tokens that precede s in text, we set $k^*$ to an average value over a small set of texts ending with the subject s.

The way in which these tokens are added is easy, but they changed it between ROME and MEMIT. While in ROME they sampled 20 texts to compute the prefix: ten of length 5 and ten of length 10, when looking at the MEMIT code there are only 5 samples of length 10. I have not found the details of this anywhere, but it seems like they have been experimenting with adding different types of noise tokens to see which one was better.

I do not think that it was an error. The reason for generating these paraphrased sentences is to test the model's ability to handle the same information presented differently. If they are deliberately adding variations to the paraphrased prompts and still obtaining good results in their evaluations, it suggests that their approach is capable of maintaining its performance even when introduced to such variations, demonstrating its robustness.

I hope it helps you a little. If someone knows the exact impact of the evaluation procedure, I will be glad to hear about it too.

from memit.

Thank you for your answer! Yes you may be right, they probably added these random sentences to account for the variable contexts in which these prompts can appear. But then why not adding them for the neighborhood prompts? the same reasoning applies to them too.

I hope we will hear more about what is the impact of these random sentences on the evaluation.

from memit.

I hope we will hear more about what is the impact of these random sentences on the evaluation.

Me too.

from memit.