How do you calculate Efficacy Score for multi-token words in your counterfact dataset? about rome HOT 1 CLOSED

Hi! You can see how we do this in the eval code. tl;dr we multiply the probabilities, which is equivalent to summing logprobs: $$p(o) = \prod_{i \in o} p(w_i \mid w_{j < i}) = \exp \left( \log \prod_i p(w_i \mid w_{j < i}) \right) = \exp \left( \sum_i \log p(w_i \mid w_{j < i}) \right).$$

You'll notice that, in practice, we use negative log probabilities: $$p(o) = \exp \left( -\sum_i -\log p(w_i \mid w_{j < i}) \right)$$

Given $p(o), p(o^*)$, we can perform a direct comparison.

from rome.