[BUG] GRACE editor with large eps. about easyedit HOT 9 CLOSED

Hi. @pengzju You are right, this is what GRACE is supposed to do, however, the code does not act like that. Let's talk about this issue step by step. This issue happens with model: llama2. You can reproduce this error with your default hparams in ./hparams/GRACE/llama-7B.yaml but just change eps to 100 or more, like 1000 or 10000. The query key changes in a single prompt if you just use one picec of data to edit. For llama model in a single generation step, when generating answers, the query variable changes in multiple forward(self, *args) calls from GRACEAdapter(torch.nn.Module). For the dataset you provided in ./data/portability/One Hop/zsre_mend_eval_portability_gpt4.json, the rewrite acc drops from 0.9916 to 0.3273. Please check this out. Thank.

from easyedit.

Thanks!

from easyedit.

Hi, I don't quite get what you mean. From my experience, increasing eps doesn't lead to a decrease in rewrite acc because a larger deferral radius should result in a higher hit probability. Additionally, could you please explain in detail what you mean by "The key used to query in the codebook changes"?

from easyedit.

hi, could you please provide more details of your questions?

from easyedit.

I have fixed this issue (38c5c34), please update EasyEdit and try again

from easyedit.

Hi, do you have any further questions?

from easyedit.

Hi. The problem solved. Thanks for your help. I may close this issue.

from easyedit.

Hi. I need to reopen this issue. I just tested the new version. The editing speed is 5-10x slower then the previous version. Is this a normal phenomenon? By the way, does GRACE here support batch editing?

from easyedit.

Q1: It's normal. I just set n_iters from 100 to 50. In addition, increasing eps will reduce some fine-tuning costs because more Keys are shared

Q2: Nope. plz refer to GRACE paper: https://arxiv.org/abs/2211.11031

from easyedit.