Git Product home page Git Product logo

Comments (6)

zxlzr avatar zxlzr commented on May 29, 2024 1

Hi, have you solved your issue?

from easyedit.

tbozhong avatar tbozhong commented on May 29, 2024

Thank you for your attention!

To replicate the results of ROME, you can refer to examples/run_zsre_llama2.py to process the entire dataset into the parameter format accepted by editor.edit():

{
    'prompt': prompt,
    'target_new': target_new,
    'ground_truth': ground_truth,
    'subject': subject,
    'portability': {},
    'locality': {}
}

By doing so, you will be able to reproduce the ROME results.

from easyedit.

NamburiSrinath avatar NamburiSrinath commented on May 29, 2024

Thanks for quick response @tbozhong,

I have the JSON file zsre_mend_eval_portability_gpt4.json and here's the command I ran:
python run_zsre_llama2.py --editing_method "ROME" --hparams_dir "../hparams/ROME/gpt2-xl.yaml" --data_dir "../data/data"

Traceback (most recent call last):
  File "/hdd4/srinath2/EasyEdit/examples/run_zsre_llama2.py", line 78, in <module>
    metrics, edited_model, _ = editor.edit(
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/editors/editor.py", line 197, in edit
    "pre": compute_edit_quality(self.model, self.model_name, self.hparams, self.tok, request,
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/evaluate/evaluate.py", line 70, in compute_edit_quality
    compute_portability_quality(model, model_name, hparams, tok, portability_key,
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/evaluate/portability_evaluate.py", line 24, in compute_portability_quality
    portability_correct = test_prediction_acc(model, tok, hparams, prompt, ground_truth, device)
  File "/hdd4/srinath2/EasyEdit/examples/../easyeditor/evaluate/evaluate_utils.py", line 109, in test_prediction_acc
    if isinstance(answers[0], list):
IndexError: list index out of range

I can try to debug, but still sharing here so if there's something, you can update the codebase :)

Note: If my understanding is wrong and if you just meant to refer the code and write a similar-script for GPT-2 XL, then I am mistaken and will try to do the same!

from easyedit.

NamburiSrinath avatar NamburiSrinath commented on May 29, 2024

Here's some additional information if it helps, but I believe you would already know this, but this is there the code breaks

logits variable is --

tensor([[[ 5.1888e-02,  1.1745e+00, -2.8043e+00,  ..., -6.1696e+00,                                       
          -7.5409e+00, -8.9564e-01],                                                                      
         [ 4.1800e-01,  6.7916e-01, -2.7752e+00,  ..., -5.1078e+00,                                       
          -6.5594e+00, -9.9971e-01],                                                                      
         [ 3.7734e+00,  3.5277e+00,  1.8849e+00,  ..., -3.0128e+00,                                       
          -3.5799e+00,  3.5103e+00],                                                                      
         ...,                                                                                             
         [ 4.0497e-01,  2.9791e-01, -2.4308e+00,  ..., -3.1944e+00,                                       
          -2.9315e+00,  8.2657e-01],                                                                      
         [ 1.8392e+00,  2.6660e+00, -9.3549e-01,  ..., -6.4520e+00,                                       
          -6.0360e+00,  4.5731e-03],                                                                      
         [ 4.2277e+00,  3.5396e+00, -9.0947e-01,  ..., -5.9734e+00,                                       
          -6.8100e+00,  2.2497e+00]]], device='cuda:0')

The labels and answers are [], [] after it's passed to slice_left() in evaluate_utils.py. I believe there's an issue with data-preprocessing?

from easyedit.

pengzju avatar pengzju commented on May 29, 2024

same as #43

from easyedit.

NamburiSrinath avatar NamburiSrinath commented on May 29, 2024

Sorry about the delay, yes the issue has been resolved by adding this line

hparams.max_length = 100

from easyedit.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.