[P1] Confirmation of alpaca_eval version about pyreft HOT 4 CLOSED

I am closing this issue; feel free to reopen if you have more questions! ty!

from pyreft.

@BaohaoLiao unfortunately, these results are removed locally from my disk -- but it should be very easy to reproduce. i think you just need to follow the command we give in the README. let me know if you encounter any issues.

from pyreft.

@BaohaoLiao hey thanks for your question! the version here is okay. You can install the newest version.

Alpaca-Evalv1.0 means we evaluate against text-davinci-003, not against GPT4. You can follow this to do the evaluation:
https://github.com/stanfordnlp/pyreft/tree/main/examples/loreft#offline-evaluation-with-alpaca-eval-v10

Alpaca-Evalv2.0 uses GPT-4. To be competitive with v2.0, please train with longer sequence length. We are training with max sequence length of 768, this is too limited. We have to do it to keep a fair comparison with previous works.

from pyreft.

Thank you for the details. Is it possible to release the generated outputs for alpaca_eval? I.e. the output from Llama-2 7B & LoReFT (ours) in Table 3.

It would be great if you could also release the evaluation results from alpaca_eval.

from pyreft.