Please add flag to log score for each sample (akin to Eleuther's LM Evaluation Harness) about bigcode-evaluation-harness HOT 3 OPEN

If making this change is relatively simple, you can let me know how to make the change and I can open a PR?

from bigcode-evaluation-harness.

Hi, I encounter the same need.

I find that the code can be easily modified to log the per-sample score.

Take the task humaneval as an example, the second return value in compute_code_eval (called in process_result: link) will return the logs (including whether the output is passed) of sample as a list.

In the original implementation, they just use a _ to accept the returned logs. If you want to output the logs to a file, you can just save the value by json.dump.

from bigcode-evaluation-harness.

Yes as @shuhao02 explained and as I mention in this issue you can dump the results returned by code_eval metric #211 (comment)

results, details = compute_code_eval(..)
details.to_json("details.json")

Feel free to make a PR to add a flag for the harness, but note that not all tasks use this metric

from bigcode-evaluation-harness.