Chatgpt based metrics and long inputs. about aggrefact HOT 3 CLOSED

I actually simply used "gpt-3.5-turbo" without truncation. I didn't find it making a major issue. Would you mind sharing which document contains too many tokens.

I think truncation would be fine if there are two many tokens.

from aggrefact.

New File.txt
I tried to analyse the issue a bit and prepared a short script so that you can easily approve my observations.

I analysed the token length of the documents using tiktoken.
(Better would be to analyse the created chats, but that is harded as the response has to fit within the token limit as well. Also chatgpt is non deterministic. So this would need sending a couple of request to the api and so take some time.)

I set the threshold for the token length to 3000 so that there is buffer for the summary, the prompt and the response.

import pandas
import tiktoken

enc = tiktoken.encoding_for_model("gpt-3.5-turbo")

df = pandas.read_csv("../Code/human_alignment/AggreFact-main/data/aggre_fact_final.csv")

df["length"] = df.apply(lambda row: len(enc.encode(row["doc"])), axis=1)
df_long_docs = df[df.length > 3000]
print("number of problematic entries:", len(df_long_docs))
print("the indexes:", df_long_docs.index.tolist())
long_texts = set(df_long_docs["doc"].tolist())
print("number of unique problematic docs:", len(long_texts))
print(long_texts)

which prints

number of problematic entries: 5
the indexes: [73, 567, 4437, 5060, 5132]
number of unique problematic docs: 1
{{'Here are some of the clips from our interviews hosted by Andrew Neil and Jo Coburn, with films ...'}

(I shortend the text here to not blow the message. The full text in the attached file)
So it seems this is not a big issue indeed as there is only one really long doc. And 5 out of 10k entries is also no deal.

from aggrefact.

Thanks for the analysis! I think you could simply truncate these problematic entries and this not not affect the final result a lot!

from aggrefact.