Comments (3)
I actually simply used "gpt-3.5-turbo" without truncation. I didn't find it making a major issue. Would you mind sharing which document contains too many tokens.
I think truncation would be fine if there are two many tokens.
from aggrefact.
New File.txt
I tried to analyse the issue a bit and prepared a short script so that you can easily approve my observations.
I analysed the token length of the documents using tiktoken.
(Better would be to analyse the created chats, but that is harded as the response has to fit within the token limit as well. Also chatgpt is non deterministic. So this would need sending a couple of request to the api and so take some time.)
I set the threshold for the token length to 3000 so that there is buffer for the summary, the prompt and the response.
import pandas
import tiktoken
enc = tiktoken.encoding_for_model("gpt-3.5-turbo")
df = pandas.read_csv("../Code/human_alignment/AggreFact-main/data/aggre_fact_final.csv")
df["length"] = df.apply(lambda row: len(enc.encode(row["doc"])), axis=1)
df_long_docs = df[df.length > 3000]
print("number of problematic entries:", len(df_long_docs))
print("the indexes:", df_long_docs.index.tolist())
long_texts = set(df_long_docs["doc"].tolist())
print("number of unique problematic docs:", len(long_texts))
print(long_texts)
which prints
number of problematic entries: 5
the indexes: [73, 567, 4437, 5060, 5132]
number of unique problematic docs: 1
{{'Here are some of the clips from our interviews hosted by Andrew Neil and Jo Coburn, with films ...'}
(I shortend the text here to not blow the message. The full text in the attached file)
So it seems this is not a big issue indeed as there is only one really long doc. And 5 out of 10k entries is also no deal.
from aggrefact.
Thanks for the analysis! I think you could simply truncate these problematic entries and this not not affect the final result a lot!
from aggrefact.
Related Issues (14)
- Dataset Release HOT 1
- Reproduction Scores HOT 1
- Go Figure not included?
- Integrating human faithfulness ratings from HELM paper. HOT 1
- New paper version HOT 1
- Converting of chatgpt responses. HOT 1
- Sentence level dataset HOT 2
- 请问这些数据是怎么来的呢? HOT 1
- What are BertExt, BertExtAbs, and BertSum? HOT 6
- What is PegasusDynamic in your data? HOT 3
- Summarizer `missing` HOT 1
- Release of AggreFact Unify dataset HOT 4
- Question regarding the difference between Table 3 and 4 in the paper HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aggrefact.