jef1056 / clean-discord Goto Github PK

Thanks for your great work. It's really very educational for me.

I have a problem with the split.py script though.

I did use DiscordChatExporter and retrieved about ~400k messages.
Then I used clean.py, here is the output :

(regex): Writing data to clean
Regex cleaning...: 100%|| 3/3 [00:18<00:00,  6.01s/it]
(detox): Writing data to clean-detox
Detoxifying...: 100%|| 3/3 [00:17<00:00,  5.69s/it]
DONE

{"messages_total": 415473, "conversations_total": 23072, "removed_total": 87036, "num_files": 3, "num_channels": 1 [...]

Then I used the split.py on my "cleaned" folder but I got an empty train file :

Train size: 0 files	Val size: 3 files
Files will be compressed.
Writing train...: 0it [00:00, ?it/s]
Writing val...: 100%|| 3/3 [00:02<00:00,  1.35it/s]
Merging files...: 100%|| 3/3 [00:04<00:00,  1.40s/it]

What am I missing ?

changing line 74 of workers.py from
if msg!=[]: temp["conversations"].append(msg.strip().replace("\t", " ")); msg=[]
to
if msg!=[]: temp["conversations"].append(str(msg).strip().replace("\t", " ")); msg=[]
fixed this