fakenewschallenge / fnc-1 Goto Github PK

Since the data is in CSV format and the train_bodies.txt file contains column values which span multiple lines, it may cause problems for many csv readers or for people using simple split of delimiter values. I have parsed the data in JSON format and also included a script which does the same. I think it will give an easy to use data for participants.

Will send a pull request.

It seems that test_bodies.csv is not encoded entirely in UTF-8. If you open it with an editor and check the bottom of the file (I used Atom), you'll see the missing characters. If you switch to cp1252 or some other encodings, you'll see some characters fixed and others break. What encoding should we use?

partially in ref to #3

do we want to have a library for data manipulation? maybe example implementations as other benchmarks / baselines?

cc: @delip