kcentric / deep_nlp_on_sf_literature Goto Github PK

There was a large output text file when originally uploading the LLM directory, which caused upload to stop mid-process. Some files might have been left out; check and add them.

There is one (relatively minor, but significant) issue in list_of_dicts_to_list_of_lists functionality (GPT_NER_Round_1.py). Basically, the RoBERTa tokenization isn't exactly matching up to the sentences with the original sentences that I tokenized.

For example, a term like "desert cat" has one label ("CONCEPT") in the GPT-annotated data. When I tokenize for RoBERTa, it should have "desert" - CONCEPT, "Ġcat" - CONCEPT where the "Ġ" represents a space as represented by RoBERTa tokenizer.

Instead (not for "desert cat", I think but for a term right before it) the tokenization process is generating a CONCEPT label for "Ġcat" but not for "desert" (something like that). If you run GPT_NER_Round_1.py as a script (deny creation of new data when it prompts you for that) you should be able to see this issue in the console output.

• Change "main files" to Data Processing
• Add "Analysis/Visualization" with LDA and Visualization as subfolders
• Ultimately main.py should be the ONLY file directly in the main directory. So organization should look like, a lot of directories and one main.py sticking out clearly