thepanacealab / smmt Goto Github PK

View Code? Open in Web Editor NEW

130.0 13.0 36.0 534 KB

Social Media Mining Toolkit (SMMT) main repository

License: GNU General Public License v3.0

Python 98.65% Shell 1.35%

annotation twitter-api data-annotation data-acquisition data-preprocessing spacy gathering tweets

smmt's People

Contributors

Stargazers

Watchers

smmt's Issues

Get empty output files after hydration

Hello,

I was trying to use get_metadata.py to hydrate the CSV file. I got four output files but they are all empty. It seems that my Twitter API credentials are valid as shown in the screenshot below but the program stopped at the first 100 rows. Does anyone know the possible reason for this issue? Thanks in advance!

Can't encode unicode symbol /u039f (greek letter omicron)

currently getting 1808100 - 1808200
creating ziped master json file
creating minimized json master file
creating CSV version of minimized json master file
Traceback (most recent call last):
  File "get_metadata.py", line 187, in <module>
    main()
  File "get_metadata.py", line 183, in main
    f.writerow([data["favorite_count"], data["source"], data["text"].encode('utf-8'), data["in_reply_to_screen_name"], data["is_retweet"], data["created_at"], data["retweet_count"], data["id_str"].encode('utf-8')])
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python37-32\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u039f' in position 16: character maps to <undefined>

Cannot unpack dailies

get_metadata.py can't continue empty output file

if an empty output file already exists, get_metadata.py throws an error due to:

SMMT/data_acquisition/get_metadata.py

Line 81 in c2a7c02

f.seek(-2, os.SEEK_END)

This exasperates Issue #12
Since a empty output file is written if no valid API token is added.

get_metadata.py can't handle output file with incomplete json

If get_metadata.py gets interrupted during hydration, a incomplete json object may be written to the output file.
If you try to restart the script, this leads to a json decode error.

Proprosed fix:
Check if last line of the output file is invalid, and remove it.
Example:
https://stackoverflow.com/questions/6685378/using-python-to-remove-incomplete-line-from-the-end-of-a-json-formatted-log-file

hydrate tsv file ISSUE

When i try to use get_metadata.py to hydrate tsv file, the result is empty. Why and what can i do to get the resuly? Grateful for your answer. Many thanks

Fix scraping vs hydrating language on tool

Change text to reflect actual meaning of tool functionality

thepanacealab / smmt Goto Github PK

smmt's People

Contributors

Stargazers

Watchers

Forkers

smmt's Issues

Get empty output files after hydration

Can't encode unicode symbol /u039f (greek letter omicron)

Cannot unpack dailies

get_metadata.py can't continue empty output file

get_metadata.py can't handle output file with incomplete json

hydrate tsv file ISSUE

Fix scraping vs hydrating language on tool

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent