maraujo / pysocialwatcher Goto Github PK
View Code? Open in Web Editor NEWSocial Watcher on Facebook Marketing API
License: MIT License
Social Watcher on Facebook Marketing API
License: MIT License
Apologies if this is a very basic question, but I'm still learning my ropes with Python, and wanted to use this project. I carried out the installation instructions, and then initiated jupyter lab running a python 2 kernel to run the code. However, when I try to run watcherAPI, I keep running into the same error- "No Module named main". Any help would be greatly appreciated, I haven't been able to find any tutorial notebooks either, but I'm happy to write one once I'm able to do this.
I tried executing a basic query for NY ex-pats (JSON below) using utils.process_audience_from_response
but the query was unable to complete because the response did not contain a JSON. Do you know if the API has changed and is now returning non-JSON responses?
query:
{ "name": "Expats Distribution in New York",
"geo_locations": [
{
"name": "cities",
"values": [{"key":"2490299"}],
"location_types": ["home"]
}
],
"behavior": [
{"or" : [6019673233983], "name" : "Expats (Zimbabwe)"},
{"or" : [6027149006383], "name" : "Expats (Vietnam)"},
{"or" : [6026404871583], "name" : "Expats (Venezuela)"},
{"or" : [6023620475783], "name" : "Expats (US)"},
{"or" : [6019396649183], "name" : "Expats (United States)"},
{"or" : [6021354152983], "name" : "Expats (UK)"},
{"or" : [6019673501783], "name" : "Expats (Uganda)"},
{"or" : [6023516430783], "name" : "Expats (UAE)"},
{"or" : [6019366994183], "name" : "Expats (Turkey)"},
{"or" : [6019377644783], "name" : "Expats (Switzerland)"},
{"or" : [6023516315983], "name" : "Expats (Sri Lanka)"},
{"or" : [6019366943583], "name" : "Expats (Spain)"},
{"or" : [6027148973583], "name" : "Expats (South Korea)"},
{"or" : [6019564383383], "name" : "Expats (South Africa)"},
{"or" : [6023516403783], "name" : "Expats (Singapore)"},
{"or" : [6027149004983], "name" : "Expats (Serbia)"},
{"or" : [6023357000583], "name" : "Expats (Senegal)"},
{"or" : [6025000813183], "name" : "Expats (Saudi Arabia)"},
{"or" : [6025670492783], "name" : "Expats (Rwanda)"},
{"or" : [6025000815983], "name" : "Expats (Russia)"},
{"or" : [6027148962983], "name" : "Expats (Romania)"},
{"or" : [6019520122583], "name" : "Expats (Puerto Rico)"},
{"or" : [6021354882783], "name" : "Expats (Portugal)"},
{"or" : [6019396657183], "name" : "Expats (Poland)"},
{"or" : [6018797091183], "name" : "Expats (Philippines)"},
{"or" : [6027149008183], "name" : "Expats (Peru)"},
{"or" : [6016916394383], "name" : "Expats (Pakistani)"},
{"or" : [6027147189983], "name" : "Expats (Pakistan)"},
{"or" : [6018797004183], "name" : "Expats (Nigeria)"},
{"or" : [6023516368383], "name" : "Expats (New Zealand)"},
{"or" : [6023287393783], "name" : "Expats (Netherlands)"},
{"or" : [6023356955383], "name" : "Expats (Nepal)"},
{"or" : [6023516338783], "name" : "Expats (Morocco)"},
{"or" : [6023676072183], "name" : "Expats (Mexico)"},
{"or" : [6027147160983], "name" : "Expats (Malaysia)"},
{"or" : [6068613839383], "name" : "Expats (Latvia)"},
{"or" : [6018796980983], "name" : "Expats (Kenya)"},
{"or" : [6023676028783], "name" : "Expats (Japan)"},
{"or" : [6019396654583], "name" : "Expats (Italy)"},
{"or" : [6025000823583], "name" : "Expats (Israel)"},
{"or" : [6019396650783], "name" : "Expats (Ireland)"},
{"or" : [6027149017383], "name" : "Expats (Iran)"},
{"or" : [6019564344583], "name" : "Expats (Indonesia)"},
{"or" : [6016916298983], "name" : "Expats (India)"},
{"or" : [6019396638383], "name" : "Expats (Hungary)"},
{"or" : [6023676022783], "name" : "Expats (Hong Kong)"},
{"or" : [6059793664583], "name" : "Expats (Honduras)"},
{"or" : [6018797373783], "name" : "Expats (Haiti)"},
{"or" : [6019673808383], "name" : "Expats (Guatemala)"},
{"or" : [6023676017583], "name" : "Expats (Greece)"},
{"or" : [6019673448383], "name" : "Expats (Ghana)"},
{"or" : [6019367052983], "name" : "Expats (Germany)"},
{"or" : [6019367014383], "name" : "Expats (France)"},
{"or" : [6068209522983], "name" : "Expats (Finland)"},
{"or" : [6018797165983], "name" : "Expats (Ethiopia)"},
{"or" : [6023287351383], "name" : "Expats (Estonian)"},
{"or" : [6019673777983], "name" : "Expats (El Salvador)"},
{"or" : [6027148953383], "name" : "Expats (Egypt)"},
{"or" : [6019673762183], "name" : "Expats (Dominican Republic)"},
{"or" : [6018797127383], "name" : "Expats (Cuba)"},
{"or" : [6019673525983], "name" : "Expats (Colombia)"},
{"or" : [6019452369983], "name" : "Expats (China)"},
{"or" : [6025054896983], "name" : "Expats (Chile)"},
{"or" : [6019396764183], "name" : "Expats (Canada)"},
{"or" : [6018797036783], "name" : "Expats (Cameroon)"},
{"or" : [6019564340583], "name" : "Expats (Brazil)"},
{"or" : [6043702804583], "name" : "Expats (Belgium)"},
{"or" : [6023356562783], "name" : "Expats (Bangladesh)"},
{"or" : [6023675997383], "name" : "Expats (Austria)"},
{"or" : [6021354857783], "name" : "Expats (Australia)"},
{"or" : [6025000826583], "name" : "Expats (Argentina)"}
],
"ages_ranges": [
{"min":18}
],
"genders": [0]
}
error:
ValueErrorTraceback (most recent call last)
<ipython-input-24-33f40b778386> in <module>()
25
26 watcher.load_credentials_direct(FB_KEY,FB_APPLICATION_ID)
---> 27 watcher.run_data_collection(SETTINGS_FILE_NAME)
28 files.download('results.csv')
/usr/local/lib/python2.7/dist-packages/pysocialwatcher/main.pyc in run_data_collection(json_input_file_path)
177 PySocialWatcher.check_input_integrity(input_data_json)
178 collection_dataframe = PySocialWatcher.build_collection_dataframe(input_data_json)
--> 179 collection_dataframe = PySocialWatcher.perform_collection_data_on_facebook(collection_dataframe)
180 return collection_dataframe
181
/usr/local/lib/python2.7/dist-packages/pysocialwatcher/main.pyc in perform_collection_data_on_facebook(collection_dataframe)
142 print_info("Data Collection Complete")
143 save_temporary_dataframe(collection_dataframe)
--> 144 post_process_collection(collection_dataframe)
145 save_after_collecting_dataframe(collection_dataframe)
146 return collection_dataframe
/usr/local/lib/python2.7/dist-packages/pysocialwatcher/utils.pyc in post_process_collection(collection_dataframe)
327 print_info("Computing Audience column")
328 collection_dataframe["audience"] = collection_dataframe["response"].apply(
--> 329 lambda x: process_audience_from_response(x))
330 return collection_dataframe
331
/usr/local/lib/python2.7/dist-packages/pandas/core/series.pyc in apply(self, func, convert_dtype, args, **kwds)
2549 index_col : int or sequence, default 0
2550 Column to use for index. If a sequence is given, a MultiIndex
-> 2551 is used. Different default from read_table
2552 encoding : string, optional
2553 a string representing the encoding to use if the contents are
pandas/_libs/src/inference.pyx in pandas._libs.lib.map_infer()
/usr/local/lib/python2.7/dist-packages/pysocialwatcher/utils.pyc in <lambda>(x)
327 print_info("Computing Audience column")
328 collection_dataframe["audience"] = collection_dataframe["response"].apply(
--> 329 lambda x: process_audience_from_response(x))
330 return collection_dataframe
331
/usr/local/lib/python2.7/dist-packages/pysocialwatcher/utils.pyc in process_audience_from_response(literal_response)
319
320 def process_audience_from_response(literal_response):
--> 321 audience = json.loads(literal_response)["data"]["users"]
322 return int(audience)
323
/usr/lib/python2.7/json/__init__.pyc in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
337 parse_int is None and parse_float is None and
338 parse_constant is None and object_pairs_hook is None and not kw):
--> 339 return _default_decoder.decode(s)
340 if cls is None:
341 cls = JSONDecoder
/usr/lib/python2.7/json/decoder.pyc in decode(self, s, _w)
362
363 """
--> 364 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
365 end = _w(s, end).end()
366 if end != len(s):
/usr/lib/python2.7/json/decoder.pyc in raw_decode(self, s, idx)
380 obj, end = self.scan_once(s, idx)
381 except StopIteration:
--> 382 raise ValueError("No JSON object could be decoded")
383 return obj, end
ValueError: No JSON object could be decoded
This isn't really an issue as much as me not understanding how "and" queries work. I'm trying to collect data for all Hispanic ex-pats from Mexico with the following query:
{ "name": "Hispanic ex-pats",
"behavior": [
{"and" : [6003133212372, 6003277229371], "name" : "Hispanic Mexican Ex-pats"}
],
"geo_locations" :
[
{
"name" : "countries",
"location_types" : ["home"],
"values" : [
"US"
]
}
],
"ages_ranges": [
{"min":18, "max" : 25}
],
"genders": [0]
}
But this generates an invalid query (according to Facebook). Do you know whether the query is written wrong?
�[32m2020-02-24 00:02:57�[0m �[35mMSI-GYOUZA�[0m �[34mroot[18388]�[0m �[30mERROR�[0m �[31mError Message:(#2635) You are calling a deprecated version of the Ads API. Please update to the latest version: v6.0.�[0m
�[32m2020-02-24 00:02:57�[0m �[35mMSI-GYOUZA�[0m �[34mroot[18388]�[0m �[30mERROR�[0m �[31m"{u'message': u'(#2635) You are calling a deprecated version of the Ads API. Please update to the latest version: v6.0.', u'code': 2635, u'type': u'OAuthException', u'fbtrace_id': u'ApMcc7aMH33Kblh9Q_oWDc4'}"�[0m
Seems like when I try to make requests greater than 300 the progress halts indefinitely (with two accounts in credentials.csv). How do I make sure I continue to get data?
Apologies if this is a very basic question, but I'm still learning my ropes with Python, and wanted to use this project. I carried out the installation instructions, and then initiated jupyter lab running a python 2 kernel to run the code. However, when I try to run watcherAPI, I keep running into the same error- "No Module named main". Any help would be greatly appreciated, I haven't been able to find any tutorial notebooks either, but I'm happy to write one once I'm able to do this.
Hi,
first, thanks for creating this package!
I was just trying to get the MWE from the World Bank to run (https://worldbank.github.io/connectivity_mapping/intro.html), and immediately ran into this error message:
'{"error":{"message":"Invalid parameter","type":"OAuthException","code":100,"error_data":{"blame_field_specs":[["targeting"]]},"error_subcode":1870199,"is_transient":false,"error_user_title":"Update the value of the location_types field","error_user_msg":"All location targeting will now reach people living in or recently in the locations that you selected. Please remove all values from the location_types field.","fbtrace_id":"XXXXXXXX"}}'
It seems Facebook has changed how people's locations are targeted: https://www.facebook.com/business/help/365561350785642?id=176276233019487
I'd fix it myself, but am pretty much a newbie when it comes to Python. Do you maybe have a fix for this?
Thanks again in any case,
Carlo
Hi,
i've been trying to collect some data from facebook in order to learn how to use pySocialWatcher. I've setup an account with ID and Token, followed your instructions and typed command:
watcher.run_data_collection("pysocialwatcher/input_examples/jdafermos_example.json")
where my "jdafermos_example.json" file is:
{ "name": "Data in Crete",
"geo_locations": [
{ "name": "regions", "values": [{"key":"4170"}], "location_types": ["region"] }
],
"genders": [0],
"ages_ranges": [
{"min":18, "max":75}
]
}
The skeleton file is saved, but when it tries to collect the data an error message about "Param currency on field delivery_estimate" appears which includes the following lines:
FatalException: '{"error":{"message":"(#100) Param currency on field delivery_estimate: This param is not valid","type":"OAuthException","code":100,"fbtrace_id":"Ahe7jEVHUwoWhbm2wUTnMVQ"}}'
pysocialwatcher.utils.FatalException: 'FatalError: Check logging for clue. No way to proceed from here.'
Do you know if there is any solution to this?
Thank you
I've gotten the next error.
Error Message:(#2635) You are calling a deprecated version of the Ads API. Please update to the latest version: v14.0.
We should update API calls to a new version
Hello, I noticed that the process fail when the Facebook API responds with a new error. It's not a json, but a html page.
The source of the web page is always the same:
<!DOCTYPE html>\n<html lang="en" id="facebook">\n <head>\n <title>Facebook | Error</title>\n <meta charset="utf-8">\n <meta http-equiv="cache-control" content="no-cache">\n <meta http-equiv="cache-control" content="no-store">\n <meta http-equiv="cache-control" content="max-age=0">\n <meta http-equiv="expires" content="-1">\n <meta http-equiv="pragma" content="no-cache">\n <meta name="robots" content="noindex,nofollow">\n <style>\n html, body {\n color: #141823;\n background-color: #e9eaed;\n font-family: Helvetica, Lucida Grande, Arial,\n Tahoma, Verdana, sans-serif;\n margin: 0;\n padding: 0;\n text-align: center;\n }\n\n #header {\n height: 30px;\n padding-bottom: 10px;\n padding-top: 10px;\n text-align: center;\n }\n\n #icon {\n width: 30px;\n }\n\n h1 {\n font-size: 18px;\n }\n\n p {\n font-size: 13px;\n }\n\n #footer {\n border-top: 1px solid #ddd;\n color: #9197a3;\n font-size: 12px;\n padding: 5px 8px 6px 0;\n }\n </style>\n </head>\n <body>\n <div id="header">\n <a href="//www.facebook.com/">\n <img id="icon" src="//static.facebook.com/images/logos/facebook_2x.png" />\n </a>\n </div>\n <div id="core">\n <h1 id="sorry">Sorry, something went wrong.</h1>\n <p id="promise">\n We\'re working on it and we\'ll get it fixed as soon as we can.\n </p>\n <p id="back-link">\n <a id="back" href="//www.facebook.com/">Go Back</a>\n </p>\n <div id="footer">\n Facebook\n <span id="copyright">\n © 2018\n </span>\n <span id="help-link">\n ·\n <a id="help" href="//www.facebook.com/help/">Help Center</a>\n </span>\n </div>\n </div>\n <script>\n document.getElementById(\'back\').onclick = function() {\n if (history.length > 1) {\n history.back();\n return false;\n }\n };\n\n // Adjust the display based on the window size\n if (window.innerHeight < 80 || window.innerWidth < 80) {\n // Blank if window is too small\n document.body.style.display = \'none\';\n };\n if (window.innerWidth < 200 || window.innerHeight < 150) {\n document.getElementById(\'back-link\').style.display = \'none\';\n document.getElementById(\'help-link\').style.display = \'none\';\n };\n if (window.innerWidth < 200) {\n document.getElementById(\'sorry\').style.fontSize = \'16px\';\n };\n if (window.innerWidth < 150) {\n document.getElementById(\'promise\').style.display = \'none\';\n };\n if (window.innerHeight < 150) {\n document.getElementById(\'sorry\').style.margin = \'4px 0 0 0\';\n document.getElementById(\'sorry\').style.fontSize = \'14px\';\n document.getElementById(\'promise\').style.display = \'none\';\n };\n </script>\n </body>\n</html>\n
If I'm not wrong, at the moment the error is handle by the function handle_send_request_error()
in utils.py
. As this seems to be a temporary problem on Facebook, which can be the best way to handle it and trying to restore the process? By calling perform_collection_data_on_facebook()
?
Thanks!
I can't run any example. I've got the "WARNING Facebook Error code 2" and the "WARNING Facebook Error Message: (#2) Service temporarily unavailable". In the "dataframe_skeleton_.csv" I obtained only NAs.
That happened to me if I either used my own token and application ID or one from your credential.
What should I do?
Thanks
Hey I was trying to fiddle with the SLEEP constant as per the documentation below:
"Advaced Configurations
#####Change Sleep time between requests to 10s:
from pysocialwatcher import constants
constants.SLEEP = 10"
but the correct constant seems to be 'SLEEP_TIME' rather than 'SLEEP'
Best,
Julian
Facebook updated their ads API to v8.0 in August of this year. Have you managed to make pySocialWatcher run with these changes? Changing the 3 constants as you mentioned in the readme unfortunately didn't do the trick.
Thanks!
Hello, For the below code, watcher.run_data_collection("pySocialWatcher/pysocialwatcher/quick_example.json")
Which file do you want to pass as quick_example.json? There is no such file in the folder. In the facebook credential I gave the user access token and respective user id for that app (see below), is it correct?
Please help.
Facebook changed the interests and in this case the program has been failure error code: 100. How can I skip this error and keep going for work.
I had to update REACHESTIMATE_URL
in constants.py because one of the tokens that I was using was not working with the old API (recently updated from 2.11 to 3.0).
REACHESTIMATE_URL = "https://graph.facebook.com/v3.0/act_{}/delivery_estimate"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.