philshem / gmaps_popular_times_scraper Goto Github PK

Hi I tried to work with the program but its getting errors. plz help.
20231115_223829.log

currently the code requires a path to a CSV file, with the first column containing a gmaps URL

Would be cool to smarten it up a bit, and let the user point to EITHER a CSV path, OR a single gmaps URL. This way you could loop over variables in a batch script, which could be fed dynamically.

I'm using Linux Mint. When I run the script using a single URL I get a datatype error:

`/home/elena/Scrivania/places scraper/gmaps_popular_times_scraper/scrape_gm.py:35: DtypeWarning: Columns (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38) have mixed types. Specify dtype option on import or set low_memory=False.
urls = pd.read_csv(sys.argv[1])
ERROR: (a=window.gtbExternal.pageT()) 20231028_195921
ERROR: if (window['wtf']&& window['wtf']['trace']&& 20231028_195921
ERROR: window['wtf']['trace']['timeStamp']){window['wtf']['trace']['timeStamp']('application.' + t);} 20231028_195921
ERROR: } 20231028_195921

.. etc...
`

When I run the script using the CSV file, my output gives a generic error:
ERROR: https://maps.app.goo.gl/X7J2w3QXUXssXXzb9 20231028_200733

I've tried several solutions but I can't seem to fix the problem. Any suggestions?

I have currently a 30 second sleep period to make sure the page and data is rendered. This is to avoid missing data. Better code would be starting with 5 seconds and then doing a while loop with increasing sleep, until 60 seconds or so.

This would help the overall performance of the code over many places. (It’s important to collect data from each place at least 1x per hour)

Because it’s not trivial to get running, would be cool to put this out as a docket image, or at least a docket build script. I know there are some python-selenium images that may be built on.