đ HOLA đ HOLA đ HOLA ! ENJOY OUR GOOGLE MAPS SCRAPER đ TO EFFORTLESSLY EXTRACT DATA SUCH AS NAMES, ADDRESSES, PHONE NUMBERS, REVIEWS, WEBSITES, AND RATINGS FROM GOOGLE MAPS WITH EASE! đ€
Home Page: https://www.omkar.cloud/
License: MIT License
How to write the result after each query?
Now when i run 100 queries and something goes wrong on the 99th query all data is lost
Traceback (most recent call last):
File "D:\google-maps-scraper\main.py", line 3, in
from src import tasks_to_be_run
File "D:\google-maps-scraper\src_init_.py", line 1, in
from .scrape_google_maps_links_task import ScrapeGoogleMapsLinksTask
File "D:\google-maps-scraper\src\scrape_google_maps_links_task.py", line 5, in
from src.scrape_google_maps_places_task import ScrapeGoogleMapsPlacesTask
File "D:\google-maps-scraper\src\scrape_google_maps_places_task.py", line 5, in
class ScrapeGoogleMapsPlacesTask(BaseTask):
File "D:\google-maps-scraper\src\scrape_google_maps_places_task.py", line 6, in ScrapeGoogleMapsPlacesTask
task_config = TaskConfig(output_filename="all",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: TaskConfig.init() got an unexpected keyword argument 'log_time'
can I get location data ?
When I search for a query, the return output contains data in different languages (e.g. Arabic). How can I set the language of all the output to be only English?
Also, the returned field price_range is always Null, probably needs fixing as Google sets price ranges for every location.
D:\python\google-maps-scraper>python main.py
Traceback (most recent call last):
File "D:\python\google-maps-scraper\main.py", line 3, in <module>
from src import tasks_to_be_run
File "D:\python\google-maps-scraper\src\__init__.py", line 1, in <module>
from .scrape_google_maps_links_task import ScrapeGoogleMapsLinksTask
File "D:\python\google-maps-scraper\src\scrape_google_maps_links_task.py", line 5, in <module>
from src.scrape_google_maps_places_task import ScrapeGoogleMapsPlacesTask
File "D:\python\google-maps-scraper\src\scrape_google_maps_places_task.py", line 5, in <module>
class ScrapeGoogleMapsPlacesTask(BaseTask):
File "D:\python\google-maps-scraper\src\scrape_google_maps_places_task.py", line 6, in ScrapeGoogleMapsPlacesTask
task_config = TaskConfig(output_filename="all",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: TaskConfig.__init__() got an unexpected keyword argument 'log_time'Hey guys!
Congratulations on the incredible work. I have a trivial question, I want to perform a for loop, in which the query inside the config.py file is updated. For example: read a list of 5 cities, and for each of them, I want to return the results of schools and restaurants. Do you have any suggestions on how to do this?
Recently it was working but now the script is returning chrome version compatability issues.
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 120
Current browser version is 119.0.6045.200 with binary path C:\Program Files\Google\Chrome\Application\chrome.exe
Stacktrace:
GetHandleVerifier [0x00586E73+174291]
(No symbol) [0x004B0AC1]
(No symbol) [0x001C6FF6]
(No symbol) [0x001EE912]
(No symbol) [0x001EE150]
(No symbol) [0x001EB6D4]
(No symbol) [0x001E7A0F]
(No symbol) [0x0021E259]
(No symbol) [0x0021DBBA]
(No symbol) [0x00216DA6]
(No symbol) [0x001F1034]
(No symbol) [0x001F1F8D]
GetHandleVerifier [0x006249CC+820268]
sqlite3_dbdata_init [0x006E4EBE+652494]
sqlite3_dbdata_init [0x006E48D9+650985]
sqlite3_dbdata_init [0x006D962C+605244]
sqlite3_dbdata_init [0x006E586B+654971]
(No symbol) [0x004BFEBC]
(No symbol) [0x004B8428]
(No symbol) [0x004B854D]
(No symbol) [0x004A5858]
BaseThreadInitThunk [0x76DC7BA9+25]
RtlInitializeExceptionChain [0x776EBD2B+107]
RtlClearBits [0x776EBCAF+191]
Retrying
DevTools listening on ws://127.0.0.1:51345/devtools/browser/de9b7c2e-24c2-45f6-8a7b-aaeb9ec71ddd
DevTools listening on ws://127.0.0.1:51356/devtools/browser/37ed28ab-c300-439d-9a61-6813e8198f14
Traceback (most recent call last):
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\botasaurus\create_driver_utils.py", line 231, in create_selenium_driver
driver = AntiDetectDriver(
^^^^^^^^^^^^^^^^^
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\botasaurus\anti_detect_driver.py", line 31, in init
super().init(*args, **kwargs)
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 69, in init
super().init(DesiredCapabilities.CHROME['browserName'], "goog",
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\selenium\webdriver\chromium\webdriver.py", line 92, in init
super().init(
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 272, in init
self.start_session(capabilities, browser_profile)
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 364, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "C:\Users\Developer\Envs\maps_Venv\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 120
Current browser version is 119.0.6045.200 with binary path C:\Program Files\Google\Chrome\Application\chrome.exe
Stacktrace:
GetHandleVerifier [0x00586E73+174291]
(No symbol) [0x004B0AC1]
(No symbol) [0x001C6FF6]
(No symbol) [0x001EE912]
(No symbol) [0x001EE150]
(No symbol) [0x001EB6D4]
(No symbol) [0x001E7A0F]
(No symbol) [0x0021E259]
(No symbol) [0x0021DBBA]
(No symbol) [0x00216DA6]
(No symbol) [0x001F1034]
(No symbol) [0x001F1F8D]
GetHandleVerifier [0x006249CC+820268]
sqlite3_dbdata_init [0x006E4EBE+652494]
sqlite3_dbdata_init [0x006E48D9+650985]
sqlite3_dbdata_init [0x006D962C+605244]
sqlite3_dbdata_init [0x006E586B+654971]
(No symbol) [0x004BFEBC]
(No symbol) [0x004B8428]
(No symbol) [0x004B854D]
(No symbol) [0x004A5858]
BaseThreadInitThunk [0x76DC7BA9+25]
RtlInitializeExceptionChain [0x776EBD2B+107]
RtlClearBits [0x776EBCAF+191]
During handling of the above exception, another exception occurred:
can you please add email too
It seems like there's a problem with the 'els' elements. They have a 'None' value so they're probably not getting identified correctly.
Traceback (most recent call last):
File "C:\Users\said\AppData\Local\Programs\Python\Python39\lib\site-packages\bose\base_task.py", line 208, in run_task
result = self.run(driver, data)
File "C:\Users\said\Desktop\My_Files\google-maps-scraper-master\src\scrape_google_maps_links_task.py", line 101, in run
links = get_links()
File "C:\Users\said\Desktop\My_Files\google-maps-scraper-master\src\scrape_google_maps_links_task.py", line 84, in get_links
scroll_till_end(1)
File "C:\Users\said\Desktop\My_Files\google-maps-scraper-master\src\scrape_google_maps_links_task.py", line 81, in scroll_till_end
if len(els) >= max_results:
TypeError: object of type 'NoneType' has no len()
Hi Chetan,
nuktaa@Bose-Scraper:~/google-maps-scraper$ sudo python3 main.py
Downloading Driver in build/ directory ...
Traceback (most recent call last):
File "main.py", line 7, in
t.begin_task()
File "/usr/local/lib/python3.8/dist-packages/bose/base_task.py", line 144, in
begin_task
return run_task(False, 0)
File "/usr/local/lib/python3.8/dist-packages/bose/base_task.py", line 106, in
run_task
create_directories(self.task_path)
File "/usr/local/lib/python3.8/dist-packages/bose/base_task.py", line 51, in create_directories
_download_driver()
File "/usr/local/lib/python3.8/dist-packages/bose/base_task.py", line 29, in _download_driver
download_driver()
File "/usr/local/lib/python3.8/dist-packages/bose/download_driver.py", line 45, in download_driver
download_driver_in_path()
File "/usr/local/lib/python3.8/dist-packages/bose/download_driver.py", line 11, in download_driver_in_path
path = chromedriver_autoinstaller.install(path='build/')
File "/usr/local/lib/python3.8/dist-packages/chromedriver_autoinstaller/init.py", line 21, in install
chromedriver_filepath = utils.download_chromedriver(path, no_ssl)
File "/usr/local/lib/python3.8/dist-packages/chromedriver_autoinstaller/utils.py", line 203, in download_chromedriver
chrome_version = get_chrome_version()
File "/usr/local/lib/python3.8/dist-packages/chromedriver_autoinstaller/utils.py", line 109, in get_chrome_version
path = get_linux_executable_path()
File "/usr/local/lib/python3.8/dist-packages/chromedriver_autoinstaller/utils.py", line 153, in get_linux_executable_path
raise ValueError("No chrome executable found on PATH")
ValueError: No chrome executable found on PATH
a
Hi,
Below is the error I am getting while running the program in the command prompt:
(I am end user of this program)
Traceback (most recent call last):
File "I:\1\main.py", line 16, in
launch_tasks(*tasks_to_be_run)
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bose\launch_tasks.py", line 54, in launch_tasks
current_output = task.begin_task(current_data, task_config)
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bose\base_task.py", line 249, in begin_task
final = run_task(False, 0)
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bose\base_task.py", line 190, in run_task
create_directories(self.task_path)
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bose\base_task.py", line 134, in create_directories
_download_driver()
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bose\base_task.py", line 34, in download_driver
download_driver()
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bose\download_driver.py", line 50, in download_driver
download_driver_in_path()
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\bose\download_driver.py", line 12, in download_driver_in_path
path = chromedriver_autoinstaller_fix.install(path='build/')
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\chromedriver_autoinstaller_fix_init.py", line 21, in install
chromedriver_filepath = utils.download_chromedriver(path, no_ssl)
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\chromedriver_autoinstaller_fix\utils.py", line 283, in download_chromedriver
major_version = get_major_version(chromedriver_version)
File "C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\chromedriver_autoinstaller_fix\utils.py", line 212, in get_major_version
return version.split(".")[0]
AttributeError: 'NoneType' object has no attribute 'split'
I've checked the documentation but there wasn't an explicit explication of how to integrate proxies with the Bose framework.
Is this possible? If yes, can you explain how it should be done, please?
Task Started
[INFO] Downloading Driver for Chrome Version 116 in build/ directory. This is a one-time process. Download in progress...
WARNING:root:Can not find chromedriver for currently installed chrome version.
Traceback (most recent call last):
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 825, in move
os.rename(src, real_dst)
FileNotFoundError: [WinError 3] O sistema nĂŁo conseguiu localizar o caminho especificado: 'build/116/chromedriver.exe' -> 'build/chromedriver.exe'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\deost\OneDrive\Ambiente de Trabalho\Google Maps Scraper\google-maps-scraper\main.py", line 19, in
launch_tasks(*tasks_to_be_run)
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\launch_tasks.py", line 54, in launch_tasks
current_output = task.begin_task(current_data, task_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 214, in begin_task
final = run_task(False, 0)
^^^^^^^^^^^^^^^^^^
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 155, in run_task
create_directories(self.task_path)
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 99, in create_directories
_download_driver()
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 34, in _download_driver
download_driver()
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\download_driver.py", line 50, in download_driver
move_driver()
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\download_driver.py", line 39, in move_driver
move_chromedriver()
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\download_driver.py", line 38, in move_chromedriver
shutil.move(src_path, dest_path)
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 845, in move
copy_function(src, real_dst)
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 436, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "C:\Users\deost\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 256, in copyfile
with open(src, 'rb') as fsrc:
^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'build/116/chromedriver.exe'
I've got the following for the pro version: not running:
$ python main.py
Traceback (most recent call last):
File "C:\Users\jbard\google-maps-scraper-pro-master\main.py", line 3, in
from src import tasks_to_be_run
File "C:\Users\jbard\google-maps-scraper-pro-master\src_init_.py", line 1, in
from .scrape_google_maps_links_task import ScrapeGoogleMapsLinksTask
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_links_task.py", line 3, in
from src.scrape_google_maps_places_task import ScrapeGoogleMapsPlacesTask
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 5, in
class ScrapeGoogleMapsPlacesTask(BaseTask):
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 6, in ScrapeGoogleMapsPlacesTask
task_config = TaskConfig(output_filename="all",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: TaskConfig.init() got an unexpected keyword argument 'log_time'
My folder looks like this after running pip install -r requirements.txt
doesnt open website that it scrapes and after reading others issue... I have been waiting for 5 mins for it to record a scrape
Hello, thank you for you efforts ! It's a great job what you've done :)
I had some issues while trying to scrape information using "exact_name" + "city" (below the debug screenshot). It seems that it works only when maps is showing a list of results. Do you have an idea how to fix this ?
thank you again !
I've been getting this more recently: regardless of the search query (Which, on pro with same connectivity/speed, has slowed significantly compared to when it used to work)..
Result is that tasks fails, no update to output file CSV....
Config file:
queries = [
{"keyword": "restaurants near al sahafah",
"has phone": True,
"min_reviews": 100,
"min_rating": 4.5},
{"keyword": "used funiture near al rabi",
"has phone": True,
"min_reviews": 100,
"min_rating": 4.5},
]
number_of_scrapers = 12
Output:
Traceback (most recent call last):
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 190, in run_task
result = self.run(driver, data)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_links_task.py", line 261, in run
result = self.parallel(
^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 95, in parallel
result = (Parallel(n_jobs=n, backend="threading")(delayed(run)(l) for l in data_list))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 1098, in call
self.retrieve()
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 975, in retrieve
self._output.extend(job.get(timeout=self.timeout))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\pool.py", line 774, in get
raise self._value
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib_parallel_backends.py", line 620, in call
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 288, in call
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 288, in
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 89, in run
raise e
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 84, in run
result = callable(driver, data)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_links_task.py", line 160, in save_google
return ScrapeGoogleMapsPlacesTask().run(driver, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 82, in run
results = get_maps_data(links)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 76, in get_maps_data
ls = remove_nones(list(map(get_data, links)))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 73, in get_data
print('Done: ' + out_dict.get('title', ''))
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 22-27: character maps to
Traceback (most recent call last):
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 190, in run_task
result = self.run(driver, data)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_links_task.py", line 261, in run
result = self.parallel(
^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 95, in parallel
result = (Parallel(n_jobs=n, backend="threading")(delayed(run)(l) for l in data_list))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 1085, in call
if self.dispatch_one_batch(iterator):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 901, in dispatch_one_batch
self._dispatch(tasks)
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 819, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib_parallel_backends.py", line 597, in init
self.results = batch()
^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 288, in call
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 288, in
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 89, in run
raise e
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 84, in run
result = callable(driver, data)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_links_task.py", line 160, in save_google
return ScrapeGoogleMapsPlacesTask().run(driver, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 82, in run
results = get_maps_data(links)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 76, in get_maps_data
ls = remove_nones(list(map(get_data, links)))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 73, in get_data
print('Done: ' + out_dict.get('title', ''))
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 6-11: character maps to
Task Started
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Launched Browser
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Scrolling...
Fetched 119 links.
0 https://www.google.com/maps/place/Noodlesta/data=!4m7!3m6!1s0x3e2ee3b3a821fb93:0xe1bb86f8a6097fca!8m2!3d24.7898201!4d46.648781!16s%2Fg%2F11h3g9vdr4!19sChIJk_shqLPjLj4Ryn8JpviGu-E?authuser=0&hl=en&rclk=1
1 https://www.google.com/maps/place/Plov-+%D8%A8%D9%84%D9%88%D9%81+%D8%A8%D8%AE%D8%A7%D8%B1%D9%8A%E2%80%AD/data=!4m7!3m6!1s0x3e2ee3e3b8cf5a87:0xcfcd40de62c8e0fb!8m2!3d24.7890161!4d46.6486717!16s%2Fg%2F11s3qmn5wc!19sChIJh1rPuOPjLj4R--DIYt5Azc8?authuser=0&hl=en&rclk=1
2 https://www.google.com/maps/place/Jones+The+Grocer/data=!4m7!3m6!1s0x3e2ee3a1b0f194ab:0xab9da48fce4b89e1!8m2!3d24.7618125!4d46.6384375!16s%2Fg%2F11v0byxs8s!19sChIJq5TxsKHjLj4R4YlLzo-knas?authuser=0&hl=en&rclk=1
3 https://www.google.com/maps/place/Malga/data=!4m7!3m6!1s0x3e2ee5bc4d870dc1:0x24e46f1673226cec!8m2!3d24.800512!4d46.5983031!16s%2Fg%2F11kqltsv0n!19sChIJwQ2HTbzlLj4R7GwicxZv5CQ?authuser=0&hl=en&rclk=1
4 https://www.google.com/maps/place/Maestro+Pizza/data=!4m7!3m6!1s0x3e2ee315ae7daa21:0x18c8146a2645475b!8m2!3d24.789698!4d46.6490995!16s%2Fg%2F11b6gcfbl4!19sChIJIap9rhXjLj4RW0dFJmoUyBg?authuser=0&hl=en&rclk=1
5 https://www.google.com/maps/place/%D8%B4%D8%AC%D8%B1%D8%A9+%D8%A7%D9%84%D8%B2%D9%8A%D8%AA%D9%88%D9%86%E2%80%AD/data=!4m7!3m6!1s0x3e2ee318c76614e9:0xf4687db70b249b19!8m2!3d24.7853043!4d46.6343212!16s%2Fg%2F11kb7g3rxj!19sChIJ6RRmxxjjLj4RGZskC7d9aPQ?authuser=0&hl=en&rclk=1
6 https://www.google.com/maps/place/Barbar+Al+Yasmin/data=!4m7!3m6!1s0x3e2ee5493b0d449b:0x624d929897fabfe2!8m2!3d24.8226779!4d46.6533941!16s%2Fg%2F11kr9jw4q5!19sChIJm0QNO0nlLj4R4r_6l5iSTWI?authuser=0&hl=en&rclk=1
7 https://www.google.com/maps/place/PSP/data=!4m7!3m6!1s0x3e2ee35fd3ac1429:0x4da78f89f0c22763!8m2!3d24.7756427!4d46.6338867!16s%2Fg%2F11svqgn0pp!19sChIJKRSs01_jLj4RYyfC8ImPp00?authuser=0&hl=en&rclk=1
8 https://www.google.com/maps/place/%D8%A7%D9%88%D9%83%D9%86%D9%88+%D8%AF%D9%88%D8%A7%D9%85%E2%80%AD/data=!4m7!3m6!1s0x3e2ee3a560d6a89d:0x806b791b8166c6e0!8m2!3d24.793224!4d46.653549!16s%2Fg%2F11sv0vfjtb!19sChIJnajWYKXjLj4R4MZmgRt5a4A?authuser=0&hl=en&rclk=1
9 https://www.google.com/maps/place/Shrimp+Shack/data=!4m7!3m6!1s0x3e2ee33f5d738f17:0xb61d87c35f8d81b!8m2!3d24.7937583!4d46.6549949!16s%2Fg%2F11ggqtq511!19sChIJF49zXT_jLj4RG9j4NXzYYQs?authuser=0&hl=en&rclk=1
10 https://www.google.com/maps/place/FLOD+%7C+Healthy+Food/data=!4m7!3m6!1s0x3e2ee3b350490327:0x2b6e08ab1da32f7f!8m2!3d24.7901899!4d46.6485952!16s%2Fg%2F11sj948_05!19sChIJJwNJULPjLj4Rfy-jHasIbis?authuser=0&hl=en&rclk=1
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Launched Browser
Done: Casa Cacao El Celler de Can Roca
Done: HungerHub
Done: MP sandwich
Done: JW Steakhouse
Done: SULTAN AHMET
Done: CROSTA
Done: Sbarro Sahafa
Done: Bahadar154
Done: Pizza hut
Done: TACOSON
Done: 300 Degrees
Done: Al Romansiah
Done: Food Supplies Company Limited
Done: Qasdeer
Done: baodo
Done: Al tarboush al dimashqi restaurant
Done: Achar Chefna Restaurant Riyadh
Done: Om Ali
Done: Kudu
Done: Fork & Pipe
Done: Wafra Al Safwa (Safwa Cuisine)
Done: KATSU SANDO
Done: Shawarma Baker
Done: Burger King - Sahafa
Done: The Food Concept
Done: Buffalo Wild Wings - Sahafa
Done: To fitness
Done: Pasta Delizia
Done: Molto
Done: DAYS BURGER
Done: HARDEE''S
Done: Lulu restaurant
Done: Bellbird
Done: Sastoria
Done: Midori Salad
Closing Browser
Closed Browser
Task Failed! View Final Screenshot at tasks/15/final.png
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844 Safari/537.36
Launched Browser
Fetched 10 links.
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37
Launched Browser
Done: Largest Used Showroom Riyadh
Done: Computer store
Done: used furniture
Closing Browser
Closed Browser
Task Failed! View Final Screenshot at tasks/16/final.png
Task Completed!
Final screenshot
Not showing most recent search (showing previous which also failed?)
Error Log (in tasks folder)
Traceback (most recent call last):
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 190, in run_task
result = self.run(driver, data)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_links_task.py", line 261, in run
result = self.parallel(
^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 95, in parallel
result = (Parallel(n_jobs=n, backend="threading")(delayed(run)(l) for l in data_list))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 1085, in call
if self.dispatch_one_batch(iterator):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 901, in dispatch_one_batch
self._dispatch(tasks)
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 819, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib_parallel_backends.py", line 597, in init
self.results = batch()
^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 288, in call
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\joblib\parallel.py", line 288, in
return [func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 89, in run
raise e
File "C:\Users\jbard\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bose\base_task.py", line 84, in run
result = callable(driver, data)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_links_task.py", line 160, in save_google
return ScrapeGoogleMapsPlacesTask().run(driver, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 82, in run
results = get_maps_data(links)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 76, in get_maps_data
ls = remove_nones(list(map(get_data, links)))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\jbard\google-maps-scraper-pro-master\src\scrape_google_maps_places_task.py", line 73, in get_data
print('Done: ' + out_dict.get('title', ''))
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 6-11: character maps to
If there is a large number of queries eventually the program cannot support all the open files. No all.csv was generated. Probably best to save and close the files when done.
Traceback (most recent call last): File "/Users//Library/Python/3.10/lib/python/site-packages/bose/base_task.py", line 185, in run_task result = self.run(driver, data) File "/Users//google-maps-scraper/src/scrape_google_maps_links_task.py", line 261, in run result = self.parallel( File "/Users//Library/Python/3.10/lib/python/site-packages/bose/base_task.py", line 90, in parallel result = (Parallel(n_jobs=n, backend="threading")(delayed(run)(l) for l in data_list)) File "/Users//Library/Python/3.10/lib/python/site-packages/joblib/parallel.py", line 1098, in __call__ self.retrieve() File "/Users//Library/Python/3.10/lib/python/site-packages/joblib/parallel.py", line 975, in retrieve self._output.extend(job.get(timeout=self.timeout)) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/pool.py", line 771, in get raise self._value File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/Users//Library/Python/3.10/lib/python/site-packages/joblib/_parallel_backends.py", line 620, in __call__ return self.func(*args, **kwargs) File "/Users//Library/Python/3.10/lib/python/site-packages/joblib/parallel.py", line 288, in __call__ return [func(*args, **kwargs) File "/Users//Library/Python/3.10/lib/python/site-packages/joblib/parallel.py", line 288, in <listcomp> return [func(*args, **kwargs) File "/Users//Library/Python/3.10/lib/python/site-packages/bose/base_task.py", line 81, in run driver = self.create_driver(config) File "/Users//Library/Python/3.10/lib/python/site-packages/bose/base_task.py", line 73, in create_driver driver = create_driver(config) File "/Users//Library/Python/3.10/lib/python/site-packages/bose/create_driver.py", line 283, in create_driver driver = retry_if_is_error( File "/Users//Library/Python/3.10/lib/python/site-packages/bose/utils.py", line 88, in retry_if_is_error raise e File "/Users//Library/Python/3.10/lib/python/site-packages/bose/utils.py", line 81, in retry_if_is_error created_result = func() File "/Users//Library/Python/3.10/lib/python/site-packages/bose/create_driver.py", line 272, in run driver = BoseDriver( File "/Users//Library/Python/3.10/lib/python/site-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__ super().__init__(DesiredCapabilities.CHROME['browserName'], "goog", File "/Users//Library/Python/3.10/lib/python/site-packages/selenium/webdriver/chromium/webdriver.py", line 89, in __init__ self.service.start() File "/Users//Library/Python/3.10/lib/python/site-packages/selenium/webdriver/common/service.py", line 71, in start self.process = subprocess.Popen(cmd, env=self.env, File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 832, in __init__ errread, errwrite) = self._get_handles(stdin, stdout, stderr) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1594, in _get_handles p2cread, p2cwrite = os.pipe()
OSError: [Errno 24] Too many open files
What happened with the FIRSTPAGE option? For me it was the most important option V1 had. How to add it back?
Windows 10, python 11.4:
File "C:\Users\Best\AppData\Local\Programs\Python\Python311\Lib\site-packages\chromedriver_autoinstaller\utils.py", line 203, in download_chromedriver
chrome_version = get_chrome_version()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Best\AppData\Local\Programs\Python\Python311\Lib\site-packages\chromedriver_autoinstaller\utils.py", line 122, in get_chrome_version
version = output[0].decode('UTF-8').strip().split()[-1]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^
IndexError: list index out of range
I have encountered a critical issue while using the Pro version of the Google Maps scraper from the GitHub repository. In my experience, the free version of the scraper has been working seamlessly, providing accurate and relevant data. However, the professional version seems to be non-functional, failing to deliver the expected results. To ensure it wasn't an isolated incident, I meticulously tested the Pro version on two different computers, but unfortunately, the outcome remained the same â no data was scraped. To assist with identifying the root cause, I have attached a log file, hoping that the community can provide insights and help resolve this problem promptly. Your prompt assistance in rectifying this matter would be highly appreciated.
Traceback (most recent call last):
File "main.py", line 16, in
launch_tasks(*tasks_to_be_run)
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\bose\launch_tasks.py", line 54, in launch_tasks
current_output = task.begin_task(current_data, task_config)
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\bose\base_task.py", line 219, in begin_task
final = run_task(False, 0)
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\bose\base_task.py", line 160, in run_task
create_directories(self.task_path)
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\bose\base_task.py", line 104, in create_directories
_download_driver()
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\bose\base_task.py", line 33, in download_driver
from .download_driver import download_driver
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\bose\download_driver.py", line 1, in
import chromedriver_autoinstaller_fix
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\chromedriver_autoinstaller_fix_init.py", line 7, in
from . import utils
File "C:\Users\PC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\chromedriver_autoinstaller_fix\utils.py", line 19, in
from packaging import version
ModuleNotFoundError: No module named 'packaging'
Traceback (most recent call last):
File "/home/ubuntu/google-maps-scraper/main.py", line 5, in
Gmaps.places(queries, max=5)
File "/home/ubuntu/google-maps-scraper/src/gmaps.py", line 312, in places
places_obj = scraper.scrape_places(place_data)
File "/home/ubuntu/.local/lib/python3.10/site-packages/botasaurus/decorators.py", line 286, in wrapper_browser
check_and_download_driver()
File "/home/ubuntu/.local/lib/python3.10/site-packages/botasaurus/check_and_download_driver.py", line 21, in check_and_download_driver
if not has_driver():
File "/home/ubuntu/.local/lib/python3.10/site-packages/botasaurus/check_and_download_driver.py", line 10, in has_driver
driver_path = relative_path(get_driver_path(), 0)
File "/home/ubuntu/.local/lib/python3.10/site-packages/botasaurus/get_chrome_version.py", line 361, in get_driver_path
executable_name = get_filename(_get_major_version())
File "/home/ubuntu/.local/lib/python3.10/site-packages/botasaurus/get_chrome_version.py", line 353, in _get_major_version
version = get_major_version(get_chrome_version())
File "/home/ubuntu/.local/lib/python3.10/site-packages/botasaurus/get_chrome_version.py", line 135, in get_chrome_version
path = get_linux_executable_path()
File "/home/ubuntu/.local/lib/python3.10/site-packages/botasaurus/get_chrome_version.py", line 199, in get_linux_executable_path
raise ValueError("No chrome executable found on PATH")
ValueError: No chrome executable found on PATH
Peace be upon you. I am a beginner in programming and I love projects like these. I want help, if possible. I want to link this project to the SQL Server 2008 R2 database. please help me?
and storage data from scraping in database
Traceback (most recent call last):
File "/home/rlxdf/.local/lib/python3.11/site-packages/bose/base_task.py", line 117, in run_task
self.run(driver)
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 250, in run
result = get_data()
^^^^^^^^^^
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 241, in get_data
a = get_maps_data(links)
^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 220, in get_maps_data
ls = list(map(get_data, links))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 152, in get_data
out_dict['title'] = heading.text
^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webelement.py", line 84, in text
return self._execute(Command.GET_ELEMENT_TEXT)['value']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: Unable to receive message from renderer
(failed to check if window was closed: disconnected: not connected to DevTools)
(Session info: chrome=114.0.5735.90)
Stacktrace:
#0 0x5613ee72f233
#1 0x5613ee45e8e6
#2 0x5613ee4483a4
#3 0x5613ee4480c0
#4 0x5613ee446adf
#5 0x5613ee447282
#6 0x5613ee462761
#7 0x5613ee4645ee
#8 0x5613ee4646bc
#9 0x5613ee495592
#10 0x5613ee4b9c82
#11 0x5613ee490773
#12 0x5613ee4b9e4e
#13 0x5613ee4d1fce
#14 0x5613ee4b9a53
#15 0x5613ee48ef4d
#16 0x5613ee48ffbe
#17 0x5613ee6ef114
#18 0x5613ee6f2f67
#19 0x5613ee6fd6b0
#20 0x5613ee6f3bb3
#21 0x5613ee6c195a
#22 0x5613ee7184f8
#23 0x5613ee718687
#24 0x5613ee727f83
#25 0x7fcbafe3bfd4
Traceback (most recent call last):
File "/home/rlxdf/.local/lib/python3.11/site-packages/bose/base_task.py", line 117, in run_task
self.run(driver)
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 250, in run
result = get_data()
^^^^^^^^^^
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 241, in get_data
a = get_maps_data(links)
^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 220, in get_maps_data
ls = list(map(get_data, links))
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/github/google-maps-scraper/src/scraper.py", line 152, in get_data
out_dict['title'] = heading.text
^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webelement.py", line 84, in text
return self._execute(Command.GET_ELEMENT_TEXT)['value']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webelement.py", line 396, in _execute
return self._parent.execute(command, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: Unable to receive message from renderer
(failed to check if window was closed: disconnected: not connected to DevTools)
(Session info: chrome=114.0.5735.90)
Stacktrace:
#0 0x5613ee72f233
#1 0x5613ee45e8e6
#2 0x5613ee4483a4
#3 0x5613ee4480c0
#4 0x5613ee446adf
#5 0x5613ee447282
#6 0x5613ee462761
#7 0x5613ee4645ee
#8 0x5613ee4646bc
#9 0x5613ee495592
#10 0x5613ee4b9c82
#11 0x5613ee490773
#12 0x5613ee4b9e4e
#13 0x5613ee4d1fce
#14 0x5613ee4b9a53
#15 0x5613ee48ef4d
#16 0x5613ee48ffbe
#17 0x5613ee6ef114
#18 0x5613ee6f2f67
#19 0x5613ee6fd6b0
#20 0x5613ee6f3bb3
#21 0x5613ee6c195a
#22 0x5613ee7184f8
#23 0x5613ee718687
#24 0x5613ee727f83
#25 0x7fcbafe3bfd4
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/rlxdf/.local/lib/python3.11/site-packages/bose/bose_driver.py", line 272, in save_screenshot
self.get_screenshot_as_file(
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 927, in get_screenshot_as_file
png = self.get_screenshot_as_png()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 963, in get_screenshot_as_png
return b64decode(self.get_screenshot_as_base64().encode('ascii'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 975, in get_screenshot_as_base64
return self.execute(Command.SCREENSHOT)['value']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/home/rlxdf/.local/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools
(failed to check if window was closed: disconnected: not connected to DevTools)
(Session info: chrome=114.0.5735.90)
Stacktrace:
#0 0x5613ee72f233
#1 0x5613ee45e8e6
#2 0x5613ee447743
#3 0x5613ee4475e3
#4 0x5613ee4600e2
#5 0x5613ee4d1a95
#6 0x5613ee4b9a53
#7 0x5613ee48ef4d
#8 0x5613ee48ffbe
#9 0x5613ee6ef114
#10 0x5613ee6f2f67
#11 0x5613ee6fd6b0
#12 0x5613ee6f3bb3
#13 0x5613ee6c195a
#14 0x5613ee7184f8
#15 0x5613ee718687
#16 0x5613ee727f83
#17 0x7fcbafe3bfd4
Failed to save screenshot
Press Enter To Close Browser
I have a list of exact queries, like this:
{ "keyword": " KFC 101 W. Summers Drive Abbeville", "max_results" : 1, },
{ "keyword": " KFC 105 West Greenwood St Abbeville", "max_results" : 1, },
However it returns zero data, however in the screenshots I can see that it found the listings.
If I do something like "KFC in Wollongong, Australia" - it returns nothing. If I do "KFC in Illawarra, Australia"
How can I get it to return data for a list of specific queries, I've got a few hundred in a list.
Dear,
the scraper is working great, except for the language, this is my browser_config:
browser_config = BrowserConfig(
headless=True,
lang="en"
)
i tried also:
browser_config = BrowserConfig(
headless=True,
lang="en-GB"
)
still getting the results like the business category in German (since i'm in Germany)
can i just pass accept-lang="en-GB" as an argument to the driver directly in a way ?
do i need to put API google map key here ? to use this scrapper ?
OS; Debian 12
Python version 3.11.4
First I faced the error "selenium error" RecursionError: maximum recursion depth exceeded" after 10~ queries
I solved it with increasing then number in sys.setrecursionlimit(1000)
Now selenium randomly get stuck and I see no errors. This time it got stuck after 20 queries. Too bad the tool doesn't write results after each query. So far I got no results written due this problem.
Its stuck for 2 hours and when I run the scraper again it works until it get stuck again. I enabled selenium logging but I can't find my errors. Any idea what it can be so I can look into it?
Awesome script working in windows .
Can i run this script on linux ? If yes how can i do?
While running in docker i got these errors
Traceback (most recent call last):
File "main.py", line 16, in
launch_tasks(*tasks_to_be_run)
File "/usr/local/lib/python3.8/site-packages/bose/launch_tasks.py", line 54, in launch_tasks
current_output = task.begin_task(current_data, task_config)
File "/usr/local/lib/python3.8/site-packages/bose/base_task.py", line 219, in begin_task
final = run_task(False, 0)
File "/usr/local/lib/python3.8/site-packages/bose/base_task.py", line 160, in run_task
create_directories(self.task_path)
File "/usr/local/lib/python3.8/site-packages/bose/base_task.py", line 104, in create_directories
_download_driver()
File "/usr/local/lib/python3.8/site-packages/bose/base_task.py", line 34, in _download_driver
download_driver()
File "/usr/local/lib/python3.8/site-packages/bose/download_driver.py", line 47, in download_driver
major_version = get_major_version(get_chrome_version())
File "/usr/local/lib/python3.8/site-packages/chromedriver_autoinstaller_fix/init.py", line 41, in get_chrome_version
return utils.get_chrome_version()
File "/usr/local/lib/python3.8/site-packages/chromedriver_autoinstaller_fix/utils.py", line 140, in get_chrome_version
path = get_linux_executable_path()
File "/usr/local/lib/python3.8/site-packages/chromedriver_autoinstaller_fix/utils.py", line 204, in get_linux_executable_path
raise ValueError("No chrome executable found on PATH")
ValueError: No chrome executable found on PATH
Debian 12
Python 13.4
V1 is working. With V2 on the same machine I get this error:
$ python -m pip install -r requirements.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1)) (2.28.1)
Requirement already satisfied: bose in /home/user/.local/lib/python3.11/site-packages (from -r requirements.txt (line 2)) (1.8.1)
Collecting pydash
Downloading pydash-7.0.4-py3-none-any.whl (109 kB)
\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\109.6/109.6 kB 1.9 MB/s eta 0:00:00
Requirement already satisfied: beautifulsoup4>=4.11.2 in /home/user/.local/lib/python3.11/site-packages (from bose->-r requirements.txt (line 2)) (4.12.2)
Requirement already satisfied: chromedriver-autoinstaller==0.4.0 in /home/user/.local/lib/python3.11/site-packages (from bose->-r requirements.txt (line 2)) (0.4.0)
Requirement already satisfied: openpyxl>=3.0.3 in /home/user/.local/lib/python3.11/site-packages (from bose->-r requirements.txt (line 2)) (3.1.2)
Requirement already satisfied: selenium==4.5.0 in /home/user/.local/lib/python3.11/site-packages (from bose->-r requirements.txt (line 2)) (4.5.0)
Requirement already satisfied: undetected-chromedriver>=3.4.6 in /home/user/.local/lib/python3.11/site-packages (from bose->-r requirements.txt (line 2)) (3.4.7)
Requirement already satisfied: urllib3[socks]~=1.26 in /usr/lib/python3/dist-packages (from selenium==4.5.0->bose->-r requirements.txt (line 2)) (1.26.12)
Requirement already satisfied: trio~=0.17 in /home/user/.local/lib/python3.11/site-packages (from selenium==4.5.0->bose->-r requirements.txt (line 2)) (0.22.0)
Requirement already satisfied: trio-websocket~=0.9 in /home/user/.local/lib/python3.11/site-packages (from selenium==4.5.0->bose->-r requirements.txt (line 2)) (0.10.3)
Requirement already satisfied: certifi>=2021.10.8 in /usr/lib/python3/dist-packages (from selenium==4.5.0->bose->-r requirements.txt (line 2)) (2022.9.24)
Collecting typing-extensions!=4.6.0,>=3.10
Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Requirement already satisfied: soupsieve>1.2 in /home/user/.local/lib/python3.11/site-packages (from beautifulsoup4>=4.11.2->bose->-r requirements.txt (line 2)) (2.4.1)
Requirement already satisfied: et-xmlfile in /home/user/.local/lib/python3.11/site-packages (from openpyxl>=3.0.3->bose->-r requirements.txt (line 2)) (1.1.0)
Requirement already satisfied: websockets in /home/user/.local/lib/python3.11/site-packages (from undetected-chromedriver>=3.4.6->bose->-r requirements.txt (line 2)) (11.0.3)
Requirement already satisfied: attrs>=19.2.0 in /home/user/.local/lib/python3.11/site-packages (from trio~=0.17->selenium==4.5.0->bose->-r requirements.txt (line 2)) (23.1.0)
Requirement already satisfied: sortedcontainers in /home/user/.local/lib/python3.11/site-packages (from trio~=0.17->selenium==4.5.0->bose->-r requirements.txt (line 2)) (2.4.0)
Requirement already satisfied: async-generator>=1.9 in /home/user/.local/lib/python3.11/site-packages (from trio~=0.17->selenium==4.5.0->bose->-r requirements.txt (line 2)) (1.10)
Requirement already satisfied: idna in /usr/lib/python3/dist-packages (from trio~=0.17->selenium==4.5.0->bose->-r requirements.txt (line 2)) (3.3)
Requirement already satisfied: outcome in /home/user/.local/lib/python3.11/site-packages (from trio~=0.17->selenium==4.5.0->bose->-r requirements.txt (line 2)) (1.2.0)
Requirement already satisfied: sniffio in /home/user/.local/lib/python3.11/site-packages (from trio~=0.17->selenium==4.5.0->bose->-r requirements.txt (line 2)) (1.3.0)
Requirement already satisfied: exceptiongroup in /home/user/.local/lib/python3.11/site-packages (from trio-websocket~=0.9->selenium==4.5.0->bose->-r requirements.txt (line 2)) (1.1.1)
Requirement already satisfied: wsproto>=0.14 in /home/user/.local/lib/python3.11/site-packages (from trio-websocket~=0.9->selenium==4.5.0->bose->-r requirements.txt (line 2)) (1.2.0)
Requirement already satisfied: PySocks!=1.5.7,<2.0,>=1.5.6 in /home/user/.local/lib/python3.11/site-packages (from urllib3[socks]~=1.26->selenium==4.5.0->bose->-r requirements.txt (line 2)) (1.7.1)
Requirement already satisfied: h11<1,>=0.9.0 in /home/user/.local/lib/python3.11/site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium==4.5.0->bose->-r requirements.txt (line 2)) (0.14.0)
Installing collected packages: typing-extensions, pydash
Successfully installed pydash-7.0.4 typing-extensions-4.7.1
$ python3 main.py
Traceback (most recent call last):
File "/home/user/google-maps-scraper/main.py", line 1, in <module>
from bose.launch_tasks import launch_tasks
ModuleNotFoundError: No module named 'bose.launch_tasks'
Scraper went stuck with 1000+ times Revisiting. Now I got no all.csv ...
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Revisiting
Task Started
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37
DevTools listening on ws://127.0.0.1:63890/devtools/browser/5f7df4b4-7a45-4baa-b576-759cb7613322
Launched Browser
On linux I face this bug (unknown error: DevToolsActivePort file doesn't exist). To solve or debug this I need to run parameters like this:
chromeOptions.add_argument("--no-sandbox")
chromeOptions.add_argument("--disable-setuid-sandbox")
chromeOptions.add_argument("--disable-dev-shm-using")
chromeOptions.add_argument("--disable-extensions")
chromeOptions.add_argument("--disable-gpu")
chromeOptions.add_argument("start-maximized")
chromeOptions.add_argument("disable-infobars")
chromeOptions.add_argument("--headless")
How to do this with bose? I can't find it in the documentantion
It was deprecated in Python 3.10 by PEP 632 âDeprecate distutils moduleâ..
The 'els' variable, which is supposed to contain a list of elements, is None in the 'extract_links()' function
Traceback (most recent call last):
File "C:\Users\yravi\AppData\Roaming\Python\Python311\site-packages\bose\base_task.py", line 117, in run_task
self.run(driver)
File "C:\Users\yravi\Documents\google-maps-scraper-master\src\scraper.py", line 249, in run
result = get_data(driver, query, chotu_address_hierarchy, chotu_shop_category) # Call the get_data function
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yravi\Documents\google-maps-scraper-master\src\scraper.py", line 233, in get_data
links = get_links(query)
^^^^^^^^^^^^^^^^
File "C:\Users\yravi\Documents\google-maps-scraper-master\src\scraper.py", line 139, in get_links
links = extract_links(els)
^^^^^^^^^^^^^^^^^^
File "C:\Users\yravi\Documents\google-maps-scraper-master\src\scraper.py", line 135, in extract_links
return list(map(extract_link, elements))
Hi, I think only results of first page are coming. However you mentioned that 120 results at Max should come but I'm only getting the results from 1st page.
Search query= NDA coaching in Agra
Got 16 results in json file but there were a lot more than this in next pages.
So this library doesn't offer paginated resujts?
When I run main.py, I got this message:
'WARNING:root:Can not find chromedriver for currently installed chrome version.'
making the script crashes. Tried it under Windows and Ubuntu on 3 differents PC, same issue.
sorry I got this issue when running 'python main.py '
and I did not change anything in the main.py:
from src.gmaps import Gmaps
star_it = '''Love It? Star It! â https://github.com/omkarcloud/google-maps-scraper/'''
queries = ["web developers in delhi"]
Gmaps.places(queries, max=5)
But got this error:
(scraper) C:\Users\mf202\google-maps-scraper-master>python main.py
Running
Traceback (most recent call last):
File "main.py", line 1, in
from src.gmaps import Gmaps
File "C:\Users\mf202\google-maps-scraper-master\src\gmaps.py", line 6, in
from .cities import Cities
File "C:\Users\mf202\google-maps-scraper-master\src\cities.py", line 94, in
class Cities:
File "C:\Users\mf202\google-maps-scraper-master\src\cities.py", line 95, in Cities
Afghanistan = create_city_handler("AF")
File "C:\Users\mf202\google-maps-scraper-master\src\cities.py", line 76, in create_city_handler
def handle_city_data(_, prepend: str) -> list[str]:
TypeError: 'type' object is not subscriptable
Running
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/botasaurus/decorators.py", line 361, in run_task
result = func(driver, data)
File "/var/www/172-234-140-37.ip.linodeusercontent.com/public_html/google-maps-scraper/src/scraper.py", line 257, in scrape_places
perform_visit(driver, search_link)
File "/var/www/172-234-140-37.ip.linodeusercontent.com/public_html/google-maps-scraper/src/scraper_utils.py", line 26, in perform_visit
visit_gmap_with_consent()
File "/var/www/172-234-140-37.ip.linodeusercontent.com/public_html/google-maps-scraper/src/scraper_utils.py", line 6, in visit_gmap_with_consent
driver.organic_get(link, accept_cookies=True)
File "/usr/local/lib/python3.10/dist-packages/botasaurus/anti_detect_driver.py", line 325, in organic_get
accept_google_cookies(self)
File "/usr/local/lib/python3.10/dist-packages/botasaurus/accept_google_cookies.py", line 25, in accept_google_cookies
raise Exception("Unabe to load Google")
Exception: Unabe to load Google
Task failed with data: {'query': 'web developers in bangalore', 'is_spending_on_ads': False, 'max': 5, 'lang': None, 'geo_coordinates': None, 'zoom': None, 'convert_to_english': True}
Traceback (most recent call last):
File "/var/www/172-234-140-37.ip.linodeusercontent.com/public_html/google-maps-scraper/main.py", line 10, in
Gmaps.places(queries, max=5)
File "/var/www/172-234-140-37.ip.linodeusercontent.com/public_html/google-maps-scraper/src/gmaps.py", line 325, in places
result_item = process_result(min_reviews, max_reviews, category_in, has_website, has_phone, min_rating, max_rating, sort, key, scrape_reviews, reviews_max, reviews_sort, fields, lang, should_scrape_socials, convert_to_english,use_cache,places_obj)
File "/var/www/172-234-140-37.ip.linodeusercontent.com/public_html/google-maps-scraper/src/gmaps.py", line 181, in process_result
places = places_obj["places"]
TypeError: 'NoneType' object is not subscriptable
In my opinion a scraper is for bulk. Not for single searches with each different search preferences. I don't think it's practical to add search preferences for EACH search query in config.py. No serious user is doing that? And it's also a lot of extra work for people that are not familiar with text-processesing to add this for every query. It's better to have it hardcoded like in V1.
I use code that import my queries from excel and I usequeries = varlist without "max_results" : 5,. So now I can't set my own search preferences anymore. So I need it hardcoded again.
What code do I need to scrape_google_maps_places_task.py to make the search preferences hardcoded again like in V1?
I only want to fetch links with a website
I got a fresh IP. Scraped 120 queries. Had a 2 weeks break and now when scraping I get the following error after each query. Before it was working.
Traceback (most recent call last):
File "/home/arch/.local/lib/python3.11/site-packages/bose/ip_utils.py", line 7, in find_ip_details
response = requests.get(url, timeout=10)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.omkar.cloud', port=443): Max retries exceeded with url: /backend/ipinfo/ (Caused by NewConnection
Error('<urllib3.connection.HTTPSConnection object at 0x7fbfae708c50>: Failed to establish a new connection: [Errno 111] Connection refused'))
Closing Browser
Closed Browser
View Final Screenshot at tasks/546/final.png
View written JSON file at output/all.json
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36
Launched Browser
Fetched 8 links.
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Launched Browser
Done: SECURE ENERGY
Done: Secure Energy
Queries not working for specific requests such as max/min reviews. Sorting is not working either for me
No reproduction steps. Off the bat it has not been working for me.
Expected behavior: Only pull businesses with more reviews than 5 and less than 20
Actual behavior: Pulls businesses indiscriminately
Reproduces how often: 100%
Any additional information, configuration, data or output from commands that might be necessary to reproduce or understand the issue. Please try not to include screenshots of code or the command line, paste the contents as text instead. You can use GitHub Flavored Markdown to make the text look better.
I am running this script on a france based server . The out put contains french. So I tried with this option lang=Gmaps.Lang.English still i am getting the same . How can i bypass this for english?
[Description of the issue]
Expected behavior: [What you expect to happen]
Actual behavior: [What actually happens]
Reproduces how often: [What percentage of the time does it reproduce?]
Any additional information, configuration, data or output from commands that might be necessary to reproduce or understand the issue. Please try not to include screenshots of code or the command line, paste the contents as text instead. You can use GitHub Flavored Markdown to make the text look better.
Used config:
queries = [
{
"keyword": "bar delphi",
"select": ["title", "main_category", "phone" , "complete_address" "categories"],
},
]
number_of_scrapers = 16
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/bose/base_task.py", line 220, in run_task
result = self.run(driver, data)
File "/Users/xx/Documents/xx/Google Maps Scraper/google-maps-scraper-pro-master/src/scrape_google_maps_links_task.py", line 302, in run
new_results = clean(fetched_results, data)
File "/Users/xx/Documents/xx/Google Maps Scraper/google-maps-scraper-pro-master/src/scrape_google_maps_links_task.py", line 142, in clean
new_results = [sort_dict_by_keys(x, keys) for x in new_results]
File "/Users/xx/Documents/xx/Google Maps Scraper/google-maps-scraper-pro-master/src/scrape_google_maps_links_task.py", line 142, in
new_results = [sort_dict_by_keys(x, keys) for x in new_results]
File "/Users/xx/Documents/xx/Google Maps Scraper/google-maps-scraper-pro-master/src/scrape_google_maps_links_task.py", line 100, in sort_dict_by_keys
new_dict[key] = dictionary[key]
KeyError: 'complete_addresscategories'
First of all thanks for creating this scraper.
I am able to scrape google maps but I am facing an issue when exporting results that include foreign characters in them.
For example, this is how one of the extracted addresses appears on google maps:
TeĆvikiye, 34659 ĆiĆli/Ä°stanbul, TĂŒrkiye
And this is the same address in the output of the scraper (both json and csv):
TeĂ Ćžvikiye, 34659 Ă ĆŸiĂ Ćžli/Ă°stanbul, ĂÂȘñĂÆĂĆ Ă§
I believe it's an encoding issue and hopefully you might be able to find an easy solution.
Cheers
When I try to launch the Crawler, it gives me an Error "'NoneType' object is not subscriptable"
thats my main.py
from src.gmaps import Gmaps
star_it = '''Love It? Star It! â https://github.com/omkarcloud/google-maps-scraper/'''
queries = [
"Hotels",
"Cafés",
"BekleidungsgeschÀfte",
"ElektroniklÀden",
"Buchhandlungen",
"BlumengeschÀfte",
"Optiker",
"Gesundheitszentren",
"Banken",
"Nagelstudios",
"Computerreparaturdienste",
"ReisebĂŒros",
"Kunstgalerien",
"SchuhgeschÀfte",
"MöbelgeschÀfte",
"SportgeschÀfte",
"Juweliere",
"MusikgeschÀfte",
"Tierhandlungen",
"Fotostudios",
"Fast-Food-Restaurants",
"FeinkostlÀden",
"Kinderbetreuungsdienste",
"Yoga-Studios",
"Kosmetikstudios",
"FahrradgeschÀfte",
"GeschenkelÀden",
"Elektriker",
"Maler",
"Installateure",
"BaumÀrkte",
"Augenoptiker",
"ElektronikmÀrkte",
"Physiotherapeuten",
"SpielzeuglÀden",
"Architekten",
"Druckereien",
"GebrauchtwagenhÀndler",
"Sonnenstudios",
"Nagelstudios",
"Weinhandlungen",
"Spielhallen",
"Frisöre",
"ElektrogeschÀfte",
"Geschenkeshops",
"Reiseveranstalter",
"BĂ€ckereien",
"Schönheitssalons",
"Bauunternehmen",
"Tierpflegedienste",
"Gesundheitspraxen",
"GemischtwarenlÀden",
"Handy-GeschÀfte",
"Fahrschulen",
"Handwerker",
"Goldschmiede"
"Tattoo-Studios",
"Outdoor-AusrĂŒstungsgeschĂ€fte",
"Konditoreien",
"BrautgeschÀfte",
"Mietwagenfirmen",
"Karaoke-Bars",
"HaushaltswarengeschÀfte",
"Floristen",
"ImbissstÀnde",
"Haustierbetreuungsdienste",
"KunsthandwerkslÀden",
"WĂ€schereien",
"Tanzstudios",
"Second-Hand-LĂ€den",
"Gourmet-Restaurants",
"SchwimmbÀder",
"Veranstaltungsorte fĂŒr Live-Musik",
"Liquor Stores",
"Gartenbauunternehmen",
"TauchgeschÀfte",
"UhrengeschÀfte",
"RadsportgeschÀfte",
"Therapeuten",
"Ski-Shops",
"Sushi-Restaurants",
"Fensterputzer",
"SchlĂŒsseldienste",
"Cocktailbars",
"Pizzerien",
"HandtaschengeschÀfte",
"Ballettschulen",
"Teppichreinigungsdienste",
"Flughafen-Taxis",
"Fotolabors",
"HeimwerkerlÀden",
"Tattoo-Studios",
"Outdoor-AusrĂŒstungsgeschĂ€fte",
"Konditoreien",
"BrautgeschÀfte",
"Mietwagenfirmen",
"Karaoke-Bars",
"HaushaltswarengeschÀfte",
"Floristen",
"ImbissstÀnde",
"Haustierbetreuungsdienste",
"KunsthandwerkslÀden",
"WĂ€schereien",
"Tanzstudios",
"Second-Hand-LĂ€den",
"Gourmet-Restaurants",
"SchwimmbÀder",
"Veranstaltungsorte fĂŒr Live-Musik",
"Liquor Stores",
"Gartenbauunternehmen",
"TauchgeschÀfte",
"UhrengeschÀfte",
"RadsportgeschÀfte",
"Therapeuten",
"Ski-Shops",
"Sushi-Restaurants",
"Fensterputzer",
"SchlĂŒsseldienste",
"Cocktailbars",
"Pizzerien",
"HandtaschengeschÀfte",
"Ballettschulen",
"Teppichreinigungsdienste",
"Flughafen-Taxis",
"Fotolabors",
"HeimwerkerlÀden",
]
Gmaps.places(queries, max=150, has_website=False)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. đđđ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google â€ïž Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.