hsuanchi / crawler_shopee_public Goto Github PK

您好，運行後熒幕顯示
INFO ⌲ Step 1: Total shop detail fetchedd:
WARNING Exception:
WARNING Exception:
.....
有辦法解決這個問題嗎？

我在裝crawler_shopee_public\requirement.txt時有點卡關
用venv or pipenv都不能順利裝上
我猜問題在我的環境是Python 3.9?

07/10/2023 10:02:18 AM INFO ⌲ Step 0: Test the IP you're using 5 times.
07/10/2023 10:02:39 AM WARNING Exception: Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
07/10/2023 10:02:39 AM WARNING Exception: Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
07/10/2023 10:02:39 AM WARNING Exception: Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
07/10/2023 10:02:39 AM WARNING Exception: Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
07/10/2023 10:02:39 AM WARNING Exception: Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
07/10/2023 10:02:39 AM INFO - Time Coast: 21.09s

07/10/2023 10:02:39 AM INFO ⌲ Step 1: Total shop detail fetchedd:
Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
Cannot connect to host ipv4.webshare.io:443 ssl:default [信号灯超时时间已到]
07/10/2023 10:03:01 AM WARNING Exception: Cannot connect to host shopee.tw:443 ssl:default [信号灯超时时间已到]
07/10/2023 10:03:01 AM INFO - Time Coast: 21.07s

07/10/2023 10:03:01 AM INFO ⌲ Step 2: Total pdp detail fetched:

07/10/2023 10:03:01 AM INFO - Time Coast: 0.01s

07/10/2023 10:03:01 AM INFO - Time Coast: 42.17s

您好，運行後熒幕顯示
INFO ⌲ Step 1: Total shop detail fetchedd:
WARNING Exception:
WARNING Exception:
.....
有辦法解決這個問題嗎？

由于马来西亚地区的shop id 是string 所以可以适应string 来寻找商店？

Hi there, first of all, thanks for your scripts. I already install the all requirements by "pip3 install -r requirement.txt". But it keeps giving me this err info “ModuleNotFoundError: No module named 'view.clean_data'”

您好，運行後熒幕顯示
INFO ⌲ Step 1: Total shop detail fetchedd:
WARNING Exception:
WARNING Exception:
.....
有辦法解決這個問題嗎？

您好，運行後熒幕顯示
INFO ⌲ Step 1: Total shop detail fetchedd:
WARNING Exception:
WARNING Exception:
.....
有辦法解決這個問題嗎？

hi hsuanchi

今天抓取蝦皮資訊時，發現pdp_detail裡面的資料是空值，是否api有改版了

您好!我執行main.py後, 屏幕顯示
google.auth.exceptions.DefaultCredentialsError: File /tele_bot/crawler_shopee_public-master/config/crawler_bigquery.json was not found.
確認dev log 有看到 object no attribute
Exception: 'DevelopmentConfig' object has no attribute 'PROXY_URL'

無法正確執行
這有什麼solution嗎?

I got an 403 status error when I started the script.

Recently, Shopee makes api v4 access more strict.. I think this error is caused by the changes..
Any ideas to solve the error?

Thank you in advance!

08/15/2023 09:23:52 AM INFO ⌲ Step 0: Test the IP you're using 5 times.
08/15/2023 09:23:53 AM INFO └── IP: 133.165.184.67
08/15/2023 09:23:53 AM INFO └── IP: 133.165.184.67
08/15/2023 09:23:53 AM INFO └── IP: 133.165.184.67
08/15/2023 09:23:53 AM INFO └── IP: 133.165.184.67
08/15/2023 09:23:53 AM INFO └── IP: 133.165.184.67
08/15/2023 09:23:53 AM INFO <function CheckIPAddress.__call__> - Time Coast: 0.98s 

08/15/2023 09:23:53 AM INFO ⌲ Step 1: Total shop detail fetchedd:
08/15/2023 09:23:53 AM WARNING Exception: rsp status 403, https://shopee.tw/api/v4/shop/get_shop_base?entry_point=ShopByPDP&need_cancel_rate=true&request_source=shop_home_page&version=1&username=fulinxuan
08/15/2023 09:23:53 AM INFO <function ShopDetailCrawler.__call__> - Time Coast: 0.15s 

08/15/2023 09:23:53 AM INFO ⌲ Step 2: Total pdp detail fetched:
08/15/2023 09:23:53 AM INFO <function ProductDetailCrawler.__call__> - Time Coast: 0.01s 

08/15/2023 09:23:53 AM INFO <function Crawler.__call__> - Time Coast: 1.14s 

Hi 您好：
感謝您提供這個project幫助我了解蝦皮。我在使用上有一些狀況，也許是操作問題，因此想直接請教，如果有誤還請指正。我以" https://shopee.tw/api/v2/item/get? " 取得之內容似乎與蝦皮公開資訊不同，由devtool查看，蝦皮是由v4 fetch而非v2，想請教作者v2是否已非最新api？謝謝。

I had check the log today that show error 403 issue," AM WARNING Exception: rsp status 403, https://shopee.tw/api/v4/shop/get_shop_base?entry_point=ShopByPDP&need_cancel_rate=true&request_source=shop_home_page&version=1&username=pat6 116xx
" seem the api has update,