Google search results crawler, get google search results that you need
奇文共欣赏，疑义相与析
Carve out a bright future by coding.❤️
How to reach me:
| ✍️ | 👨💻 |
|---|---|
| Weekly：记录我本周看到的有价值的信息 | Code |
| Kubernetes Note：Kubernetes 学习之路 | Code |
| ML Note：机器学习之路 | Code |
| Sanic-For-Pythoneer：一份sanic使用教程 | Code |
当我搜索“lion”+"mammal"时已知google.rw结果为60900000，而google.com结果为61100000
请问是url的设置问题吗？
Can I change to get more than 9 URL results?
he get the result from the following pages? 2,3,4,5,6....
Thanks.
I don't know if this is only a problem of mine, but I'm not able to retrieve urls since 3 weeks ago
I'm using this module since 2017 with a lot of customizations, but it's the first time it stops for such a long time
Any changes at Google side that may affect it?
您好，我看了您的文章对于python抓取google搜索结果的一些了解。感谢您分享此项目。
我的疑问您是怎么通过浏览器的审查内容找到这个url的（https://{domain}/search?hl={language}&q={query}&btnG=Search&gbv=1）。
我以前写python爬虫都是直接从html中解析哪些标签的结果，得到所要的内容，对于这种url获取的方法不是很了解，请问能否有相关的教程或是文章推荐的吗？
期待你的回复谢谢。
您好，我使用下面的语句for k in mg.search(query="MoBiSys 2021", num=10, language='en'):
pprint.pprint(k)
想搜索MobiSys 2021关键字下谷歌搜索的前10个结果，但每次输出的结果的数量都不相同。而且有的时候输出结果中的text这一项的长度也会发生变化，请问下这是什么原因导致的。
Normal full page returned after mg.search_page . But null for the mg.search_url .
environment: Ubuntu 16.04 installed by pip
python version: 2 or 3(neither work)
在一个新的国外云服务器上使用时，需要点击同意使用cookie才返回结果
<!DOCTYPE html><html lang="en" dir="ltr"><head><style nonce="ZDW95Cah1sXxuL5ChDvdGw">\na, a:link, a:visited, a:active, a:hover {\n color: #1a73e8;\n text-decoration: none;\n}\nbody {\n font-family: Roboto,RobotoDraft,Helvetica,Arial,sans-serif;\n text-align: center;\n -ms-text-size-adjust: 100%;\n -moz-text-size-adjust: 100%;\n -webkit-text-size-adjust: 100%;\n}\n.box {\n border: 1px solid #dadce0;\n box-sizing: border-box;\n border-radius: 8px;\n margin: 24px auto 5px auto;\n max-width: 520px;\n padding: 24px;\n}\nh1 {\n color: #2c2c2c;\n font-size: 24px;\n hyphens: auto;\n margin: 24px 0;\n}\np, .sub, .contentText {\n color: #5f6368;;\n font-size: 14px;\n line-height: 20px;\n letter-spacing: 0.2px;\n text-align: left;\n}\n.signin {\n text-align: right;\n}\n.image {\n display: block;\n margin: 14px auto;\n}\n.basebutton {\n border-radius: 4px;\n cursor: pointer;\n font-family: Roboto,RobotoDraft,Helvetica,Arial,sans-serif;\n font-size: 14px;\n font-weight: 500;\n height: 36px;\n margin: 12px 4px 0;\n padding: 10px 24px;\n}\na.button {\n color: #fff;\n}\nbutton, .button {\n background-color: #1a73e8;\n border: none;\n color: #fff;\n}\ninput.button {\n -webkit-appearance: none;\n}\n.error {\n border: 2px solid #d93025;\n border-radius: 5px;\n color: #d93025;\n margin: auto;\n padding: 5px;\n}\n.footer {\n margin-top: 16px;\n}\n.footer a {\n margin-left: 24px;\n white-space: nowrap;\n}\n.footer a, .footer select, .footer input {\n color: #757575;\n font-size: 12px;\n}\n.languagepicker {\n display: inline-flex;\n margin: 2px 0;\n}\n#languageform,\n#languageselect {\n margin: 0 2px;\n}\n#languageselect {\n color: #757575;\n}\n.detailspage {\n margin: 24px auto 0 auto;\n max-width: 700px;\n}\n.detailspage h1, .detailspage h2 {\n font-size: 22px\n}\n.setting {\n border: 1px solid #dadce0;\n box-sizing: border-box;\n border-radius: 8px;\n margin-bottom: 11px;\n padding: 24px 24px 20px 24px;\n text-align: right;\n}\n.sub {\n padding: 24px 24px 20px 24px;\n}\nhr {\n margin: 12px -24px 12px;\n border: 0;\n border-top: 1px solid #dadce0;\n}\nfieldset {\n border: none;\n padding: 0;\n}\nlabel {\n margin: 24px;\n}\ntd {\n vertical-align: top;\n}\n.setting h2, .setting h3, h4 {\n color: #3c4043;\n margin: 0;\n text-align: left;\n}\n.yt-text {\n color: #5f6368;\n font-size: 14px;\n font-weight: 300;\n}\n.productLogoContainer {\n margin: auto;\n max-width: 360px;\n}\n\n/** Narrow screen (for example a mobile device). */\n@media only screen and (max-width: 480px) {\n body {\n margin: 18px 14px;\n }\n .imgContainer {\n min-width: 96px;\n }\n .hideOnSmallWidth {\n display: none;\n }\n .footer form {\n margin-bottom: 3px;\n }\n}\n\n@media only screen and (max-width: 380px) {\n .imgContainer {\n min-width: 72px;\n }\n}\n\n/** Normal/non-narrow screen. */\n@media not screen and (max-width: 480px) {\n body {\n margin: 18px 25px;\n }\n .imgContainer {\n width: 51%;\n }\n .hideOnNormalWidth {\n display: none;\n }\n .footer form {\n display: inline;\n }\n}\n</style><title>Before you continue to Google Search</title><meta name="viewport" content="initial-scale=1, maximum-scale=5, width=device-width"><link rel="shortcut icon" href="//www.google.com/favicon.ico"></head><body><div class="signin"><a href="https://accounts.google.com/ServiceLogin?hl=en&continue=https://www.google.com/search?q%3Dapple%26btnG%3DSearch%26gbv%3D1%26num%3D100&gae=cb-" class="basebutton button">Sign in</a></div><div class="box"><img src="//www.gstatic.com/images/branding/googlelogo/1x/googlelogo_color_68x28dp.png" srcset="//www.gstatic.com/images/branding/googlelogo/2x/googlelogo_color_68x28dp.png 2x" width="68" height="28" alt="Google"><div class="productLogoContainer"><img src="https://www.gstatic.com/ac/cb/scene_cookie_wall_search_v2.svg" alt="" width="100%" height="100%" class="image" aria-hidden="true"></div><h1>Before you continue to Google Search</h1><div class="contentText">Google uses <a href="https://policies.google.com/technologies/cookies?hl=en&utm_source=ucb" target="_blank">cookies</a> and data to:<ul><li>Deliver and maintain services, like tracking outages and protecting against spam, fraud, and abuse</li><li>Measure audience engagement and site statistics to understand how our services are used</li></ul>If you agree, we’ll also use cookies and data to:<ul><li>Improve the quality of our services and develop new ones</li><li>Deliver and measure the effectiveness of ads</li><li>Show personalized content, depending on your settings</li><li>Show personalized or generic ads, depending on your settings, on Google and across the web</li></ul>For non-personalized content and ads, what you see may be influenced by things like the content you’re currently viewing and your location (ad serving is based on general location). Personalized content and ads can be based on those things and your activity like Google searches and videos you watch on YouTube. Personalized content and ads include things like more relevant results and recommendations, a customized YouTube homepage, and ads that are tailored to your interests.<p>Click “Customize” to review options, including controls to reject the use of cookies for personalization and information about browser-level controls to reject some or all cookies for other uses. You can also visit g.co/privacytools anytime.</p></div><div><a href="https://consent.google.com/dl?continue=https://www.google.com/search?q%3Dapple%26btnG%3DSearch%26gbv%3D1%26num%3D100&gl=ES&hl=en&pc=srp&src=1" class="basebutton button">Customize</a><form action="https://consent.google.com/s" method="POST" style="display:inline;"><input type="hidden" name="gl" value="ES"><input type="hidden" name="m" value="0"><input type="hidden" name="pc" value="srp"><input type="hidden" name="continue" value="https://www.google.com/search?q=apple&btnG=Search&gbv=1&num=100"><input type="hidden" name="ca" value="r"><input type="hidden" name="x" value="8"><input type="hidden" name="v" value="cb.20220313-15-p0.en+FX+629"><input type="hidden" name="t" value="ADw3F8hwJ91yNTWN7Sri1DBk2PJbynmnAA:1647508588711"><input type="hidden" name="hl" value="en"><input type="hidden" name="src" value="1"><input type="submit" value="I agree" class="basebutton button" aria-label="Agree to the use of cookies and other data for the purposes described"/></form></div></div><div class="footer"><form action="https://consent.google.com/ml" method="get"><select id="languageselect" name="hl"><option value="af">Afrikaans</option><option value="az">azərbaycan</option><option value="bs">bosanski</option><option value="ca">català</option><option value="cs">Čeština</option><option value="cy">Cymraeg</option><option value="da">Dansk</option><option value="de">Deutsch</option><option value="et">eesti</option><option value="en-GB">English (United Kingdom)</option><option value="en" selected>English (United States)</option><option value="es">Español (España)</option><option value="es-419">Español (Latinoamérica)</option><option value="eu">euskara</option><option value="fil">Filipino</option><option value="fr-CA">Français (Canada)</option><option value="fr">Français (France)</option><option value="ga">Gaeilge</option><option value="gl">galego</option><option value="hr">Hrvatski</option><option value="id">Indonesia</option><option value="zu">isiZulu</option><option value="is">íslenska</option><option value="it">Italiano</option><option value="sw">Kiswahili</option><option value="lv">latviešu</option><option value="lt">lietuvių</option><option value="hu">magyar</option><option value="ms">Melayu</option><option value="nl">Nederlands</option><option value="no">norsk</option><option value="uz">o‘zbek</option><option value="pl">polski</option><option value="pt-BR">Português (Brasil)</option><option value="pt-PT">Português (Portugal)</option><option value="ro">română</option><option value="sq">shqip</option><option value="sk">Slovenčina</option><option value="sl">slovenščina</option><option value="sr-Latn">srpski (latinica)</option><option value="fi">Suomi</option><option value="sv">Svenska</option><option value="vi">Tiếng Việt</option><option value="tr">Türkçe</option><option value="el">Ελληνικά</option><option value="be">беларуская</option><option value="bg">български</option><option value="ky">кыргызча</option><option value="kk">қазақ тілі</option><option value="mk">македонски</option><option value="mn">монгол</option><option value="ru">Русский</option><option value="sr">српски</option><option value="uk">Українська</option><option value="ka">ქართული</option><option value="hy">հայերեն</option><option value="iw">עברית</option><option value="ur">اردو</option><option value="ar">العربية</option><option value="fa">فارسی</option><option value="am">አማርኛ</option><option value="ne">नेपाली</option><option value="mr">मराठी</option><option value="hi">हिन्दी</option><option value="as">অসমীয়া</option><option value="bn">বাংলা</option><option value="pa">ਪੰਜਾਬੀ</option><option value="gu">ગુજરાતી</option><option value="or">ଓଡ଼ିଆ</option><option value="ta">தமிழ்</option><option value="te">తెలుగు</option><option value="kn">ಕನ್ನಡ</option><option value="ml">മലയാളം</option><option value="si">සිංහල</option><option value="th">ไทย</option><option value="lo">ລາວ</option><option value="my">မြန်မာ</option><option value="km">ខ្មែរ</option><option value="ko">한국어</option><option value="ja">日本語</option><option value="zh-CN">简体中文</option><option value="zh-TW">繁體中文</option><option value="zh-HK">繁體中文 (香港)</option></select><input type="hidden" name="oldhl" value="en"><input type="hidden" name="gl" value="ES"><input type="hidden" name="m" value="0"><input type="hidden" name="pc" value="srp"><input type="hidden" name="continue" value="https://www.google.com/search?q=apple&btnG=Search&gbv=1&num=100"><input type="hidden" name="src" value="1"><input type="submit" value="Change language"></form><a href="https://policies.google.com/privacy?hl=en&utm_source=ucb">Privacy Policy</a> <a href="https://policies.google.com/terms?hl=en&utm_source=ucb">Terms of Service</a></div></body></html>
依赖cchardet怎么都安不上，直接去cchardet的repo可是那个作者不回答issue的，只能问问这儿有没有类似的问题和解决办法了。先谢
！Building wheels for collected packages: cchardet
Building wheel for cchardet (setup.py): started
Building wheel for cchardet (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /Users/rosingle/.pyenv/versions/anaconda3-5.3.0/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/t4/8bgw5py91qqgln17ymv33h2c0000gq/T/pip-install-d8c_zhdm/cchardet/setup.py'"'"'; file='"'"'/private/var/folders/t4/8bgw5py91qqgln17ymv33h2c0000gq/T/pip-install-d8c_zhdm/cchardet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/t4/8bgw5py91qqgln17ymv33h2c0000gq/T/pip-wheel-58vlfp4m --python-tag cp37
cwd: /private/var/folders/t4/8bgw5py91qqgln17ymv33h2c0000gq/T/pip-install-d8c_zhdm/cchardet/
Complete output (10820 lines):
cythonize: ['src/cchardet/_cchardet.pyx']
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.7
creating build/lib.macosx-10.7-x86_64-3.7/cchardet
copying src/cchardet/version.py -> build/lib.macosx-10.7-x86_64-3.7/cchardet
copying src/cchardet/init.py -> build/lib.macosx-10.7-x86_64-3.7/cchardet
running build_ext
building 'cchardet._cchardet' extension
creating build/temp.macosx-10.7-x86_64-3.7
creating build/temp.macosx-10.7-x86_64-3.7/src
creating build/temp.macosx-10.7-x86_64-3.7/src/cchardet
creating build/temp.macosx-10.7-x86_64-3.7/src/ext
creating build/temp.macosx-10.7-x86_64-3.7/src/ext/uchardet
creating build/temp.macosx-10.7-x86_64-3.7/src/ext/uchardet/src
creating build/temp.macosx-10.7-x86_64-3.7/src/ext/uchardet/src/LangModels
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/rosingle/.pyenv/versions/anaconda3-5.3.0/include -arch x86_64 -I/Users/rosingle/.pyenv/versions/anaconda3-5.3.0/include -arch x86_64 -Isrc/ext/uchardet/src -I/Users/rosingle/.pyenv/versions/anaconda3-5.3.0/include/python3.7m -c src/cchardet/_cchardet.cpp -o build/temp.macosx-10.7-x86_64-3.7/src/cchardet/_cchardet.o
warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/rosingle/.pyenv/versions/anaconda3-5.3.0/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/t4/8bgw5py91qqgln17ymv33h2c0000gq/T/pip-install-d8c_zhdm/cchardet/setup.py'"'"'; file='"'"'/private/var/folders/t4/8bgw5py91qqgln17ymv33h2c0000gq/T/pip-install-d8c_zhdm/cchardet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/t4/8bgw5py91qqgln17ymv33h2c0000gq/T/pip-record-4mfzp_18/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
你好，我试着运行您的示例。但是发现返回不了搜索结果，请问怎么解决？
对不起，初学者，烦劳
dbq做一次伸手党
您好，我使用了ss代理，可以正常翻墙，运行以后出现这个错误，请问是怎么一回事。
in google.py
add str named "tbs"
tbs = '&tbs=qdr:d'
url=url + tbs
news released within 24 hours
执行 pip install magic_google 出现错误：
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for cchardet
Successfully built magic_google
Failed to build cchardet
ERROR: Could not build wheels for cchardet, which is required to install pyproject.toml-based projects
换了多次代理节点，还是不行，ops~
My debug result is that
u'\n<TITLE>302 Moved</TITLE>\n
If I run the script directly, just show the link 2017-07-20 17:07:08,892 - magic_google - INFO: https://www.google.com/search?hl=en&q=python&btnG=Search&gbv=1.
It doesn't appear any more information.
Could Google detect the searching script?
Which proxy server do you suggest?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.