asyncins / antispider Goto Github PK
View Code? Open in Web Editor NEW书籍《Python3 反爬虫原理与绕过实战》配套代码
书籍《Python3 反爬虫原理与绕过实战》配套代码
第四章 信息校验与反爬虫 postman 示例
cookie 反爬虫
`import requests
from lxml import etree
url = 'http://www.porters.vip/verify/cookie/content.html'
resp = requests.get(url)
if resp.status_code == 200:
html = etree.HTML(resp.text)
res = html.cssselect('.page-header h1') #①
print(res)
else:
print('This request is fail.')`
① 处 使用的是css选择器,需要指定cookie才有内容返回,但我没有加cookie,使用xpath(改为 html.xpath('//h1/text()')
) 后就爬取到了主题,为什么?难道 xpath 与 css 选择器在重定向上有原理差异?
我参照你的书敲出的代码如下,本来想以txt格式上传,但系统又报错。阁下之前第九章的HTML代码也无效,我自行修改才有正常效果。这个太复杂,不会改。
<title>实现滑动验证码</title> <style> .tracks{ /*滑轨样式*/ width:390px; height:40px; background: #d0c4fe; overflow: hidden; border: 1px solid #c5c5c5; border-radius: 4px; text-align: center; } .hover{ /*滑块样式*/ left: 0px; position: absolute; margin-left: 16px; width: 50px; height: 38px; background: #ad99ff; text-align: center; line-height: 38px; } .hover:hover{ background: #fff; } .slidertips{ /*提示信息样式*/ height: 38px; line-height: 38px; color:#fff; visibility: hidden; } </style> <script> $(function(){ var tracks=document.getElementById('tracks'), sliderblock=document.getElementById('sliderblock'), slidertips=document.getElementById('slidertips'); }) //滑块宽度 var sliderblockWidth=$('#sliderblock').width(); //滑轨长度 var tracksWidth=$('#tracks').width(); var mousemove=false;//mousedown状态 sliderblock.addEventListener('mousedown',function(e){ //监听mousedown事件,记录滑块起始位置 mousemove=true; startCoordinateX=e.clientX //滑块起始位置 }) var distanceCoordianteX=0;//滑块起始位置 tracks.addEventListener('mousemove',function(e){ //监听鼠标移动 if(mousemove){//鼠标点击滑块后才跟踪移动 distanceCoordianteX=e.clientX-startCoordinateX;//滑块当前位置 if(distanceCoordianteX>tracksWidth-sliderblockWidth){ //通过限制滑块位移距离,避免滑块向右移出滑轨 distanceCoordianteX=tracksWidth-sliderblockWidth; }else if(distanceCoordianteX<0){ //通过限制滑块位移距离,避免滑块向左移出滑轨 distanceCoordianteX=0; } //根据移动距离显示滑块位置 sliderblock.style.left=distanceCoordianteX+'px'; } }) sliderblock.addEventListener('mouseup',function(e){ //鼠标松开视为完成滑动,记录滑块当前位置并调用验证方法 var endCoordinateX=e.clientX; verifySliderRetuls(endCoordinateX); }) function verifySliderRetuls(endCoordinateX){//验证滑动结果 mousemove=false;//此时鼠标已松开,防止滑块跟随鼠标移动 //允许误差3像素 if(Math.abs(endCoordinateX-startCoordinateX-tracksWidth)docker运行splash后访问http://localhost:8050后,在render框里面输入链接https://www.baidu.com,点击后一直在Initializing。。。。
020-11-05 12:11:24+0000 [-] Log opened.
2020-11-05 12:11:24.204062 [-] Splash version: 3.3.1
2020-11-05 12:11:24.204543 [-] Qt 5.9.1, PyQt 5.9.2, WebKit 602.1, sip 4.19.4, Twisted 18.9.0, Lua 5.2
2020-11-05 12:11:24.204648 [-] Python 3.5.2 (default, Nov 12 2018, 13:43:14) [GCC 5.4.0 20160609]
2020-11-05 12:11:24.204834 [-] Open files limit: 1048576
2020-11-05 12:11:24.204923 [-] Can't bump open files limit
2020-11-05 12:11:24.308447 [-] Xvfb is started: ['Xvfb', ':639602014', '-screen', '0', '1024x768x24', '-nolisten', 'tcp']
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
2020-11-05 12:11:24.392224 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles
2020-11-05 12:11:24.392458 [-] memory cache: enabled, private mode: enabled, js cross-domain access: disabled
2020-11-05 12:11:24.525890 [-] verbosity=1, slots=20, argument_cache_max_entries=500, max-timeout=90.0
2020-11-05 12:11:24.527004 [-] Web UI: enabled, Lua: enabled (sandbox: enabled)
2020-11-05 12:11:24.527598 [-] Site starting on 8050
2020-11-05 12:11:24.527729 [-] Starting factory <twisted.web.server.Site object at 0x7f53dff16cc0>
2020-11-05 12:11:24.528053 [-] Server listening on http://0.0.0.0:8050
2020-11-05 12:11:46.112275 [-] "172.17.0.1" - - [05/Nov/2020:12:11:45 +0000] "GET / HTTP/1.1" 200 7679 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0"
2020-11-05 12:11:46.152861 [-] "172.17.0.1" - - [05/Nov/2020:12:11:45 +0000] "GET /_ui/style.css HTTP/1.1" 200 2591 "http://localhost:8050/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0"
2020-11-05 12:11:51.300247 [-] "172.17.0.1" - - [05/Nov/2020:12:11:50 +0000] "GET /info?wait=0.5&images=1&expand=1&timeout=90.0&url=http%3A%2F%2Fgoogle.com&lua_source= HTTP/1.1" 200 5320 "http://localhost:8050/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0"
2020-11-05 12:11:54.908156 [-] "172.17.0.1" - - [05/Nov/2020:12:11:54 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://localhost:8050/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0"
2020-11-05 12:12:05.672961 [-] "172.17.0.1" - - [05/Nov/2020:12:12:05 +0000] "GET /info?wait=0.5&images=1&expand=1&timeout=90.0&url=http%3A%2F%2Fwww.baidu.com&lua_source= HTTP/1.1" 200 5329 "http://localhost:8050/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0"
2020-11-05 12:13:05.673362 [-] Timing out client: IPv4Address(type='TCP', host='172.17.0.1', port=51986)
我尝试了3.3.1跟3.5这两个版本,都是这个问题,操作系统也尝试了centos7.8,ubuntu 20.04/18都尝试过还是这个样子,绝望了,请大佬指导下
图片右上角。已登录->跳转到提问页
想问你一下关于刷机的事
老哥,怎么我在网上找不到这本书呢?听名字应该是本好书,能否贴一个链接买来支持你一下 😄
原书第99页,print('This request is fial'),失败的英文是fail,作者拼错勒!
原书第16页代码执行时报以下错误:
pyppeteer.errors.BrowserError: Browser closed unexpectedly:
[0310/181827.222752:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
系统:Ubuntu 18.04.4 LTS
解决办法:修改相应代码
browser = await launch()
修改为
browser = await launch({'args': ['--no-sandbox']})
真是奇怪,居然找不到这本书。这方面的技术看来是只可意会,不可言传。PDF都没有吗?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.