Comments (3)
这一看就没生效,请求是你自己发起的
想要用框架的代理,可以用下面的写法
from feapder.network.proxy_pool import ProxyPool
proxy_pool = ProxyPool()
proxy = proxy_pool.get_proxy()
feapder版本为 1.8.8
from feapder.
亲测,feapder v1.8.8 版本管用。
import feapder
from feapder.network.proxy_pool import ProxyPool
from curl_cffi import requests
class AirSpiderDemo(feapder.AirSpider):
def start_requests(self):
url = "http://myip.ipip.net/"
yield feapder.Request(url, verify=False, method="GET")
def download_midware(self, request):
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Accept-Language": "zh-CN,zh;q=0.9",
"Cache-Control": "max-age=0",
"Proxy-Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"
}
# 代理 IP
proxy_pool = ProxyPool()
proxy = proxy_pool.get_proxy()
print(proxy)
# 绕过某些网站的反爬
response = requests.get(request.url, headers=headers, proxies=proxy, impersonate="chrome110")
return request, response
def parse(self, request, response):
print(response.text)
if __name__ == "__main__":
AirSpiderDemo(thread_count=1).start()
打印结果如下:
{'http': 'http://223.109.206.190:8976', 'https': 'http://223.109.206.190:8976'}
当前 IP:1.196.233.223 来自于:** 河南 信阳 电信
备注:这里我使用的是“隧道IP”,所以两个打印的两个IP会有区别,如果使用的是“独享IP”,打印的这两个IP将是一致的。
from feapder.
补充:feapder v1.8.6 版本的使用方法
import feapder
from feapder.network.proxy_pool import ProxyPool
from curl_cffi import requests
class AirSpiderDemo(feapder.AirSpider):
def start_requests(self):
url = "http://myip.ipip.net/"
yield feapder.Request(url, verify=False, method="GET")
def download_midware(self, request):
headers = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Accept-Language": "zh-CN,zh;q=0.9",
"Cache-Control": "max-age=0",
"Proxy-Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"
}
# 代理 IP
proxies = ProxyPool().get()
print(proxies)
# 绕过某些网站的反爬
response = requests.get(request.url, headers=headers, impersonate="chrome110", proxies=proxies)
return request, response
def parse(self, request, response):
print(response.text)
if __name__ == "__main__":
AirSpiderDemo(thread_count=1).start()
from feapder.
Related Issues (20)
- UpdateItem 批量更新数据问题 HOT 2
- 解析不了web,python3.11、feapder1.8.5 HOT 1
- 因为代理实效导致的重试还是使用实效的那个IP HOT 5
- 单机多进程模式下,MySQL 连接报错:通常每个套接字地址(协议/网络地址/端口)只允许使用一次 HOT 1
- 想要在start()启动爬虫的时候可以携带可变的初始url作为参数
- render=True报错 HOT 3
- feapder v1.8.8 使用代理IP报错? HOT 2
- mysql 查询 有bug HOT 2
- 指定parser_name和callback不生效
- 關於BatchSpider
- PLAYWRIGHT 默认开启无痕 HOT 2
- feapder requests能支持curl_cffi类似绕过指纹吗 HOT 2
- 批次爬虫获取redis中的任务时一直阻塞,任务存在就是不去执行,重新执行采集程序又正常了 HOT 3
- response.re_first()报错 HOT 1
- 浏览器渲染功能是否可以添加支持drissionpage库? HOT 4
- mongo使用url连接副本集,不能使用yeild item来进行存储 HOT 1
- 如何让AirSpider在调用时每次的浏览器实例都为最新 HOT 1
- 怎么配合类似指纹浏览器那样方式工作?
- 增加Playwright 启动浏览器方式
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from feapder.