Git Product home page Git Product logo

Comments (15)

ydf0509 avatar ydf0509 commented on July 30, 2024

seleiun版本高了。

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

那应该使用那个版本?

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

或者能更新http库,适应新的特性

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

这个api接口有没有说明文档,设置代理,设置header,cookie,有没有requests.session功能

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

大佬,这个是不是不能设置代理?

from nb_http_client.

ydf0509 avatar ydf0509 commented on July 30, 2024

那应该使用那个版本?

高版本的selieum去掉了phantomjs支持了,这些报错就算你看不懂英文报错提示,你复制到百度 google网上很容易搜索的

from nb_http_client.

ydf0509 avatar ydf0509 commented on July 30, 2024

这个api接口有没有说明文档,设置代理,设置header,cookie,有没有requests.session功能

cookie session 代理ip,自己去看http模块需要,这只举例性能。

代理ip一看你就是爬虫了,那cpu速度就不是最关键的了,你能对第三方网站每秒爬3000次吗不反扒吗?如果你无法自己学习python内置的http模块的用法,那就老老实实用requests爬虫就好了。

我这个主要是针对公司内部的http api调用,没有反扒和代理。

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

明白了,cookie可以设置在headers中,使用代理ip可以无限制爬取,使用golang测试可以对网站做到每秒1w次,但是golang的http请求不稳定,线程多开还没python好

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

看到你的自动收缩最强多线程后,没有去分析你的代码,还以为都是用C/C++编写的,我不知道还有底层的http网络请求,是我学识太浅薄了

from nb_http_client.

ydf0509 avatar ydf0509 commented on July 30, 2024

看到你的自动收缩最强多线程后,没有去分析你的代码,还以为都是用C/C++编写的,我不知道还有底层的http网络请求,是我学识太浅薄了

这是从三方包requests调用urllib3,urllib3调用http分析得到的,并不是偶然看了谁写的文章才知道http模块。

from nb_http_client.

ydf0509 avatar ydf0509 commented on July 30, 2024

看到你的自动收缩最强多线程后,没有去分析你的代码,还以为都是用C/C++编写的,我不知道还有底层的http网络请求,是我学识太浅薄了

你都爬虫了,还能一秒钟使用1万次代理吗?1秒钟100次算是非常牛逼的代理了。

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

隧道代理,动态中转,请求无限制,入口ip固定不变,出口IP随机,每次请求都不一样,配置足够可以百万并发

from nb_http_client.

ydf0509 avatar ydf0509 commented on July 30, 2024

隧道代理,动态中转,请求无限制,入口ip固定不变,出口IP随机,每次请求都不一样,配置足够可以百万并发

不可能的,阿布云就是隧道代理,每秒100次的套餐高达10万元年费,每秒5次的3600元年费。每秒100万次年费得需要10亿人民币,你有这么多钱?

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

https://www.lunaproxy.com/
https://www.bright.cn/
https://www.luminatidaili.com/
这几个了解一下,按流量收费的

from nb_http_client.

FynnFbc avatar FynnFbc commented on July 30, 2024

www.lunaproxy.com 这个平台有个无限制套餐,虽然很贵,如果是长期高并发很划算
无流量限制,无IP限制

from nb_http_client.

Related Issues (5)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.