Git Product home page Git Product logo

python-coroutine's Introduction

线程,协程对比和Python爬虫实战说明

这篇文章写的是我对线程和协程的理解,有错误之处欢迎指出。

举一个餐馆的例子。我们把一个餐厅当做一个进程,如果这个餐厅只有1个服务员,那么这个餐厅是 单线程的;如果这个餐厅有2个服务员,那么这个餐厅是双线程的。 如果这个餐厅的服务员像海底捞的服务员一样,一次只服务一桌的客人,在这桌的客人离开前不会服务 下一桌的客人,那么我们称这个线程是阻塞的。如果一个服务员同时服务多桌客人,那么我们称这个线程 是非阻塞的。
实现了非阻塞的线程我们称之为协程,也就是说协程是依赖线程的存在。
下面就是Python的爬虫实战对比说明
本来想使用aiohttp库的,但为了对比效果更好,我使用requests库
我先写了一个本地的flask应用,在返回网页前等待3秒模拟真实网络爬虫
flask-run
单线程爬虫:
single-thread
共花费301

多线程爬虫:
mult-thread
共花费75

单线程协程爬虫:
single-coroutine
只花费9
可以看出单线程的协程爬虫速度已经非常快了

多线程协程爬虫:
mult-coroutine
只花费3

总结

多线程和协程结合,能够使得爬虫的速度达到极致

参考资料:

python-coroutine's People

Contributors

last2win avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-coroutine's Issues

多线程和协程结合实例

我自己根据多线程和协程结合参考写的 来爬取b站用户信息,会报错 coroutine 'run' was never awaited
不知道有没有写好的,借鉴一下``

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.