azybler / mycelium Goto Github PK
View Code? Open in Web Editor NEWThis project forked from larroy/mycelium
An asynchronous high performance web crawler
Home Page: http://pedro.larroy.com
License: Other
This project forked from larroy/mycelium
An asynchronous high performance web crawler
Home Page: http://pedro.larroy.com
License: Other
The Mycelium high performance web crawler ========================================= <pre> ,---------. +-----------+ .( ) | | ( Interwebs..) <--| crawler |---> MongoDB (_ ___) | | mycelium.crawl \.------.) +-----------+ </pre> This is a high performance web crawler which uses asyncrhonous IO through libevent so it can handle thousands of connections; addresses the [C10K problem]( http://www.kegel.com/c10k.html). It handles robots.txt. It mantains a single session per dns host, so it can crawl thousands of hosts in parallel without hammering the same host. It listens on a tcp socket for urls to crawl. (By default port 1024) It saves the crawled documents to mongodb. It's highly recommended to use curl compiled with asyncrhonous DNS (libares), otherwise DNS resolving inside curl will block the program. For the impatient ----------------- $ ./bootstrap.sh $ ./build.py $ scons $ build/release/crawler $ echo 'http://google.com' | nc localhost 1024 How to use it ------------- - Initialize the git submodules: $ git submodule init $ git submodule update - Build the 3rd-party libraries: $ ./build.py - Compile the sources with SCons: $ scons - Alternatively you might build with system curl: $ scons --system_curl - But as said previously, synchronous DNS resolving will harm the performance and block. So it's not recommended unless curl has been compiled with c-ares, as it will be done by build.py Running ------- - The crawler is configured with environment variables: CRAWLER_PORT: port to listen for urls CRAWLER_PARALLEL: number of parallel crawlers to run CRAWLER_DB_HOST: mongodb host for storing the documents, default is "localhost" CRAWLER_DB_NAMESPACE: database.collection, defaults to "mycelium.crawl" Example / Screenshots: ---------------------- Start the crawler: <pre> piotr@gomasio:130:~/devel/mycelium$ build/release/crawler log4cxx: Large window sizes are not allowed. 2011-11-23 00:43:14,481 - INFO crawlog - Initializing system... down: 0.00 iB 0.00 KB/s down: 0.00 iB 0.00 KB/s </pre> The status gets printed every few seconds, displaying the total amount downloaded and the current download speed. Now you can start downloading urls by sending them to the control socket: <code> zcat feeds.txt.gz | nc localhost 1024 </code> <pre> 2011-11-23 00:54:11,408 - INFO crawlog - handle id: 0 retrieving robots: slashdot.org down: 0.00 iB 0.00 KB/s 2011-11-23 00:54:11,838 - INFO crawlog - handle id: 0 DONE: http://slashdot.org/robots.txt HTTP 200 2011-11-23 00:54:11,838 - INFO crawlog - handle id: 0 retrieving content: slashdot.org 2011-11-23 00:54:13,777 - INFO crawlog - handle id: 0 DONE: http://slashdot.org/ HTTP 200 </pre> Interactive commands -------------------- There are a few interactive commands for debug and diagnosis, typing help on stdin shows them: <pre> help on_read_interactive_cb: read: 5 commands: qlen dumpq reschedule status help quit down: 0.00 iB 0.00 KB/s </pre> - qlen: Shows the amount of urls in the queue distributor, total, and per host - dumpq: Show the urls in the queues - reschedule: To force IDLE handles back to work, should not be any need to use this unless curl malfunctions, which happened with some versions of curl - status: Show the status of the internal curl handlers - quit: obvious Building ======== Execute: scons Dependencies ============ Some (might be incomplete) list of libraries that are required: - z - boost_filesystem - boost_system - boost_regex - log4cxx - pthread - curl - event - ssl - libidn11-dev Other software: - scons - flex - autoconf (for building curl and c-ares) Troubleshoting ============== AttributeError: 'SConsEnvironment' object has no attribute 'CXXFile': File "/home/piotr/devel/mycelium/SConstruct", line 210: variant_dir='build/{0}'.format(build), duplicate=0) File "/usr/lib/scons/SCons/Script/SConscript.py", line 614: return method(*args, **kw) File "/usr/lib/scons/SCons/Script/SConscript.py", line 551: return _SConscript(self.fs, *files, **subst_kw) File "/usr/lib/scons/SCons/Script/SConscript.py", line 260: exec _file_ in call_stack[-1].globals File "/home/piotr/devel/mycelium/src/SConscript", line 11: env.CXXFile(target='Robots_flex.cc',source='robots.ll') To solve this problem make sure flex is installed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.