azybler / mycelium Goto Github PK

The Mycelium high performance web crawler
=========================================

<pre>
   ,---------.       +-----------+
 .(           )      |           |
(   Interwebs..)  <--|  crawler  |--->  MongoDB
 (_         ___)     |           |      mycelium.crawl
   \.------.)        +-----------+

</pre>

This is a high performance web crawler which uses asyncrhonous IO through
libevent so it can handle thousands of connections; addresses the [C10K problem](
http://www.kegel.com/c10k.html).

It handles robots.txt. It mantains a single session per dns host, so it can
crawl thousands of hosts in parallel without hammering the same host.

It listens on a tcp socket for urls to crawl. (By default port 1024)

It saves the crawled documents to mongodb.

It's highly recommended to use curl compiled with asyncrhonous DNS (libares),
otherwise DNS resolving inside curl will block the program.

For the impatient
-----------------
$ ./bootstrap.sh
$ ./build.py
$ scons
$ build/release/crawler
$ echo 'http://google.com' | nc localhost 1024


How to use it
-------------

- Initialize the git submodules:

$ git submodule init
$ git submodule update


- Build the 3rd-party libraries:

$ ./build.py

- Compile the sources with SCons:

$ scons

- Alternatively you might build with system curl:

$  scons --system_curl

- But as said previously, synchronous DNS resolving will harm the performance
  and block. So it's  not recommended unless curl has been compiled with
  c-ares, as it will be done by build.py

Running
-------

- The crawler is configured with environment variables:
    CRAWLER_PORT: port to listen for urls
    CRAWLER_PARALLEL: number of parallel crawlers to run
    CRAWLER_DB_HOST: mongodb host for storing the documents, default is "localhost"
    CRAWLER_DB_NAMESPACE: database.collection, defaults to "mycelium.crawl"

Example / Screenshots:
----------------------

Start the crawler:

<pre>
piotr@gomasio:130:~/devel/mycelium$ build/release/crawler
log4cxx: Large window sizes are not allowed.
2011-11-23 00:43:14,481 - INFO  crawlog - Initializing system...
down: 0.00 iB 0.00 KB/s
down: 0.00 iB 0.00 KB/s
</pre>

The status gets printed every few seconds, displaying the total amount
downloaded and the current download speed.

Now you can start downloading urls by sending them to the control socket:

<code>
zcat feeds.txt.gz | nc localhost 1024
</code>


<pre>
2011-11-23 00:54:11,408 - INFO  crawlog - handle id: 0 retrieving robots: slashdot.org
down: 0.00 iB 0.00 KB/s
2011-11-23 00:54:11,838 - INFO  crawlog - handle id: 0 DONE: http://slashdot.org/robots.txt HTTP 200
2011-11-23 00:54:11,838 - INFO  crawlog - handle id: 0 retrieving content: slashdot.org
2011-11-23 00:54:13,777 - INFO  crawlog - handle id: 0 DONE: http://slashdot.org/ HTTP 200
</pre>

Interactive commands
--------------------

There are a few interactive commands for debug and diagnosis, typing help on
stdin shows them:

<pre>
help
on_read_interactive_cb: read: 5
commands: qlen dumpq reschedule status help quit
down: 0.00 iB 0.00 KB/s
</pre>

- qlen: Shows the amount of urls in the queue distributor, total, and per host
- dumpq: Show the urls in the queues
- reschedule: To force IDLE handles back to work, should not be any need to use
  this unless curl malfunctions, which happened with some versions of curl
- status: Show the status of the internal curl handlers
- quit: obvious

Building
========

Execute:

scons

Dependencies
============

Some (might be incomplete) list of libraries that are required:

- z
- boost_filesystem
- boost_system
- boost_regex
- log4cxx
- pthread
- curl
- event
- ssl
- libidn11-dev

Other software:

- scons
- flex
- autoconf (for building curl and c-ares)


Troubleshoting
==============

AttributeError: 'SConsEnvironment' object has no attribute 'CXXFile':
File "/home/piotr/devel/mycelium/SConstruct", line 210:
variant_dir='build/{0}'.format(build), duplicate=0)
File "/usr/lib/scons/SCons/Script/SConscript.py", line 614:
return method(*args, **kw)
File "/usr/lib/scons/SCons/Script/SConscript.py", line 551:
return _SConscript(self.fs, *files, **subst_kw)
File "/usr/lib/scons/SCons/Script/SConscript.py", line 260:
exec _file_ in call_stack[-1].globals
File "/home/piotr/devel/mycelium/src/SConscript", line 11:
env.CXXFile(target='Robots_flex.cc',source='robots.ll')

To solve this problem make sure flex is installed.