An open source information retrieval system written in C++11 and Python. Aspires to be an alternative to Nutch / Lucene. It uses MongoDB as an storage engine.
The Mycelium Information retrieval system
=========================================
Check the latest up-to-date user documentation at http://pedro.larroy.com/mycelium/sphinx/
For the impatient
-----------------
$ ./bootstrap.sh
$ ./build.py
$ scons
$ build/release/crawler
$ echo 'http://google.com' | nc localhost 1024
Status of the modules / Features
================================
- Crawler:
Feature complete.
- Tokenizer/Stemmer:
Work in progress.
- Inverted index:
TODO
- Search frontend:
TODO
How to use it
-------------
- Initialize the git submodules:
$ git submodule init
$ git submodule update
- Build the 3rd-party libraries:
$ ./build.py
- Compile the sources with SCons:
$ scons
- Alternatively you might build with system curl:
$ scons --system_curl
- But as said previously, synchronous DNS resolving will harm the performance
and block. So it's not recommended unless curl has been compiled with
c-ares, as it will be done by build.py
Running
-------
- The environment variables that affect some configuration parameters are:
* Specific for the crawler:
MYCELIUM_CRAWLER_PORT: port to listen for urls
MYCELIUM_CRAWLER_PARALLEL: number of parallel crawlers to run
* General for all the tools that interact with the DB:
MYCELIUM_DB_HOST: mongodb host for storing the documents, default is "localhost"
MYCELIUM_DB_NS: database.collection, defaults to "mycelium.crawl"
Dependencies
============
The software is build on debian / ubuntu systems, although it should be fairly
easy to port to other platforms.
Some (might be incomplete) list of libraries that are required:
- z
- boost_filesystem
- boost_system
- boost_regex
- log4cxx
- pthread
- curl
- event
- ssl
- libidn11-dev
Other software:
- scons
- flex
- autoconf (for building curl and c-ares)
Troubleshoting
==============
AttributeError: 'SConsEnvironment' object has no attribute 'CXXFile':
File "/home/piotr/devel/mycelium/SConstruct", line 210:
variant_dir='build/{0}'.format(build), duplicate=0)
File "/usr/lib/scons/SCons/Script/SConscript.py", line 614:
return method(*args, **kw)
File "/usr/lib/scons/SCons/Script/SConscript.py", line 551:
return _SConscript(self.fs, *files, **subst_kw)
File "/usr/lib/scons/SCons/Script/SConscript.py", line 260:
exec _file_ in call_stack[-1].globals
File "/home/piotr/devel/mycelium/src/SConscript", line 11:
env.CXXFile(target='Robots_flex.cc',source='robots.ll')
To solve this problem make sure flex is installed.
$ ./bootstrap.sh
Submodule '3rd_party/c-ares' () registered for path '3rd_party/c-ares'
Submodule '3rd_party/curl' () registered for path '3rd_party/curl'
Submodule '3rd_party/minimongo' () registered for path '3rd_party/minimongo'
Submodule '3rd_party/mongodb' () registered for path '3rd_party/mongodb'
fatal: Needed a single revision
Unable to find current revision in submodule path '3rd_party/minimongo'