A basic implementation of Facebook's Haystack. The following sections contain instructions for setting up the environment, building the package, running unit/integration tests, and running a demo.
To setup the environment, simply run the following two commands:
setup.sh
. env_vars
The first command installs/builds all the dependencies into ${HOME}/local
.
If you want everything to be installed somewhere else, then you can specify the
prefix path, e.g., setup.sh ${HOME}/opt
. If you do provide a different
prefix path, then it must be an absolute path. The following
packages/applications are installed or built by setup.sh
.
- cmake
- mongodb
- mongodb C client driver
- mongodb C++ client driver
- redis
- redis C client driver
- flask
- haystack - these are the haystack components I wrote
WARNING: the script takes about 10-13 minutes to install/build everything, and it is important to make sure that you have enough disk space, so it might be a good choice to first clean your personal Andrew space before proceeding with this.
env_vars
is a file that contains various definitions for different path
configurations, e.g., PATH, so it must be sourced in order to be able to run the
unit tests and demo correctly.
To run the unit tests, you can run the test_all
executable, which you can do
as follows:
./build/test/test_all
or alternatively
cd build/test
./test_all
One of the tests launches Redis and MongoDB, so the definitions in env_vars
must be in the environment for the test to pass.
To run a demo, first make sure the definitions in env_vars
are in the
environment, then launch all of the services by running start_all.sh
. To
access the service, simply enter a URL of the form <unix.andrew.cmu.edu>:51239
on your favorite browser, where <unix.andrew.cmu.edu> is the specific Andrew
Unix machine the service was launched from, e.g., unix7.andrew.cmu.edu. This
will give you a very bare bones page with a Choose File and Upload buttons.
Select a file to upload, and then go back to the root page, which should now
include a needle with Show and Delete links. Clicking on Show will show
the contents of the file that was uploaded, and clicking on Delete will delete
the needle from the Directory and the Store.
The front end web server is not implemented well, so it may be necessary to refresh any of the pages visited to get the full contents. Also, the contents of whatever is uploaded are simply dumped on the HTML, so it may be preferable to upload plain text files.
The Haystack components are the same as those presented in the Facebook paper:
- Directory: the executable corresponding to this component is
dir_app.cc
. The Directory receives requests from the front-end, and communicates with the Store when it creates a new needle. - Cache: The Cache receives requests from the front-end, communicates with the Redis cache, and makes requests from the Store.
- Store: The store receives needles from the Directory, and receives requests from the Cache.
setup.sh
: installs and builds the package.start_all.sh
: launches all of the components for a demo.src
: my own componentshaystack
: the haystack componentsproxy
: the front-end for the demotest
: unit/integration tests for the haystack components
These are for my own personal use.
http://<CDN>/<Cache>/<Machine ID>/<Logical volume, photo>
- Maps logical volumes to physical volumes.
- Load balances writes across logical volumes.
- Load balances reads across physical volumes.
- Marks some volumes as write enabled, others only read-enabled.
- Writes go through directory, because it chooses where the write is done.
- Distributed hash table.
- If object is not located here, then it fetches item from Store.
- Contains files
/hay/haystack/<logical volume, object id>
, each of which contains many photos. - Each machine keeps open file descriptors for each file that it manages.
- Maintains in-memory mapping of photo IDs to filesystem metadata, including
- file name,
- file offset,
- flags (e.g., is it deleted)
- size in bytes.
- Install virtualenv in home directory:
pip install --user virtualenv
. - Create virtualenv project directory:
virtualenv proj
. - Activate environment:
source proj/bin/activate
. - Install flask in virtual environment:
pip install flask
. Now flask can be used from current environment. - Finish using environment:
deactivate
. - Remove environment:
rm -fr proj
.
For details, see instructions here.
- run
install_mongo_cdriver.sh
If you ever happen to want to link against installed libraries in a given
directory, LIBDIR, you must either use libtool, and specify the full pathname of
the library, or use the -LLIBDIR
flag during linking and do at least one of
the following:
- add LIBDIR to the
LD_LIBRARY_PATH
environment variable during execution - add LIBDIR to the
LD_RUN_PATH
environment variable during linking - use the
-Wl,-rpath -Wl,LIBDIR
linker flag - have your system administrator add LIBDIR to
/etc/ld.so.conf