Vagrant project to spin up a cluster of 1 virtual machine with Hadoop v2.7.1, Spark v1.5.0 and Spark-Notebook 0.6.1

• Hadoop
• Spark
• Spark-Notebook

1. Download and install VirtualBox
2. Download and install Vagrant.
3. Run vagrant box add centos65 https://github.com/2creatives/vagrant-centos/releases/download/v6.5.1/centos65-x86_64-20131205.box
4. Git clone this project, and change directory (cd) into this project (directory).
5. Run vagrant up to create the VM.
6. Run vagrant ssh to get into your VM.
7. Run vagrant destroy when you want to destroy and get rid of the VM.

Some gotcha's.

1. Make sure you download Vagrant v1.4.3 or higher.
2. Make sure when you clone this project, you preserve the Unix/OSX end-of-line (EOL) characters. The scripts will fail with Windows EOL characters.
3. Make sure you have 4Gb of free memory for the VM. You may change the Vagrantfile to specify smaller memory requirements.
4. This project has NOT been tested with the VMWare provider for Vagrant.
5. You may change the script (common.sh) to point to a different location for Hadoop and Spark to be downloaded from. Here is a list of mirrors for Hadoop: http://www.apache.org/dyn/closer.cgi/hadoop/common/.
6. Basically, Hadoop YARN is installed, but not started. If you want YARN Service, uncomment YARN section of scripts/init-start-all-services.sh and scripts/start-all-services.sh.

If you have the resources (CPU + Disk Space + Memory), you may modify Vagrantfile to have even more HDFS DataNodes, YARN NodeManagers, and Spark slaves. Just find the line that says "numNodes = 4" in Vagrantfile and increase that number. The scripts should dynamically provision the additional slaves for you.

Edit variables in scripts/common.sh.

spark-notebook-sandbox uses CentOS 6.5 for default. It can be changed to other version/distribution. Just edit the Vagrantfile's following lines:

    node.vm.box = "centos65"
    node.vm.box_url = "https://github.com/2creatives/vagrant-centos/releases/download/v6.5.1/centos65-x86_64-20131205.box"

Default settings use jdk-7u71-linux-x64.tar.gz or java-1.7.0-openjdk.x86_64. Edit following lines from scripts/common.sh for oracle JDK:

#java
JAVA_ARCHIVE=jdk-7u71-linux-x64.tar.gz

In case of OpenJDK, edit scripts/setup-java.sh:

function installRemoteJava {
	echo "install open jdk"
	yum install -y java-1.7.0-openjdk.x86_64
}

Hadoop version is described in scripts/common.sh:

HADOOP_VERSION=2.7.1
HADOOP_ARCHIVE=hadoop-${HADOOP_VERSION}.tar.gz
HADOOP_MIRROR_DOWNLOAD=http://archive.apache.org/dist/hadoop/core/hadoop-${HADOOP_VERSION}/${HADOOP_ARCHIVE}

To build the sandbox, Make sure HADOOP_MIRROR_DOWNLOAD is available for downloading.

Spark version is described in scripts/common.sh:

SPARK_VERSION=1.5.0
SPARK_ARCHIVE=spark-${SPARK_VERSION}-bin-hadoop2.6.tgz
SPARK_MIRROR_DOWNLOAD=http://www.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_ARCHIVE}

To build the sandbox, Make sure SPARK_MIRROR_DOWNLOAD is available for downloading.

spark-notebook version is described in scripts/common.sh:

SPARKNOTEBOOK_VERSION=0.6.1
SCALA_VERSION=2.10.4
SPARKNOTEBOOK_NAME=spark-notebook-${SPARKNOTEBOOK_VERSION}-scala-${SCALA_VERSION}-spark-${SPARK_VERSION}-hadoop-${HADOOP_VERSION}-with-hive-with-parquet
SPARKNOTEBOOK_ARCHIVE=${SPARKNOTEBOOK_NAME}.tgz
SPARKNOTEBOOK_MIRROR_DOWNLOAD=https://s3.eu-central-1.amazonaws.com/spark-notebook/tgz/${SPARKNOTEBOOK_ARCHIVE}

Spark-Notebook has many archive for various settings. You may find the exact setting for your purpose on the http://spark-notebook.io tgz section. Find the notebook, scala, spark, hadoop version you want to install.

You can make the VM setup even faster if you pre-download the Hadoop, Spark, and Oracle JDK into the /resources directory.

1. /resources/hadoop-2.7.1.tar.gz
2. /resources/spark-1.5.0-bin-hadoop2.6.tgz
3. /resources/jdk-7u51-linux-x64.gz
4. /resources/spark-notebook-0.6.1.tgz

The setup script will automatically detect if these files (with precisely the same names) exist and use them instead. If you are using slightly different versions, you will have to modify the script accordingly.

You can check the following URLs to monitor the Hadoop daemons.

1. [NameNode] (http://10.10.10.101:50070/dfshealth.html)
2. [Spark] (http://10.10.10.101:8080)
3. [Spark History] (http://10.10.10.101:18080)
4. [Spark-Notebook] (http://10.10.10.101:8989)

The Vagrant box is downloaded to the ~/.vagrant.d/boxes directory. On Windows, this is C:/Users/{your-username}/.vagrant.d/boxes.

This project is started from Jee Vang's vagrant-hadoop-2.4.1-spark-1.0.1. Thanks a lot.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.