This is an open source project that attempts to answer important genomics questions on the big data platform
- Essentials
- Be able to scale cluster up and down as needed
- Hortonworks Data Platform
v2.3.4
installed on Azure - Install Java v8
- HDFS
- Spark
- HBase
- Create infrastructure for evaluation
- local sandbox for testing / development
- "small cluster", 3x 8C/32G/10T+
- "medium cluster", 5x 8C/32G/10T+
- "large cluster", 7x 8C/32G/10T+
- Optional, but nice to have
- Jypter
- Zepplin
- Write ingest code
- create schema in HBase
- ingest sample
.gvcf
files
- Write filter code
- identity filter
- Load sample datasets
- Write export code
- Write recalculate code
- Write final report in Google Documents
This project uses Maven as a build system. Maven is an old standby build system for Java. It may be downloaded here, or installed via MacPorts, e.g. sudo port install maven
. GenomicWarehouse is build on Java 1.8 or higher.
For Maven noobs beginners, there is a nice getting started guide.
To build the hadoop-commons
code:
mvn compile
To run the set of unit tests:
mvn test
To include the (longer) integration tests:
mvn failsafe:integration-test
Getting Maven to fetch source and javadoc is easy:
mvn dependency:sources
mvn dependency:resolve -Dclassifier=javadoc
To add a dependency, edit the pom.xml
file, looking for the <dependancies>
section, e.g.
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.5.2</version>
</dependency>
Should be mostly self-explanatory.
To deploy the CLI, build a jar file with an embedded classpath. This is done through this Maven plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>edu.mayo.genomics.LoadVCF</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
Maven can also collect the library jar files into target/lib
:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.5.1</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib/</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
Install on cluster:
rsync -r --progress target/warehouse-1.0-SNAPSHOT.jar target/lib cluster:path/