DataFrames are useful constructs for data scientists, popularized by R and pandas.
Implementations of these packages are not natively distributed - their DataFrames struggle with big data. bigdf is a dataframe on top of Apache Spark. It is written as an internal Scala DSL with the look and feel of Pandas to make it easy for those familiar with Pandas to use it.
A gentle(but outdated) introduction is available at:
- Choose a repos directory (make one if it doesn't exist)
export $REPOS = ~/Documents/repos
- Install scala and sbt (brew instructions below)
$ brew install scala
$ brew install sbt
- clone the bigdf repo
$ cd $REPOS
$ git clone [email protected]:ayasdi/bigdf.git
- sbt update, package, and test(optional) in the bigdf directory.
$ cd bigdf
$ sbt update
$ sbt test
$ sbt assembly
- Start a shell and begin playing. You need spark 1.4.x
$ cd $REPOS
$ ./spark/bin/spark-shell --jars ./bigdf/target/scala-2.10/bigdf-assembly-*.jar
- https://github.com/AyasdiOpenSource/bigdf/wiki/ for examples
- Look at DFTest.scala for usage
$cd $REPOS
$git clone [email protected]:mattpap/IScala.git
$cd IScala
$sbt
>++ 2.10.4
>assembly
>^D
$ipython profile create scala
Add the following line to the three files ~/.ipython/profile_scala/ipython*_config.py
c.KernelManager.kernel_cmd = ["java", "-jar", "$ISCALA_PATH/lib/IScala.jar", "--connection-file", "{connection_file}", "--parent"]"
To start notebook
$cd <my notebooks directory>
$ipython notebook --profile scala
To install spark and bigdf to your local mvn repos
$brew install maven
$mvn install:install-file -Dfile=$REPOS/bigdf-transforms/lib/spark-assembly-1.2.0-SNAPSHOT-hadoop1.0.4.jar -DgroupId=org.apache -DartifactId=spark-assembly_2.10 -Dversion=1.2.0-SNAPSHOT-hadoop1.0.4 -Dpackaging=jar
$ mvn install:install-file -Dfile=$REPOS/bigdf-transforms/bigdf/target/scala-2.10/bigdf-assembly-0.2.jar -DgroupId=com.ayasdi -DartifactId=bigdf-assembly_2.10 -Dversion=0.2 -Dpackaging=jar
To start using bigdf in a notebook
%libraryDependencies += "org.apache" %% "spark-assembly" % "1.2.0-SNAPSHOT-hadoop1.0.4"
%libraryDependencies += "com.ayasdi" %% "bigdf-assembly" % "0.2"
%resolvers += "Local Maven Repository" at "file:///Users/mohit/.m2/repository"
%update
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import com.ayasdi.bigdf._
val sc = new SparkContext("local[*]", "bigdf-intro")
val df = DF(sc, "/Users/mohit/Downloads/faredata2013/trip_fare_1.csv", ',', true)