Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.
Current version: 0.11.3rc1
Hadoop is a distributed system for counting words. Here is how it's done in Scalding.
package com.twitter.scalding.examples
import com.twitter.scalding._
class WordCountJob(args : Args) extends Job(args) {
TextLine( args("input") )
.flatMap('line -> 'word) { line : String => tokenize(line) }
.groupBy('word) { _.size }
.write( Tsv( args("output") ) )
// Split a piece of text into individual words.
def tokenize(text : String) : Array[String] = {
// Lowercase each word and remove punctuation.
text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")
}
}
Notice that the tokenize
function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)
You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).
- Getting Started page on the Scalding Wiki
- Runnable tutorials in the source.
- The API Reference, including many example Scalding snippets:
- Scalding Scaladocs provide details beyond the API References
- The Matrix Library provides a way of working with key-attribute-value scalding pipes:
- The Introduction to Matrix Library contains an overview and a "getting started" example
- The Matrix API Reference contains the Matrix Library API reference with examples
Please feel free to use the beautiful Scalding logo artwork anywhere.
There is a script (called sbt) in the root that loads the correct sbt version to build:
./sbt update
(takes 2 minutes or more)./sbt test
./sbt assembly
(needed to make the jar used by the scald.rb script)
The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:
> test-only com.twitter.scalding.FileSourceTest
Please refer to FAQ page if you encounter problems when using sbt.
We use Travis CI to verify the build:
Scalding modules are available from maven central.
The current groupid and version for all modules is, respectively, "com.twitter"
and 0.11.0
.
Current published artifacts are
scalding-core_2.9.3
scalding-core_2.10
scalding-args_2.9.3
scalding-args_2.10
scalding-date_2.9.3
scalding-date_2.10
scalding-commons_2.9.3
scalding-commons_2.10
scalding-avro_2.9.3
scalding-avro_2.10
scalding-parquet_2.9.3
scalding-parquet_2.10
scalding-repl_2.9.3
scalding-repl_2.10
The suffix denotes the scala version.
- Ebay
- Etsy
- Sharethrough
- Snowplow Analytics
- Soundcloud
To see a full list of users or to add yourself, see the wiki
For user questions, we are using the cascading-user mailing list for discussions: http://groups.google.com/group/cascading-user
For scalding development (internals, extending, release planning): https://groups.google.com/forum/#!forum/scalding-dev
In the remote possibility that there exist bugs in this code, please report them to: https://github.com/twitter/scalding/issues
Follow @Scalding on Twitter for updates.
Chat (IRC): freenode channel: #scalding
- Avi Bryant http://twitter.com/avibryant
- Oscar Boykin http://twitter.com/posco
- Argyris Zymnis http://twitter.com/argyris
Thanks for assistance and contributions:
- Sam Ritchie http://twitter.com/sritchie
- Aaron Siegel: http://twitter.com/asiegel
- Brad Greenlee: http://twitter.com/bgreenlee
- Edwin Chen http://twitter.com/edchedch
- Arkajit Dey: http://twitter.com/arkajit
- Krishnan Raman: http://twitter.com/dxbydt_jasq
- Flavian Vasile http://twitter.com/flavianv
- Chris Wensel http://twitter.com/cwensel
- Ning Liang http://twitter.com/ningliang
- Dmitriy Ryaboy http://twitter.com/squarecog
- Dong Wang http://twitter.com/dongwang218
- Kevin Lin http://twitter.com/reconditesea
- Josh Attenberg http://twitter.com/jattenberg
- Juliet Hougland https://twitter.com/j_houg
A full list of contributors can be found on GitHub.
Copyright 2013 Twitter, Inc.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0