Petrel
Tools for writing, submitting, debugging, and monitoring Storm topologies in pure Python.
NOTE: Unlike the base Storm package, which only requires Python 2.6, Petrel requires Python 2.7.
Overview
Petrel offers some important improvements over the storm.py module provided with Storm:
- Topologies are implemented in 100% Python
- Petrel's packaging support automatically sets up a Python virtual environment for your topology and makes it easy to install additional Python packages.
- "petrel.mock" allows testing of single components or single chains of related components.
- Petrel automatically sets up logging for every spout or bolt and logs a stack trace on unhandled errors.
Installation
- System
- Python 2.7
- libyaml
- Python packages (you install)
- virtualenv
- Python packages (installed automatically by setup.py)
- simplejson 2.6.1
- thrift 0.8.0
- PyYAML 3.10
Installing Petrel as an egg
Before installing Petrel, make sure Storm is installed and in your path. Run the following command:
storm version
This will print the version of Storm active on your system, a number such as "0.7.4". You must use a version of Petrel whose first 3 digits match this version.
Install the egg:
easy_install petrel*.egg
This will download a few dependencies and then print a message like:
Finished processing dependencies for petrel==0.7.4.0.1
Installing Petrel from source
If you plan to use use Petrel by cloning its source code repository from github.com, follow these instructions.
Ensure the following tools are installed:
- Storm
- Test with "storm version"
- Should print something like "0.7.4"
- Thrift compiler
- Test with "thrift -version"
- Should print "Thrift version 0.7.0"
- Maven (test with "mvn -version")
Clone Petrel from github. Then run:
cd Petrel/petrel
python setup.py develop
This will download a few dependencies and then print a message like:
Finished processing dependencies for petrel==0.7.4.0.1
Running the word count example
From the base Petrel directory, run:
cd samples/wordcount
./buildandrun --config topology.yaml
This will run the topology in local mode. You can run it on a real cluster by providing an additional parameter, the name of the topology:
./buildandrun --config topology.yaml wordcount
Topology definition
import randomsentence import splitsentence import wordcount def create(builder): builder.setSpout("spout", randomsentence.RandomSentenceSpout(), 1) builder.setBolt("split", splitsentence.SplitSentenceBolt(), 1).shuffleGrouping("spout") builder.setBolt("count", wordcount.WordCountBolt(), 1).fieldsGrouping("split", ["word"])
Topology Configuration
Petrel's "--config" parameter accepts a YAML file with standard Storm configuration options. Petre also provides some Petrel-specific settings. See below.
topology.message.timeout.secs: 150
topology.ackers: 1
topology.workers: 5
topology.max.spout.pending: 1
worker.childopts: "-Xmx4096m"
topology.worker.childopts: "-Xmx4096m"
# Controls how Petrel installs its own dependencies, e.g. simplejson, thrift, PyYAML.
petrel.pip_options: "--no-index -f http://10.255.3.20/pip/"
# If you prefer, you can configure parallelism here instead of in setSpout() or
# setBolt().
petrel.parallelism.splitsentence: 1
Building and submitting topologies
Use the following command to package and submit a topology to Storm:
petrel submit --sourcejar ../../jvmpetrel/target/storm-petrel-*-SNAPSHOT.jar --config localhost.yaml
The above command builds and submits a topology in local mode. It will run until you stop it with Control-C. This mode is useful for simple development and testing.
If you want to run the topology on a Storm cluster, run the following command instead:
petrel submit --sourcejar ../../jvmpetrel/target/storm-petrel-*-SNAPSHOT.jar --config localhost.yaml wordcount
You can find instructions on setting up a Storm cluster here:
https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster
Build
- Get the topology definition by loading the create.py script and calling create().
- Package a JAR containing the topology definition, code, and configuration.
- Files listed in manifest.txt, e.g. additional configuration files
Deploy and Run
To deploy and run a Petrel topology on a Storm cluster, each Storm worker must have the following installed:
- Python 2.7
- setuptools
- virtualenv
Note that the worker machines don't require Petrel itself to be installed. Only the submitting machine needs to have Petrel. Each time you submit a topology using Petrel, it creates a custom jar file with the Petrel egg and and your Python spout and bolt code. These files in the wordcount example show how this works:
- buildandrun
- manifest.txt
Because Petrel topologies are self contained, it is easy to run multiple versions of a topology on the same cluster, as long as the code differences are contained within virtualenv. Before a spout or bolt starts up, Petrel creates a new Python virtualenv and runs the optional topology-specific setup.sh script to install Python packages. This virtual environment is shared by all the spouts or bolts from that instance of the topology on that machine.
Monitoring
Petrel provides a "status" command which lists the active topologies and tasks on a cluster. You can optionally filter by task name and Storm port (i.e. worker slot) number.
petrel status 10.255.1.58
Logging
Petrel does not write to the standard Storm logs. Instead it creates its own set of logs underneath the topology directory. For example, if you are running a topology in local mode, you'll find the Petrel log in a subdirectory of the "storm.local.dir" directory (whose location you can find in the Storm log). For example:
./supervisor/stormdist/test+topology-1-1365766701/resources/petrel28289_randomsentence.log ./supervisor/stormdist/test+topology-1-1365766701/resources/petrel28281_virtualenv.log ./supervisor/stormdist/test+topology-1-1365766701/resources/petrel28281_wordcount.log ./supervisor/stormdist/test+topology-1-1365766701/resources/petrel28285_splitsentence.log
Petrel uses stdout to send JSON data to Storm. Any other code that writes to stdout (e.g. "print" statements) would cause the Storm worker to crash. In order to avoid this, Petrel automatically reassigns sys.stdout and sys.stderr so they write to the Petrel (i.e. Python) logger instead.
When Storm is running on a cluster, it can be useful to send certain messages (e.g. errors) to a central machine. To help support this, Petrel sets an environment variable "NIMBUS_HOST". For example, the following log file configuration declares a log handler which sends any worker log messages INFO or higher to the Nimbus host.
[handler_hand02] class=handlers.SysLogHandler level=INFO formatter=form02 args=((os.getenv('NIMBUS_HOST') or 'localhost',handlers.SYSLOG_UDP_PORT),handlers.SysLogHandler.LOG_USER)
Petrel also has a "StormHandler" class sends messages to the Storm logger. This feature has not been thoroughly tested, but can be enabled by uncommenting the following line in petrel/util.py:
#logging.StormHandler = StormHandler
Storm Logging
When running Petrel applications in Storm's local mode, the console output is a mixture of Petrel and Storm logging output. This results in a lot of messages and can be hard to follow. You can control the Storm logging output by using Petrel's "--extrastormcp" option. Any directories specified to this option will be prepended to Storm's Java class path.
For example, create a file log4j.properties in the samples/wordcount directory, with the following contents:
# Set root logger level to DEBUG and its only appender to A1.
log4j.rootLogger=DEBUG, A1
#
## A1 is set to be a ConsoleAppender
log4j.appender.A1=org.apache.log4j.ConsoleAppender
#
## A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=[%d{dd MMM yyyy HH:mm:ss}] [%t] %-5p %c %x - %m%n
log4j.logger.org.apache=ERROR
log4j.logger.backtype=ERROR
log4j.logger.com.netflix=ERROR
Now run "petrel submit" like this:
petrel submit --extrastormcp=`pwd` --config=topology.yaml
With this setting, the apache, backtype, and netflix logs will be configured at ERROR level, suppressing most of the logger messages from Storm.
Testing
Petrel provides a "mock" module which mocks some of Storm's features. This makes it possible to test individual components and simple topologies in pure Python, without relying on the Storm runtime.
def test(): bolt = WordCountBolt() from petrel import mock from randomsentence import RandomSentenceSpout mock_spout = mock.MockSpout(RandomSentenceSpout.declareOutputFields(), [ ['word'], ['other'], ['word'], ]) result = mock.run_simple_topology([mock_spout, bolt], result_type=mock.LIST) assert_equal(2, bolt._count['word']) assert_equal(1, bolt._count['other']) assert_equal([['word', 1], ['other', 1], ['word', 2]], result[bolt])
In Petrel terms, a "simple" topology is one which only outputs to the default stream and has no branches or loops. run_simple_topology() assumes the first component in the list is a spout, and it passes the output of each component to the next component in the list.
License
The use and distribution terms for this software are covered by the BSD 3-clause license 1.0 (http://opensource.org/licenses/BSD-3-Clause) which can be found in the file LICENSE.txt at the root of this distribution. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.
setup.sh
A topology may optionally include a setup.sh script. If present, Petrel will execute it before launching the spout or bolt. Typically this script is used for installing additional Python libraries. Here's an example setup.sh script:
set -e # $1 will be non-zero if creating a new virtualenv, zero if reusing an existing one. if [ $1 -ne 0 ]; then for f in Shapely==1.2.15 pyproj==1.9.0 pycassa==1.7.0 \ configobj==4.7.2 greenlet==0.4.0 gevent==1.0b3 do echo "Installing $f" pip install $f done fi