adonis2014 / streamingpro Goto Github PK

Declarative workflows for building Spark Streaming / Spark batch programe / Spark SQL Server

Spark Streaming

Spark Streaming is an extension of the core Spark API that enables stream processing from a variety of sources. Spark is a extensible and programmable framework for massive distributed processing of datasets, called Resilient Distributed Datasets (RDD). Spark Streaming receives input data streams and divides the data into batches, which are then processed by the Spark engine to generate the results.

Spark Streaming data is organized into a sequence of DStreams, represented internally as a sequence of RDDs.

StreamingPro

StreamingPro is not only a complete application, but also a extensible and programmable framework for spark streaming (also include spark,storm) that can easily be used to build your streaming application.

StreamingPro also make it possible that all you should do to build streaming program is assembling components(eg. SQL Component) in configuration file.

Features

• Pure Spark Streaming(Or normal Spark) program (Storm in future)
• No need of coding, only declarative workflows
• Rest API for interactive
• SQL-Oriented workflows support
• Data continuously streamed in & processed in near real-time
• dynamically CURD of workflows at runtime via Rest API
• Flexible workflows (input, output, parsers, etc...)
• High performance
• Scalable

downloads

Precomile version based on Spark 1.6.1 ： http://share.weiyun.com/4dffd0da6f3b6ed778a5285f67c30dc8

Precomile version based on Spark 2.0.0: http://share.weiyun.com/cf8c438cb4abcb51fa4b1d0f2c374fed

Spark 2.0 based on Scala 2.10

Documents

• Three steps to run your first application
• 三步跑起你的第一个应用
• 利用StreamingPro实现SQL-交互式查询
• 使用Spark SQL 构建流式处理程序
• 使用Spark SQL构建批处理程序
• 流式计算常见模块用法说明
• 用线性回归无编码实现文章浏览数预测
• Properties
• Build
• Run your first application
• Submit application
• dynamically CURD of workflows at runtime via Rest API
• Recovery
• Useful modules introduction
• Other runtime support

Architecture

If no picture show,please click me

If github is too slow to view ,please click me

Declarative workflows for building Spark Streaming

If no picture show,please click me

Implementation

If no picture show,please click me