配置解析 spark common config(spark.*，还有a

细节方面的TODO: logging 用户需求，并行度自由选择：repartition 增加或减少并行度，

M1 TODO about seatunnel HOT 4 CLOSED

garyelephant commented on May 17, 2024

M1 TODO

from seatunnel.

Comments (4)

garyelephant commented on May 17, 2024

细节方面的TODO:

logging
用户需求，并行度自由选择：repartition 增加或减少并行度，输出文件个数。
统一的插件定义的文档格式支持丰富的udf, Text(markdown), 支持多语言
把plugin单独拎出来作为一个archifact，方便插件依赖。
dstream／ spark streaming window operation： https://spark.apache.org/docs/latest/streaming-programming-guide.html#window-operations, #42
Monitoring: StreamingListener/ App monitor
性能调优：input调优，如 dstream／receiver个数调优https://spark.apache.org/docs/latest/streaming-programming-guide.html, filter调优, output调优
input，output协作完成 exactly once 语义：ack, 幂等, 事务
支持离线计算

from seatunnel.

garyelephant commented on May 17, 2024

BaseInput的getDStream 返回类型不是通用的，预计在实现input插件时将遇到问题。

abstract class BaseInput(config: Config) extends Plugin {

  /**
   * No matter what kind of Input it is, all you have to do is create a DStream to be used latter
   * */
  def getDStream: DStream[(String, String)]

  /**
   * Things to do after filter and before output
   * */
  def beforeOutput: Unit = {}

  /**
   * Things to do after output, such as update offset
   * */
  def afterOutput: Unit = {}

}

from seatunnel.

garyelephant commented on May 17, 2024

Spark Benchmark:
https://github.com/intel-hadoop/HiBench

from seatunnel.

garyelephant commented on May 17, 2024

2017年11月17日

（1）改为从command arguments读取：
--conf spark.driver.extraJavaOptions=-Dconfig.path=application.conf

（2）spark.submit.deployMode

val spark = SparkSession
   .builder()
   .appName("SparkApp")
   .master("spark: //192.168.60.80:7077")
   .config("spark.submit.deployMode","cluster")
   .enableHiveSupport()
   .getOrCreate()

from seatunnel.

Recommend Projects