vipshop / saturn Goto Github PK

Currently, metric data stores in ZK which will consume a lot of zk performance. and the metric calculation is also very slow to iterate the whole zk data.

these metric data is only needed in console dashboard, so these data is not needed to stored permanent.

help operator and alarm system to process some special logic according to this critical level

When assign a job's partition(such as j1) to a executor(such as e1), we need to judge whether there exists a node that is $jobs/j1/servers/e1/status, if not , this assignment should be ignored.

1. listen to $jobs/j1/servers/e1/status add event, resharding the executor of this job online when event fired.
2. listen to $jobs/j1/servers/e1/status remove event, resharding the executor of this job offline when event fired.
3. readjust the logic of exeutor online/offline, listen to $SaturnExecutors/executors/xxx/ip add or remove event, add a empty shell of executor on add event fired, remove the executor on remove event fired.
4. readjust the logic of putBackBalancing, just put back the shard to executor that can run the job.
5. change jobScheduler. write $jobs/j1/servers/e1/status after job initialization, and shutdown JobScheduler when init throw exception.

Saturn uses zookeeper heavily for reading and writing.

we need to find to way to improve such behaviors performance to support more jobs in one zookeeper cluster

NamespaceShardingManager add zk's ConnectionStateListener

1. Stop NamespaceShardingManager when zk's client is not connected
2. Executor all resharding when zk's client is reconnected

server information cleanup failed because executor's ip node is deleted before job really shutdown.

介绍说本项目是当当的elastic-job的增强版本，具体增强的点。可以介绍下吗？

Job's name must just include the charactors that should match the regex [0-9a-zA-Z_]*

1.add one more embedded Demo， and remove the embedded part from spring demo
2.explain the utils code in spring demo
3.add readme.txt in each examples

wait for job finish before really shutdown

本人使用spring demo job 发现无法使用注解情况，是否功能还未完善？

add quickstart.bat and quickstart.sh to quick run the project
also add optional embedded zk in console

The job is enabled, already there is a leader executor, start a new executor online, the result is, the new executor cannot get the job's sharding items, and the zk's node of the job has leader/sharding/processing.

Saturn excutor how deal with the Script job ?？

Saturn uses zookeeper heavily for status update, notification ...
One zookeeper maybe can't support enough jobs
We need to support multiple zookeeper clusters to scale Saturn's capacity horizontally

wiki，架构图，quickstart都没有？

need to run executor & console in foreground for container

Github say that, saturn is a javascript project.....

fix the NullPointerException

Fix the executor's jobNameList when ExecuteAllShardingTask, ExecuteJobEnableShardingTask, and ExecuteJobForceShardShardingTask

support for deploying to inner mvn repo

add new job type: message job.

The sharding logic supports marathon's application.
But when executor needs to be upgraded, we need to a way to migrate the job from the old application to the new application without stopping the job

1、define Dockerfile for saturn-console and saturn-executor
2、quick start demo by docker-compose on zookeeper,saturn-console and saturn-executor

integrated with marathon api with basic authentication code

Remove depandency on haproxy/ngx to support console HA.

Saturn console has done the election between multiple nodes. We can use only master to serve the console management.

If any request to slave node, it will forward the request to master node.

The solution also supports console APIs.

When executor start, it will check its namespace does exist in zookeeper or not.

If the namespace is not in zookeeper, the executor should exit.

Provide a utility in console to scan the zookeeper's junk data

and provide a way to clean such unused data by operator

Using the data to determine which executor is meeting some issues.