Git Product home page Git Product logo

v-mars / jobor Goto Github PK

View Code? Open in Web Editor NEW
121.0 8.0 28.0 17.17 MB

支持秒级分布式定时任务系统, A high performance distributed task scheduling system, Support multi protocol scheduling tasks

License: GNU Affero General Public License v3.0

Makefile 0.30% Go 92.35% Dockerfile 0.56% Shell 0.18% PLpgSQL 6.61%
task jobs scheduer quartz cron cronjob grpc dispatcher raft high-availability

jobor's Introduction

Jobor分布式定时任务

✨ 功能特性v3.0.4

  • 为简化使用和理解成本,之前v2版本的Master Raft模式已经移除,目前Master节点只支持单节点,如有需要请联系作者购买企业版
  • worker高可用,并且通过路由标识,worker可以部署在不同环境,实现不同环境worker的权限控制,worker的系统环境依赖(Python,Golang,执行依赖的文件)
  • 调度server与worker通过grpc通信
  • 支持LDAP(openldap,AD)协议用户认证
  • 支持SSO/OIDC/Oauth协议用户认证
  • 支持多种任务脚本 [ api/restful请求, shell, python3 ] e.g
  • 基于Casbin实现的权限认证
  • 任务日志自动清理功能

提示

Jobor V3已经修复已知所有Bug,并且新增功能:父子任务执行、执行节点组、worker节点支持agent、ssh两种模式、任务归属负责人、支持国产HTTP,RPC框架CloudWeGo(字节跳动)、支持golang任务类型执行。

架构图

avatar

部署(推荐使用docker模式运行)

第一步,准备好Master/Server和Worker服务启动配置:
server: conf/config.yaml
worker: conf/worker.yaml

第二步,拉取依赖镜像:
docker pull iocean/jobor:server-v3.0.5
docker pull iocean/jobor:worker-v3.0.4
docker pull iocean/jobor:worker-go-v3.0.4
docker pull iocean/jobor:worker-py-v3.0.4

第三步,启动Master/Server和Worker服务:
Server/Master:
docker run -itd --name jobor-server --restart=always --net=host -v /etc/localtime:/etc/localtime -v ${HOST_DIR}/conf:/data/conf -v ${HOST_DIR}/log:/data/log --net=host iocean/jobor:server-v3.0.5
通用 Worker:
docker run -itd --name jobor-worker --restart=always --net=host -v /etc/localtime:/etc/localtime -v ${HOST_DIR}/conf:/data/conf -v ${HOST_DIR}/log:/data/log --net=host iocean/jobor:worker-v3.0.4
执行 Python Worker:
docker run -itd --name jobor-worker-py --restart=always --net=host -v /etc/localtime:/etc/localtime -v ${HOST_DIR}/conf:/data/conf -v ${HOST_DIR}/log:/data/log --net=host iocean/jobor:worker-py-v3.0.4
执行 Golang Worker:
docker run -itd --name jobor-worker-go --restart=always --net=host -v /etc/localtime:/etc/localtime -v ${HOST_DIR}/conf:/data/conf -v ${HOST_DIR}/log:/data/log --net=host iocean/jobor:worker-go-v3.0.4

构建

go build -o ./app ./main.go
go build -o ./app ./cmd/worker/main.go

启动Server(推荐使用docker模式运行)

go build -o ./app ./main.go

./app -c ./conf/config.yaml

启动Worker(推荐使用docker模式运行)

go build -o ./app ./cmd/worker/main.go
./app -c ./conf/worker.yaml

默认

username: admin password: admin

DB

数据库推荐使用:MySQL8.0
字符集:utf8mb4
在配置文件最后一行粘贴以下语句
/etc/mysql/conf.d/mysql.cnf

[mysqld]
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION

Jobor预览

avatar avatar avatar avatar avatar

TODO

task

  • 支持server/controller/master(通过raft一致性算法)的高可用,一个Raft集群通常包含2*N+1个服务器,允许系统有N个故障服务器(企业版)。
  • ldap(支持openldap,AD 认证)
  • SSO/OIDC/Oauth(支持SSO/OIDC/Oauth认证)
  • server <-- gRPC --> worker
  • task abort
  • task timeout
  • api/restful [GET, POST, PUT, DELETE] task
  • shell task
  • python3 task
  • golang task
  • father task
  • children task
  • worker 预执行(如:执行python 前先执行 pip install xx)
  • worker 节点支持:agent和ssh两种模式
  • 路由标识多选
  • 任务缓存执行

🤝 特别感谢

支持

1、希望大家多多支持,给项目一个star

2、该项目花费了作者大量时间,如果你觉的该项目对你有用,希望可以友情赞助一下

开源、企业版区别

功能 开源 企业
前端源码 静态产物 支持
Master高可用 - 支持
Worker高可用 支持 支持
LDAP认证 支持 支持
OIDC认证 支持 支持
Shell任务 支持 支持
API/HTTP任务 支持 支持
Python任务 支持 支持
Golang任务 支持 支持
父子任务 支持 支持
定制功能 Issues 立刻
服务支持 Issues 微信群

交流/商务联系

如果您只是使用本项目的话,您可以在提出您使用中需要改进的地方,我会尽快修改。
如果您是想基于此项目二次开发的话,您可以提出您在开发过程中的任何疑问,我会尽快答复并讲解。

还可提供关联其他常用系统服务

  • 还可提供SSO(OIDC)统一认证服务
  • 运维CMDB系统服务
  • 运维工单服务系统
  • 持续集成发布系统,支持k8s、docker、虚拟机、静态资源发布,支持当下流程的染色发布、灰度发布、迭代流水线发布等,代码打包支持原生命令行、Dockerfile、Hook提交等构建方式

jobor's People

Contributors

her-cat avatar joshle avatar laixiaohuai avatar xhrg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jobor's Issues

Upgrade dependency "github.com/gin-gonic/gin"

Background

Repo github.com/v-mars/jobor depends on github.com/gin-gonic/[email protected].

https://github.com/v-mars/jobor/blob/main/go.mod#L13

However, comparing version v1.8.0 of github.com/gin-gonic/gin from proxy.golang.org and github, there are inconsistencies.

commit time of the copy on github.com

"committer": {
      "name": "GitHub",
      "email": "[email protected]",
      "date": "2022-05-30T07:16:10Z"
    }

commit time of the copy on proxy.golang.org

{"Version":"v1.8.0","Time":"2022-05-28T07:23:00Z"}

So the checksum from the code in github does not match the checksum saved in sum.golang.org. The v1.8.0 tag of github.com/gin-gonic/gin might have been retagged after a minor edition on github. I guess you use proxy.golang.org to get dependencies, but that also shows that your project is depending on the copy of github.com/gin-gonic/[email protected] before its edition. Depending upon such inconsistent tag version may also result in some unexpected errors as well as build errors due to different proxy settings.

For example, when someone who does not use proxy.golang.org, say GOPROXY=direct, attempts to get github.com/gin-gonic/[email protected], the following errors occur.

go: downloading github.com/gin-gonic/gin v1.8.0
go: github.com/gin-gonic/gin@v1.8.0: verifying module: checksum mismatch
        downloaded: h1:gcAUpegZ91QWGm03w0CjjJ0+rjxmG3pbBbT0aiT82s8=
        sum.golang.org: h1:4WFH5yycBMA3za5Hnl425yd9ymdw1XPm4666oab+hv4=

SECURITY ERROR
This download does NOT match the one reported by the checksum server.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

So, this is a reminder in the hope that you can get rid of this unreliable version of project github.com/gin-gonic/gin.

Solution

1. Bump the version of dependency github.com/gin-gonic/gin

I would recommend bumping the version of github.com/gin-gonic/gin to a new release to ensure dependency copy in proxy.golang.org and github in sync.

References

数据库初始化报错

直接启动server docker 报错
install.go:129: [Error] Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INSERT IGNORE INTO api (id, name, title, dom, path, method, `enabl' at line 30

按要求配置了mysqld,mysql启动失败,报错如下

2023-10-11T09:25:41.567997Z 0 [ERROR] [MY-000077] [Server] /usr/sbin/mysqld: Error while setting value 'STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION' to 'sql_mode'.
2023-10-11T09:25:41.571308Z 0 [ERROR] [MY-010119] [Server] Aborting

二进制部署,访问网页报错,unexpected fault address 0x0

访问网页报错,http://192.168.0.236:5000/#/

[GIN] 2023/08/13 - 14:44:40 | 200 | 131.37023ms | 192.168.0.11 | GET "/static/js/chunk-elementUI.8e263ae8.js"
[GIN] 2023/08/13 - 14:44:40 | 200 | 169.491892ms | 192.168.0.11 | GET "/static/js/chunk-libs.a1ab8e61.js"
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x4664df]

goroutine 32 [running]:
runtime.throw({0x123b7db?, 0x7f290fc3d2c0?})
/opt/tool/go/src/runtime/panic.go:992 +0x71 fp=0xc000fd6ab8 sp=0xc000fd6a88 pc=0x435f71
runtime.sigpanic()
/opt/tool/go/src/runtime/signal_unix.go:825 +0x305 fp=0xc000fd6b08 sp=0xc000fd6ab8 pc=0x44c1c5
aeshashbody()
/opt/tool/go/src/runtime/asm_amd64.s:1343 +0x39f fp=0xc000fd6b10 sp=0xc000fd6b08 pc=0x4664df
runtime.mapiternext(0xc000127100)
/opt/tool/go/src/runtime/map.go:934 +0x2cb fp=0xc000fd6b80 sp=0xc000fd6b10 pc=0x40fd2b
runtime.mapiterinit(0xc0001dc750?, 0x0?, 0xc00256c150?)
/opt/tool/go/src/runtime/map.go:861 +0x228 fp=0xc000fd6ba0 sp=0xc000fd6b80 pc=0x40fa08
reflect.mapiterinit(0x10a2f40?, 0xc0001dc758?, 0xc000fd6cb0?)
/opt/tool/go/src/runtime/map.go:1373 +0x19 fp=0xc000fd6bc8 sp=0xc000fd6ba0 pc=0x462919
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
/opt/tool/golang/pkg/mod/github.com/modern-go/[email protected]/unsafe_map.go:112
github.com/json-iterator/go.(*sortKeysMapEncoder).Encode(0xc00256c2d0, 0xc0001882e8, 0xc00007f7a0)
/opt/tool/golang/pkg/mod/github.com/json-iterator/[email protected]/reflect_map.go:291 +0x225 fp=0xc000fd6d30 sp=0xc000fd6bc8 pc=0xbf5da5
github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc0002f5f20, 0xc000a1df80, 0xc000127100?)
/opt/tool/golang/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:219 +0x82 fp=0xc000fd6d68 sp=0xc000fd6d30 pc=0xbee522

可以有docker版本的么

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

登录报错

[GIN] 2021/05/31 - 17:56:58 | 200 | 1.06781ms | 127.0.0.1 | POST "/api/login"
[ERROR] 2021-05-31 17:56:58 - 用户: nil, 方法: POST, URL: /api/login, CODE: 200, 耗时: 1ms, Body数据: {"username":"admin","password":"admin"},
ERROR: Error 1054: Unknown column 'user_type_id' in 'where clause'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.