Git Product home page Git Product logo

airflow-jobs's Introduction

airflow-jobs

请注意: 项目默认分支为 development, 所有pr默认提交到 development 分支

git 操作文档

clickhouse建表未明确字段自定义值记录

Dag 测试同步 github commit

{
    "owner" : "apache",
    "repo" : "tomcat",
    "since" : "1980-01-01T00:00:00Z",
    "until" : "2021-12-12T00:00:00Z"
}

Dag 测试同步 github commit

{
    "HSOT" : "apache",
    "PORT" : "tomcat",
    "USER" : "1980-01-01T00:00:00Z",
    "PASSWD" : "2021-12-12T00:00:00Z"
}

#需要设置更多的token

更新docker镜像

a.停止容器运行: docker-compose stop
b.删除容器: docker-compose rm
c.更新docker镜像: docker pull airflowjobs/airflow:dev-python3.8
d.设置airflow用户: echo -e "AIRFLOW_UID=$(id -u)" > .env
e.查看airflow用户设置是否成功:ll -a--->cat .env
f.启动docker镜像: docker-compose up

关于os 数据插入或更新的时间戳

search_key{
    .....
    "updated_at":"时间戳 本条记录更新(包含插入)的时间戳"
    'updated_at': int(datetime.datetime.now().timestamp() * 1000)
}
要保证时间戳为13位整数

关于os中的日期格式

所有os中的时间都是 2022-01-16T16:21:23Z,时区单独存储,原始时间字符串单独存储

关于clickhouse 反推数据建表的数据提供

1、数据字段要求完整
2、数据要求明确,不能有null
3、数据中不能出现空列表 如[]
4、数据中如果出现[]中嵌套多层json结构需要说明提醒(一层的json可以),或者再嵌套列表也需要说明提醒

关于启动airflow需添加.env文件

在airflow-jobs目录下执行
echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env

为不同的docker容器创建网络

docker network create -d bridge dev-network

airflow-jobs's People

Contributors

fivestarsky avatar ynang avatar crystaldust avatar shanchenqi avatar hexaemeronfsk avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.