kaxil / airflowctl Goto Github PK
View Code? Open in Web Editor NEWA CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects
License: Apache License 2.0
A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects
License: Apache License 2.0
I am trying out airflowctl, and when I setup a new environment and then run it in the background, the PID that is saved in the background_process_ids file is always off by one. It is actually the PID of the shell that ran the command. At the moment I just edit the file, increment the number, and then subsequent commands (stop for eg) then work.
Hello there, I'm not sure if this is the right place to ask this question, but I was wondering what is the recommended way to use airflowctl
, specifically settings.yaml
file, if you want to put an airflow project into a GitHub repo.
I initially thought that you create the project in your dev environment using airflowctl init <project_name> ...
as suggested in the docs. Then, you push the project to the GitHub repo, and once the project is ready for deployment, you just clone the repo into the production environment where you simply run airflowctl build
provided settings.yaml
already contains all the necessary variables for the project to work, but that would be a bad idea from a security perspective if you included connections with credentials in that file, which is my case.
So I don't know If I just have to add settings.yaml
to .gitignore
and create it manually after I clone the repo in production since I already have to do so with the .env
file.
I will appreaciate any suggestion or guidance you can provide. Thanks in advance.
I suspect that this line is raising an exception where none should be raised. The error I get is: Key 'venv_path' not found in settings file
. But I believe the venv_path
is optional and should fall back on None
if the project hasn't configured one (if I understand the code correctly).
Hi, I'm a new user of this project.
I wonder if this project intends to support usecase where users have existing virutal environment.
FYI, I am trying to integrate this project as our local development environment. Since we already installed airflow package, it doesn't need to re-install airflow dependencies.
Airflow installation can fail because of the following:
Running of Airflow can fail because of the following:
When i run the following command:
❯ airflowctl start
I got this error:
Verifying Airflow installation...
/bin/sh: 1: source: not found
Error starting Airflow: Command 'source /home/peter/Desktop/Code/airflow/airflow_2.6.3/.venv/bin/activate && airflow version' returned non-zero exit status 127.
I installed pyenv and airflowctl on Ubuntu 20.04.6 LTS
Any clue about why the "Verifying Airflow installation... process " use sh rather than bash?
Thanks in advance
Similar to the following, there should be a "Next Steps" doc printed after someone runs airflowctl build
to show them the following and their usage:
For init
:
For build
:
Lines 431 to 441 in e4f8fe1
Add an option --persona
in the airflowctl init
command with the following options:
Add directories for those persona in
Build new DAGs for each persona and based on the selected persona, copy those example DAGs over when someone runs `airflowctl init --persona "data-engineer".
all the dags we write for use on gcp composer composer lineage, so probably wouldnt render in vanilla apache airflow.
google store their composer code in this repo: https://github.com/GoogleCloudPlatform/composer-airflow
the branch name designates which version of airlfow+composer to install: https://github.com/GoogleCloudPlatform/composer-airflow/branches/all
these are not published to public pip
on init
offer the options to specify:
airflowctl is affected by the chicken-and-egg problem of Python version. No surprisingly it has the same burden as other Python environment & project managers (e.g., hatch
, pipx
, pyenv
) :
If we start with pip
, the version used in airflowctl built must match the same version as your pip environment:
$ python -V
Python X.Y.Z
$ python -m pip install airflowctl
$ airflowctl init --python-version=X.Y.Z .
$ airflowctl build .
Otherwise pyenv
is invoked to get the correct python version (forcing users to use a specific version manager).
The above is the ideal instruction if we're going to use pip
. With the current instructions, you can run into a lot of issues if you have one of these situation:
--python-version
is not the same as the version used in pip install airflowctl
--python-version
is not specifiedpyenv
is not availablepython
is a different version than the one installed with pip
. For example:
python
and pip
are relatedairflowctl
.Here are my suggestions:
pip
as the primary method.airflowctl
behaviors should not depend on the Python version it is installed to.airflowctl build
should fail if python_version
does not exist in $PATH
. Error message should ask users to have the correct Python version.
brew
, apt
, yum
, winget
, choco
, etc.)Alternatively, although a bigger rewrite, switch to using hatch
or rye
.
airflowctl build
appears to install packages into the "root" (i.e. the non-venv) python environment, which causes dependency conflicts and ultimately fails to initialize the project properly. The expected behavior is to have the packages installed in the venv newly created by airflowctl
. Below is the process I go through for setting up an airflowctl project:
"actl"
).actl
and pip install airflowctl
.airflowctl init ...
to initialize a new project.airflowctl build ...
to set the project up.I'm using python 3.9 on CentOS.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.