Google (GCP) Cloud Composer (Apache Airflow)
Create Cloud Composer (Apache Airflow) Environment
gcloud composer environments create z-cloud-composer1 \
--location us-east1 \
--zone us-east1-b \
--machine-type n1-standard-1 \
--env-variables=gcp_project=project123,gcs_bucket=gs://mybucket,gce_zone=us-east1-b
Delete Cloud Composer (Apache Airflow) Environment
gcloud composer environments delete z-cloud-composer1
--location us-east1
Upload DAG from source (source could be local, gcs path, url, etc. It could also be a path to directory with multiple DAGs.
gcloud composer environments storage dags import
--environment z-cloud-composer1
--location us-east1
--source /tmp/airflow_dags/airflow_demo1.py
Remove/Delete DAG from Cloud Composer
gcloud composer environments storage dags delete
--environment z-cloud-composer1
--location us-east1
my_dag_name
References:
• Google (GCP) Cloud Composer
• Apache Airflow Github
• Apache Airflow Docs