GitOps Connector

GitOps Connector is a custom component with the goal of enriching the integration of a GitOps operator and a CI/CD orchestrator so the user experience in the entire CI/CD process is smoother and more observable. The whole process can be handled and monitored from a CI/CD orchestrator.

During the reconciliation process a GitOps operator notifies on every phase change and every health check the GitOps connector. This component serves as an adapter, it "knows" how to communicate to a Git repository and it updates the Git commit status so the synchronization progress is visible in the Manifests repository. When the reconciliation including health check has successfully finished or failed the connector notifies a CI/CD orchestrator, so the CD pipelines/workflows may perform corresponding actions such as testing, post-deployment activities and moving on to the next stage in the deployment chain.

Refer to the following implementations to understand the role the GitOps Connector plays in various GitOps flows:

• GitOps with Azure DevOps and ArgoCD/Flux
• HLD based CI/CD Pipeline with GitOps connector
• GitOps with GitHub and Flux v2

Motivation

One of the biggest challenges in GitOps is observability. When a CI/CD pipeline orchestrates the deployment process by manipulating manifests in the repository, the real deployment is handled by a GitOps operator. When it happens, how it goes and when it finishes, successfully or not, we don't know. Neither does the CI/CD pipeline. To build a multistage CD pipeline there must be a backward connection from a GitOps operator to CI/CD orchestrator that reports the deployment state/result so the CD pipeline can proceed with testing, notifying, deploying to next environment.

Partly this challenge can be addressed by leveraging notifications components of GitOps operators such as FluxCD Notification Controller and ArgoCD Notifications. It's possible to configure basic notifications from Flux/ArgoCD to Azure DevOps/GitHub so the deployment phases are reflected with the Git Commit status in the repo.

So why cannot we "just" use the existing notification functionality of Argo/Flux events?

The thing is that "basic" is not enough.

What we need:

• A CD pipeline (e.g. Azure Pipeline) creates a PR to the manifests repo (e.g. GitHub) and waits until it's merged and deployment is finished. When the deployment is finished we need to find out which PR caused this deployment and invoke an Azure Pipelines API to notify a corresponding pipeline run to resume or fail.
• In Git Commit status we want to see more details on what resources have been applied and what resources didn't pass the health check.

We simply can't do what we need just by configuring notifications in FluxCD/ArgoCD. To implement the logic of what we need we want to write some code and run it somewhere. We need some freedom in getting out of the boundaries of what is "out-of-the-box" in FluxCD/ArgoCD and extending it with custom logic to build a comprehensive CD flow with Azure Pipelines and/or GitHub actions. At the same time we want to keep pipelines simple and not dependent on underlying GitOps operator. Hence, the GitOps connector.

Why cannot we enhance ArgoCD/FluxCD notification functionality to achieve the same?

Actually we can. The only thing that we'd have to do that twice. And again, when we support another GitOps operator. And separately for GitHub and Azure DevOps. And again when we support another CI/CD orchestrator. So should we? The GitOps connector decouples ArgoCD/FluxCD from GitHub/Azure DevOps. It reduces dependencies serving as a glue between them. Think of the GitOps connector as of an extension of FluxCD/ArgoCD (one for both) or, in terms of FluxCD notifications, a custom provider for GitHub, Azure DevOps, various observability dashboards (e.g. Spektate). A custom provider extracted as a separate microservice with an independent lifecycle. When we want to build same CI/CD flow with another GitOps operator, we will update the connector keeping the same pipelines setup. When we want to enrich look-n-feel of Git Commit status or use a new feature in Azure DevOps / GitHub or start supporting another CI/CD orchestrator we will update the connector without touching FluxCD/ArgoCD at all. Hence, the GitOps connector.

GitOps operators

The GitOps Connector supports the following Git Ops operators:

• FluxCD
• ArgoCD

Git Commit Status Update

The GitOps Connector supports the following Git repositories:

• Azure Repos
• GitHub

The connector reports the deployment process updates to the Git repositories with the Git commit status:

Commit Status in GitHub	Commit Status in Azure Repos

In addition to updating the Git commit status in Git repositories, the GitOps connector may optionally notify a list of custom subscribers with a json payload:

Refer to installation guide for the details on configuring a list of custom subscribers.

Notification on Deployment Completion

The GitOps connector analyzes the incoming messages from the GitOps operator and figures out when the deployment is finished, successfully or not. It notifies on this event the CI/CD orchestrator, so the orchestrator can proceed with the CD process.

The notification mechanism on deployment completion varies for different CI/CD orchestrators.

Azure Pipelines

An Azure CD pipeline is supposed to create a PR to the manifests repo and wait in agentless mode until the PR is merged and the GitOps operator finishes the deployment. It may use PR properties as a storage for the agentless task callback parameters. When the deployment is finished, the GitOps connector seeks the PR to the manifest repo that caused the deployment and looks for the PR's properties under the /callback-task-id path. It expects to find a json object with the following data:

{"taskid":"$(System.TaskInstanceId)",
 "jobid:"$(System.JobId)",
 "planurl":"$(System.CollectionUri)",
 "planid":"$(System.PlanId)",
 "projectid":"$(System.TeamProjectId)",
 "pr_num":"$(pr_num)"}

The GitOps connector uses this data to provide a callback to the agentless task with the following API:

POST {planurl}{projectid}/_apis/distributedtask/hubs/build/plans/{planid}/events?api-version=2.0-preview.1
payload = {
    'name': "TaskCompleted",
    'taskId': {taskid},
    'jobid': {jobid},
    'result': "succeeded"/"failed"
}

Refer to a sample of such agentless task for the implementation details.

GitHub Actions

If the CI/CD orchestrator is GitHub Actions, the GitOps connector sends a dispatch event on the successful deployment completion:

POST /repos/{owner}/{repo}/dispatches
payload = {
    'event_type': "sync-success",
    'client_payload': {'sha': {commmit_id},                //Commit Id in source repo that started the CI/CD process 
                      'runid': {github_workflow_run_id}    //GitHub Actions Workflow RunId that produced artifacts (e.g. Docker Image Tags)
    }

If the deployment fails, the connector doesn't send a dispatch event to GitHub Actions.

For the implementation details refer to the CD workflow in this repo that consumes CI artifacts, generates manifests and issues a PR to the manifests repo. Also, look at the publish workflow which is triggered on the deployment completion by the dispatch event from the GitOps connector.

Installation

Install GitOps Connector with Helm

Add GitOps Connector repository to Helm repos:

helm repo add gitops-connector https://azure.github.io/gitops-connector/

Prepare values.yaml file with the following attributes:

If Git Repository Type is AZDO, add the following attributes:

If CI/CD Orchestrator Type is AZDO, add the following attributes:

If Git Repository Type is GITHUB, add the following attributes:

If CI/CD Orchestrator Type is GITHUB, add the following attributes:

Optional. If there are custom Git Commit status subscribers:

A sample values.yaml file for Flux and GitHub might look like this one:

gitRepositoryType: GITHUB          
ciCdOrchestratorType: GITHUB
gitOpsOperatorType: FLUX
gitHubGitOpsRepoName: gitops-connector
gitHubGitOpsManifestsRepoName: gitops-manifests
gitHubOrgUrl: https://api.github.com/repos/kaizentm
gitOpsAppURL: https://github.com/kaizentm/gitops-manifests/commit
orchestratorPAT: <PAT>
subscribers:
    spektate: 'http://spektate-server:5000/api/flux'

A sample values.yaml file for Flux and Azure DevOps might look like this one:

gitRepositoryType: AZDO          
ciCdOrchestratorType: AZDO
gitOpsOperatorType: FLUX
azdoGitOpsRepoName: manifest-repo
azdoOrgUrl: https://dev.azure.com/MyOrg/MyProject
azdoPrRepoName: hld-repo
gitOpsAppURL: https://github.com/microsoft/spektate
orchestratorPAT: <PAT>
subscribers:
    spektate: 'http://spektate-server:5000/api/flux'

Install GitOps connector with the following command:

helm upgrade -i gitops-connector gitops-connector/gitops-connector \
--namespace <NAMESPACE> \
--values values.yaml

#  Check GitOps connector is up and running:
kubectl get pods -l=app=gitops-connector -n <NAMESPACE>

#  Check the connector logs:
kubectl logs -l=app=gitops-connector -n <NAMESPACE> -f
#  DEBUG:root:0 subscribers added.
#  INFO:timeloop:Starting Timeloop..
#  [2021-05-25 00:19:23,595] [timeloop] [INFO] Starting Timeloop..
#  [2021-05-25 00:19:23,595] [timeloop] [INFO] Registered job <function pr_polling_thread_worker at 0x7f3df4f30550>
#  [2021-05-25 00:19:23,596] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
#  INFO:timeloop:Registered job <function pr_polling_thread_worker at 0x7f3df4f30550>
#  INFO:timeloop:Timeloop now started. Jobs will run based on the interval set
#  INFO:root:Starting commit status thread
#  INFO:root:Starting periodic PR cleanup
#  INFO:root:Finished PR cleanup, sleeping for 30 seconds...

Configure FluxCD to send notifications to GitOps connector

FluxCD Notification Controller sends notifications to GitOps connector on events related to GitRepository and Kustomization Flux resources. Apply the following yaml to the cluster to subscribe GitOps connector instance on Flux notifications:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: gitops-connector
  namespace: <NAMESPACE>
spec:
  eventSeverity: info
  eventSources:
  - kind: GitRepository
    name: <Flux GitRepository to watch>
  - kind: Kustomization
    name: <Flux Kustomization to watch>
  providerRef:
    name: gitops-connector
---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: gitops-connector
  namespace: <NAMESPACE>
spec:
  type: generic
  address: http://gitops-connector:8080/gitopsphase

Configure ArgoCD to send notifications to GitOps connector

ArgoCD Notifications sends updates on every deployment phase. Apply the following yaml to the cluster to subscribe GitOps connector instance on ArgoCD notifications:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: <NAMESPACE>
data:
  config.yaml: |
    triggers:
      - name: sync-operation-failed
        condition: app.status.operationState.phase in ['Error', 'Failed']
        template: sync-operation-status-change
      - name: sync-operation-succeeded
        condition: app.status.operationState.phase in ['Succeeded']
        template: sync-operation-status-change
      - name: sync-operation-running
        condition: app.status.operationState.phase in ['Running']
        template: sync-operation-status-change
      - name: sync-operation-progressing
        condition: app.status.health.status in ['Progressing']
        template: sync-operation-status-change
      - name: sync-operation-healthy
        condition: app.status.health.status in ['Healthy'] && app.status.operationState.phase in ['Succeeded']
        template: sync-operation-status-change
      - name: sync-operation-unhealthy
        condition: app.status.health.status in ['Unknown', 'Suspended', 'Degraded', 'Missing']
        template: sync-operation-status-change
    templates:
      - name: sync-operation-status-change
        webhook:
          test-receiver:
            method: POST
            body: |
              {
                "commitid": "{{.app.status.operationState.operation.sync.revision}}",
                "phase": "{{.app.status.operationState.phase}}",
                "sync_status": "{{.app.status.sync.status}}",
                "health": "{{.app.status.health.status}}",
                "message": "{{.app.status.operationState.message}}",
                "resources": {{toJson .app.status.resources}}
              }
---
apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: <NAMESPACE>
stringData:
  notifiers.yaml: |
    webhook:
    - name: test-receiver
      url: http://gitops-connector:8080/gitopsphase
      headers:
      - name: Content-Type
        value: application/json
type: Opaque      

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.