diggerhq / digger Goto Github PK

Some advantages of this rewrite:

• run native golang library in workflow without dockerfile
  • This will lead to speedup in action running
  • ability to distribute to other CI systems easily
• aligned with other tools in infra community

For some scenarios when the setup is merge to main and then apply we can support an approval flow via slack or other mechanism. This could also include conditional approval, for example "I want to manually approve all PRs except the ones that only modify resources from the following whitelist"

bin folder looks a little bit ugly

a config option that'd change behavior to run terragrunt plan and terragrunt apply command instead of terraform

similar to #114 if user tries to connect a digger action and there is no digger.yml file AND no terraform files exist in the root directory then no errors are shown to the user instead the following error is shown:

terraform plan failed. dir: .
2023/04/07 04:39:35 Error executing plan: terraform plan failed. exit status 1
Error: Process completed with exit code 1.

We would like to improve this by showing a clearer error message

for example in this case diggerhq/digger_demo_multienv#12 files changed does not include prod so if we comment digger plan -p prod nothing happens and it is confusing for the user. We should instead comment to the user and say something like "warning: you requested plan for a project which does not have any files modified"

For supporting deployments to multiple environments users need to be able to specify a -backend-config option which is passed to terraform init command, in this command they will be able to specify which bucket or remote state location as well as state file they will be using. This will unlock all kinds of workflows such as apply to prod on merge (apply to dev on PR), multiple states and so on ..

Opinions differ on this; see merge-apply dilemma

• Some prefer to iteratively apply on a branch, then merge once
• Others prefer to only deploy from main, creating additional fix PRs if apply breaks

Currently the default in Digger is branch-deploy model

We might want an additional optional strategy to support apply-on-merge

Allow digger to run in the context of gitlab pipelines

show what are the possible commands to use when user comments digger or digger help or another incorrect digger command

"if I apply a one line change in big YAML for helm_release, in the output I'll see two gigantic documents with one different line which makes the output near unreadable.

I think, if you will able to make compact but informative output it may be a big plus."

by @AntonChekunov

GCP bucket names are globally unique. We assumed that they would not be globally unique. We had hardcoded the value of bucket as digger-lock so now we want to instead make the user define it using the environment name directly

if the environment variable is not set and we are using GCP then we throw an error asking the user to set this value GOOGLE_STORAGE_BUCKET

I'm interested in this project and I forked it and git clone, however it is too big, almost 200 MB in fact, so that I had given up once. I was confused that why so big it is, and I run command below:

git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | awk '/^blob/ {print substr($0,6)}' | sort --numeric-sort --key=2 | cut --complement --characters=13-40 | numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

And I found there are 3 files not used now but in git commit log.

Perhaps the team is not very familiar with Git, but a large repository can prevate many interested individuals from cloning it. Hope maintainers can remove these unnecessary files at once and make it easier to git clone.

The following doesn't throw any warnings even though its invalid digger.yml, we need to ensure that at minimum a list of valid projects is defined in the digger.yml otherwise we throw an error

project:
  name: terraform-monorepo
services:
  project_a_d:
    service_name: project_a_d
    path: ./project_a/development
  project_a_n:
    service_name: project_a_n
    path: ./project_a/non-production
  project_a_p:
    service_name: project_a_p
    path: ./project_a/production
  project_a_s:
    service_name: project_a_s
    path: ./project_a/shared
  project_b_d:
    service_name: project_b_d
    path: ./project_b/development
  project_b_n:
    service_name: project_b_n
    path: ./project_b/non-production
  project_b_p:
    service_name: project_b_p
    path: ./project_b/production

digger currently supports Github, but we also want to extend it to other SCM vendors:

• Bitbucket
• Azure DevOps
• GitLab

However, to do that the codebase needs to be adapted:

• Start using a PullRequestService struct
• Add SCMProvider interface as property

Then instead of using Github or Bitbucket directly, we'll call PullRequestService which will rely on the underlying SCM provider.
The right SCM provider will be injected when creating a PullRequestService.

Example of implementation, check below

There seems to already be a PullRequestManager (our interface):

type PullRequestManager interface {
	GetChangedFiles(prNumber int) ([]string, error)
	PublishComment(prNumber int, comment string) error
}

The PR Service:

type PullRequestService struct {
	pr PullRequestManager
}

func (prs *PullRequestService) PublishComment(prNumber int, comment string) error {
    return prs.pr.PublishComment(prNumber, comment)

Bitbucket implementation example:

type Bitbucket struct {
}

func (bb *Bitbucket) GetChangedFiles(prNumber int) ([]string, error) {
	return []string{}, nil
}

func (bb *Bitbucket) PublishComment(prNumber int, comment string) error {
	return nil
}

Github implementation example:

type Github struct {
}

func (gh *Github) GetChangedFiles(prNumber int) ([]string, error) {
	return []string{}, nil
}

func (gh *Github) PublishComment(prNumber int, comment string) error {
	return nil
}

Full workflow (kept easy to be readable):

//Detect scm vendor (or will be defined in digger.yaml)
scmVendor := getScmVendor(&config)
var prManager PullRequestManager

// Use switch or if/else
if scmVendor == "Bitbucket" {
    prManager := prManager.(Bitbucket)
}

if scmVendor == "Github" {
     prManager := prManager.(Github)
}

prService := PullRequestService {
    pr: prManager
}
prService.PublishComment(2, "digger apply")

Since we have an event of

on: 
    push: 
        branches: [ "main" ]

Results in the following error from digger side:

Failed to parse GitHub context. error parsing GitHub context JSON: unknown GitHub event: push
Error: Process completed with exit code 3.

We need to make sure this event is properly handled even if nothing is performed

write workflow file to fail checks if code isn't formated with black

digger currently supports DynamoDB for locks. However, in Azure DynamoDB service doesn't exist (there's a CosmosDB, but not the same schema).

A more native way in Azure would be to use a Storage Account. Actually the same storage account where the terraform state is kept. We would rely on Storage Account table to store our lock.

But before that, we need to adjust the codebase to start using multiple cloud providers.

• Start using a LockService struct
• Add CloudLock interface as property

Then instead of using DynamoDB directly, we'll call LockService which will rely on the underlying CloudLock provider.
The right lock provider will be injected when creating a LockService.

Example of implementation, check below

// Add the right param and return type
type CloudLock interface {
	Lock()
        Unlock()
        ForceUnlock()
}

The Lock Service:

type LockService struct {
	lockProvider CloudLock
}

func (ls *LockService) Lock() {
    return ls.lockProvider.Lock()

StorageAccount in Azure, implementation example:

type StorageAccountLock struct {
}

func (sal *StorageAccountLock) Lock()  {
	// add table with data to storage account
}

Full workflow (kept easy to be readable):

//Detect cloud vendor (or will be defined in digger.yaml)
lockVendor := getLockVendor(&config)
var cloudLock CloudLock

// Use switch or if/else
if lockVendor == "azure" {
    cloudLock := cloudLock.(StorageAccountLock)
}

if lockVendor == "aws" {
     cloudLock := cloudLock.(DynamoDBLock)
}

lockService := LockService {
    lockProvider: cloudLock
}
lockService.Lock()

Currently we are using names of default credential parameters such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to give digger access to account for locks. These credentials are also used by terraform to access cloud account since they are the default credentials.

For some use cases such as #55 these sets of keys need to be different for more fine grained scoped access.

Proposal to solve this is to optionally prefix all digger specific keys with DIGGER_. Digger should look for DIGGER_ first and if not found then use AWS_ACCESS_KEY_ID as keys for lock access. This change should not impact the way the terraform is run by terraform already.

For AWS

Use the following keys for lock aquisition

DIGGER_AWS_ACCESS_KEY_ID
DIGGER_AWS_SECRET_ACCESS_KEY

If not available we use the default keys

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

For GCP

Use the following parameter as service credentials for lock aquisition

DIGGER_ GCP_CREDENTIALS

if not available we use the default parameter value

GCP_CREDENTIALS

For Azure

Use the following parameters for account access

DIGGER_ARM_SUBSCRIPTION_ID
DIGGER_ARM_TENANT_ID
DIGGER_ARM_CLIENT_ID
DIGGER_ARM_CLIENT_SECRET

If not available we revert back to the following keys

ARM_SUBSCRIPTION_ID
ARM_TENANT_ID
ARM_CLIENT_ID
ARM_CLIENT_SECRET

Hi

I wanted to try digger but running in a few issues:

• The enviroment variables to use for dynamodb credentials are the same as s3 backend. We use a local minio so those are not the same for us.
• Missing documentation on what is needed for dynamodb.

currently in our demo repositories the action files are pointing to diggerhq/tfrun which is the old name of digger repo, we need to make sure they all point to diggerhq/digger for example: https://github.com/diggerhq/digger_demo/blob/main/.github/workflows/plan.yml#L39

and all the other demo repository yml files need to be updated

In this case I run digger plan from PR#2 and the following comment is posted to indicate that the lock is already acquired by PR#2.

common scenario: production "gate" to have a manual approval by a human

Currently testing the workflow proposed in https://github.com/diggerhq/digger-gcp-lock-demo.

One difference is that I have configured Workload identity federation to authenticate with Google using OIDC.

Also have made a change in the Use gcloud CLI by listing the content of the bucket to prove that service account has storage.buckets.get permission on it.

Workflow:

name: CI

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]
    types: [ closed, opened, synchronize, reopened ]
  issue_comment:
    types: [created]
    if: contains(github.event.comment.body, 'digger')
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    permissions:
      contents: read
      id-token: write
      pull-requests: write

    env:
      SERVICE_ACCOUNT: sa-ar-digger-gha-any@prj-c-artifacts-62d0.iam.gserviceaccount.com
      GOOGLE_STORAGE_BUCKET: terraform-digger-gha-pr-locks-814b

    steps:
      - uses: actions/checkout@v3

      - name: Checkout Pull Request
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          PR_URL="${{ github.event.issue.pull_request.url }}"
          PR_NUM=${PR_URL##*/}
          echo "Checking out from PR #$PR_NUM based on URL: $PR_URL"
          hub pr checkout $PR_NUM
        if: github.event_name == 'issue_comment'
        
      - id: auth
        uses: google-github-actions/auth@v1
        with:
          token_format: access_token
          workload_identity_provider: ${{ secrets.GCP__GHA__WORKLOAD_IDENTITY_PROVIDER }}
          service_account: ${{ env.SERVICE_ACCOUNT }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v1

      - name: Use gcloud CLI
        run: |
          gcloud info
          gsutil ls gs://${{ env.GOOGLE_STORAGE_BUCKET }}

      - name: digger tfrun
        uses: diggerhq/[email protected]
        env:
          LOCK_PROVIDER: gcp
          GITHUB_CONTEXT: ${{ toJson(github) }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Output:

...
Run gcloud info
Installation Properties: [/opt/hostedtoolcache/gcloud/424.0.0/x64/properties]
User Config Directory: [/home/runner/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/home/runner/.config/gcloud/configurations/config_default]

Account: [sa-ar-digger-gha-any@prj-c-artifacts-62d0.iam.gserviceaccount.com]
Project: [prj-c-artifacts-62d0]

Current Properties:
  [auth]
    credential_file_override: [/home/runner/work/digger-test/digger-test/gha-creds-43e0236a61a1ec40.json] (environment)
  [core]
    account: [sa-ar-digger-gha-any@prj-c-artifacts-62d0.iam.gserviceaccount.com] (property file)
    disable_usage_reporting: [True] (property file)
    project: [prj-c-artifacts-62d0] (environment)
  [metrics]
    environment: [github-actions-setup-gcloud] (environment)
    environment_version: [1.1.0] (environment)

Logs Directory: [/home/runner/.config/gcloud/logs]
Last Log File: [/home/runner/.config/gcloud/logs/2023.04.04/14.49.[55](https://github.com/nuecho/digger-test/actions/runs/4609142469/jobs/8145864767#step:6:56).0233[65](https://github.com/nuecho/digger-test/actions/runs/4609142469/jobs/8145864767#step:6:66).log]

git: [git version 2.40.0]
ssh: [OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022]


gs://terraform-digger-gha-pr-locks-814b/test.txt

...

Run diggerhq/[email protected]
Run curl -sL [https://github.com/diggerhq/digger/releases/download/${actionref}/digger-Linux-X64](https://github.com/diggerhq/digger/releases/download/$%7Bactionref%7D/digger-Linux-X64) -o digger
Digger config read successfully
Lock has been created successfully
GitHub context parsed successfully
GitHub event processed successfully
GitHub event converted to commands successfully
Lock nuecho/digger-test#default
failed to get bucket attributes: googleapi: Error 403: sa-ar-digger-gha-any@prj-c-artifacts-62d0.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist)., forbidden
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xcfef04]

goroutine 1 [running]:
digger/pkg/gcp.(*GoogleStorageLock).Lock(0xc00007f4e0, 0xc0007841a0?, {0xc0007841a0, 0x1a})
	/home/runner/work/digger/digger/pkg/gcp/gcp_lock.go:41 +0x384
digger/pkg/utils.(*ProjectLockImpl).Lock(0xc00070bcc0, {0xc0007841a0, 0x1a}, 0x1)
	/home/runner/work/digger/digger/pkg/utils/locking.go:60 +0x369
digger/pkg/digger.DiggerExecutor.Plan({{0x0, 0x0}, {0x100b75a, 0x7}, {0xc0000387f8, 0x6}, {0x100b75a, 0x7}, {0x1007b20, 0x1}, ...}, ...)
	/home/runner/work/digger/digger/pkg/digger/digger.go:233 +0x253
digger/pkg/digger.RunCommandsPerProject({0xc00070bc70?, 0x1, 0xc0001181e0?}, {0xc0000387f8, 0x6}, {0xc0000387ff, 0xb}, {0xc0001182a0, 0xc}, 0x1, ...)
	/home/runner/work/digger/digger/pkg/digger/digger.go:76 +0x605
main.main()
	/home/runner/work/digger/digger/cmd/digger/main.go:67 +0x709
Error: Process completed with exit code 2.  

While debugging #111 the likely cause is DynamoDB resource mismatch
It could be addressed by fixing #112 - but creating a separate one just to make sure that DynamoDB specifically gets addressed

We currently read a configuration file for specifying config options. This configuration file is specified directly in the repository. We only support digger.yml at the moment, we want to also support digger.yaml along with it if it exists.

If both files exists then its good to throw an error saying that you are not sure which one to chose, please delete one of the files.

The following should not trigger a comment action

Support running digger within GCP code build CI/CD system along with starting documentation for it

It would be cool, if instead of having to set up Github tokens for the AWS keys, this could be neatly integrated with 99designs/aws-vault. On privately hosted Github Actions Runners, this would make things a lot more secure.

As mentioned in ISS #103

To make things like [#111] easier to debug
Can so far thinking of the following info that can be useful:

• which cloud provider?
• which CI provider?
• call stack trace

Also we probably need more granular exception tree structure with some hints to what exactly went wrong

cc @motatoes @Spartakovic @veziak

Different projects might require different versions of Terraform
Use case at Wise as shared by Pavel

For passing extra args to terraform init, terraform plan and terraform apply command

Allow digger to run in bitbucket pipelines

I keep getting the following error:

Error locking project: ResourceNotFoundException: Requested resource not found

Any way to debug what is going on?

projects:
- name: 00_myvm
  dir: dev/env/00_myvm
  terragrunt: true

https://github.com/diggerhq/digger/blob/main/pkg/utils/mocks.go#L17

and

https://github.com/diggerhq/digger/blob/main/pkg/aws/dynamo_mock.go

are identical implementation of map-based lock. We should get rid of one of them in favour of the other as a refactoring step

Support running digger in Jenkins pipeline

Help command should take precedence over environment variables.
digger --help

Error: GITHUB_TOKEN is not defined

Expected behavior:
Print help text

As mentioned in #111 Private Github runners needs support since we don't package terraform or terragrunt into the image, and there is no official image for the runners then users who attempt to run digger get errors that the terraform executable is not found:

time=2023-04-11T08:21:27Z level=error msg=exec: "terraform": executable file not found in $PATH
time=2023-04-11T08:21:27Z level=error msg=Unable to determine underlying exit code, so Terragrunt will exit with error code 1
2023/04/11 08:21:27 Error executing plan: error: exit status 1

Options:

• Provide some maintained official image that contains these executable backed in for digger cli to run in
• Provide some instructions for users to create and install required dependencies in their own runners

Currently the requirement to support cloud account keys (AWS, GCP, Azure) is due to the fact that we need them to provision and use resources for locking. This adds extra cumbersome steps of getting keys and supplying them after forking the demo repository. Furthermore, after the initial fork the action runs and fails which gives impression that user did something wrong.

Proposal is to make locking optional and since the demo repo uses null resources it won't really be needed, instead we will demonstrate the digger in the most basic from allows you to collaborate on pull requests through several flows such as comment-based flow and see plans or applies directly in the pull request.

In case a user does not supply keys we proceed to perform the terraform action as usual but display a warning to the user saying that locking isn't being actioned since no keys were supplied and that they need to supply cloud account keys in order to allow locking.

We want to add options for disabling usage data collection using either environment variable or an option in digger.yml config pkg/utils/usage.go.

Environment variable

COLLECT_USAGE_DATA=true # default value

the above will set the global setting for

digger.yml config

collect_usage_data: true
projects:
- name: dev
   ....

setting it in the config will override any environment variable that was previously set. The value will be true by default.

Digger cli pipeline should not fail when an event is triggered with no actions to perform. It should not fail for example when an event exists for on push to master and there is nothing to be performed in this case as described in #103 #103 (comment)

Pulumi is an alternative to Terraform that uses regular programming languages instead of a proprietary language like HCL.

Digger should be able to automatically merge a pull request after the approval requirements have been met and it has been applied

Is it possible to allow the digger apply comment to execute only if the PR is currently mergeable? (i.e. approved when approvers are mandatory, and possibly up to date with the base branch when required, etc..)

In Atlantis: https://www.runatlantis.io/docs/command-requirements.html#mergeable

Thx @fleroux514 for suggesting this!