eddiewebb / circleci-queue Goto Github PK

View Code? Open in Web Editor NEW

74.0 3.0 75.0 559 KB

CircleCI orb to block/queue jobs to enforce max concurrency limits

License: MIT License

Shell 100.00%

circleci-orbs

circleci-queue's Introduction

CircleCI Concurrency Control Orb

CircleCI Orb to limit workflow concurrency.

Why? Some jobs (typically deployments) need to run sequentially and not parallel, but also run to completion. So CircleCI's native auto-cancel is not quite the right fit. See https://github.com/eddiewebb/circleci-challenge as an example using blue/green cloud foundry deployments.

Basic Usage

This adds concurrency limits by ensuring any jobs with this step will only continue once no previous builds are running. It supports a single argument of how many minutes to wait before aborting itself and it requires a single Environment Variable CIRCLECI_API_KEY, which must be a Personal API Token (rather than a project-specific API Permissions token). This can be created at Personal API Tokens under Users Settings. Note that the account must have write access (at least the Contributor role) on the Project you wish to enable this orb for. However, if the dont-quit parameter is enabled, view-only access (the Viewer role) is sufficient.

Screenshots / Examples

Suppose we have a workflow take takes a little while to run. Normally the build (#18) will run immediately, with no queuing.

Someone else on the team makes another commit, since the first build (#18) is still running, it will queue build #19.

It's late afternoon, everyone is pushing their commits in to ensure they are good before they leave for the day. Build #20 also queues.

Meanwhile, build #19 is now allowed to move forward since build #18 finished.

Oh No! Since 1 minute is abnormally long for things to be queued, build #20 aborts itself, letting build #19 finish uninterrupted.

Setup

See https://circleci.com/orbs/registry/orb/eddiewebb/queue#usage-examples for current examples

Note

Queueing is not supported on forked repos. If a queue from a fork happens the queue will immediately exit and the next step of the job will begin.

circleci-queue's People

Contributors

Stargazers

Watchers

circleci-queue's Issues

Orb is gone from CircleCI store

It just disappeared

[2.2.1] `until_front_of_line` reports null timestamps and continues without waiting

RELEASE CANDIDATE READY

➡️ Potential fix for this issue is ready ⬅️

For anyone willing to validate on non-critical workloads, please use eddiewebb/queue@dev:506 and report any issues on #112

Orb version

2.2.1

What happened

I have this configuration:

      - queue/until_front_of_line:
          only-on-branch: master
          only-on-workflow: orange_build
          block-workflow: true
          time: '10'
          circleci-api-key: CIRCLECI_TOKEN

But when two concurrent orange_build workflows runs the second one reports null timestamp and continues:

This Workflow Timestamp: null
Oldest Workflow Timestamp: null
Front of the line, WooHoo!, Build continuing

Downgrading to 1.8.4 solves the issues. But of course missing the new only-on-workflow parameter.

Expected behavior

Should wait for the previous workflow to finish.

BLOCKED - Change to Circle CI V2 APIs

Is your feature request related to a problem? Please describe.

We are currently seeing stale data returned from CircleCI on the v1.1 endpoints for running jobs. CircleCI has said that we should migrate off of those APIs. Unfortunatley, there isn't a clear 2.0 API paradigm to switch over to, so we need to determine what the best set of APIs to use is (and we will end up making more calls).

Describe the solution you'd like

Remove dependencies on the deprecated V1.1 CircleCI API.

Additional context
CircleCI has said they have a filed an internal incident for the V1.1 data issue, but I don't think we can rely on a fix there.

[semver:patch] PIPELINE_ID is now an envar!

Orb version

we can drop need for https://github.com/eddiewebb/circleci-queue/blob/main/src/%40orb.yml#L56 and just use CIRCLE_PIPELINE_ID

What happened

Expected behavior

Spaces in job name causes bug in queue/until_front_of_line

Orb version

3.1.4

What happened

The command queue/until_front_of_line does not work when the name of the job contains spaces. This may also affect other jobs/commands.

This is the output of a CircleCI Job when job name contains spaces. Job is running alone, no other concurrent job at the moment of grabbing this log:

Block: false
Expecting CCI Personal Access TOKEN Named: CIRCLECI_API_KEY
Using API key for user: "<-redacted->" on host https://circleci.com
staging matches queueable branch names
Max Queue Time: 600 seconds.
Only blocking execution if running previous jobs on branch: staging
Getting queue ordering
Workflow: <-redacted-> is from pipeline #169
Orb parameter block-workflow is false. Use Job level queueing.
Only blocking execution if running previous jobs matching this job: release job name with spaces
API Call for existing jobs returned no matches. This means job is alone.
This Job's Pipeline #: 169
Front of Queue (fifo) Pipeline #: 
This build (307), pipeline (169) is queued, waiting for build(null) pipeline () to complete.
Total Queue time: 0 seconds.
Only blocking execution if running previous jobs on branch: staging
Getting queue ordering
Workflow: <-redacted-> is from pipeline #169
Orb parameter block-workflow is false. Use Job level queueing.
Only blocking execution if running previous jobs matching this job: release job name with spaces
API Call for existing jobs returned no matches. This means job is alone.
This Job's Pipeline #: 169
Front of Queue (fifo) Pipeline #: 
This build (307), pipeline (169) is queued, waiting for build(null) pipeline () to complete.
Total Queue time: 11 seconds.
Only blocking execution if running previous jobs on branch: staging
Getting queue ordering
Workflow: <-redacted-> is from pipeline #169
Orb parameter block-workflow is false. Use Job level queueing.
Only blocking execution if running previous jobs matching this job: release job name with spaces
API Call for existing jobs returned no matches. This means job is alone.
This Job's Pipeline #: 169
Front of Queue (fifo) Pipeline #: 
This build (307), pipeline (169) is queued, waiting for build(null) pipeline () to complete.
Total Queue time: 22 seconds.
Only blocking execution if running previous jobs on branch: staging
Getting queue ordering
Workflow: <-redacted-> is from pipeline #169
Orb parameter block-workflow is false. Use Job level queueing.
Only blocking execution if running previous jobs matching this job: release job name with spaces
API Call for existing jobs returned no matches. This means job is alone.
This Job's Pipeline #: 169
Front of Queue (fifo) Pipeline #: 
This build (307), pipeline (169) is queued, waiting for build(null) pipeline () to complete.
Total Queue time: 33 seconds.

It seems as if the command that checks for other concurrent jobs fails because the name of the job has spaces.

This is the output of a CircleCI Job when the job name contains underscores instead of spaces:

Block: false
Expecting CCI Personal Access TOKEN Named: CIRCLECI_API_KEY
Using API key for user: "<-redacted->" on host https://circleci.com
staging matches queueable branch names
Max Queue Time: 600 seconds.
Only blocking execution if running previous jobs on branch: staging
Getting queue ordering
Workflow: <-redacted-> is from pipeline #168
Orb parameter block-workflow is false. Use Job level queueing.
Only blocking execution if running previous jobs matching this job: release_job_name_without_spaces
This Job's Pipeline #: 168
Front of Queue (fifo) Pipeline #: 168
API shows no conflicting jobs/workflows. However it is possible a previous workflow has pending jobs not yet visible in API. To avoid a race condition we will verify out place in queue.
Rerunning check 1/1
Only blocking execution if running previous jobs on branch: staging
Getting queue ordering
Workflow: <-redacted-> is from pipeline #168
Orb parameter block-workflow is false. Use Job level queueing.
Only blocking execution if running previous jobs matching this job: release_job_name_without_spaces
This Job's Pipeline #: 168
Front of Queue (fifo) Pipeline #: 168
Front of the line, WooHoo!, Build continuing

Expected behavior

Job name should have no influence over the queue/until_front_of_line or other commands/jobs.

[2.2.1] ERROR: Server returned error code: 404

Orb version

What happened

Env var CIRCLECI_API_KEY is set.
workflow queue fails with next error:

ERROR: Server returned error code: 404
{
"message" : "{"message":"Not Found","documentation_url":"https://docs.github.com/rest/repos/repos#get-a-repository\"}"
}

Expected behavior

block concurrent workflow execution on same branch

Change in CircleCI API behaviour seems to have broken .build_parameters.CIRCLE_JOB check

Thanks for a really useful Orb!

Orb version

1.6.1

What happened

main queueable
This build will block until all previous builds complete.
Max Queue Time: 60 minutes.
Only blocking execution if running previous jobs on branch: main
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY is set.
API access successful
Checking time of workflow: c64f3cde-8b5f-4b97-bf45-9065d7f38658
Workflow was created at: 2021-10-21T08:05:14Z
Checking time of workflow: 07ab1c33-cfbf-4ca4-81f4-81a96177c600
Workflow was created at: 2021-10-21T08:04:43Z
Orb parameter block-workflow is false.
Only blocking execution if running previous jobs matching this job: staging
Oldest job: null
This Workflow Timestamp: "2021-10-21T08:04:43Z"
Oldest Workflow Timestamp: null
API shows no previous jobs/workflows, but it is possible a previous workflow has pending jobs not yet visible in API.
Rerunning check 1/1
Only blocking execution if running previous jobs on branch: main
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY is set.
API access successful
Checking time of workflow: c64f3cde-8b5f-4b97-bf45-9065d7f38658
Workflow was created at: 2021-10-21T08:05:14Z
Checking time of workflow: 07ab1c33-cfbf-4ca4-81f4-81a96177c600
Workflow was created at: 2021-10-21T08:04:43Z
Orb parameter block-workflow is false.
Only blocking execution if running previous jobs matching this job: staging
Oldest job: null
This Workflow Timestamp: "2021-10-21T08:04:43Z"
Oldest Workflow Timestamp: null
Front of the line, WooHoo!, Build continuing
CircleCI received exit code 0

As you can see from the last few lines, the oldest job and oldest workflow aren't being correctly calculated.
From debugging the jq, the cause is the build_parameters field in the JSON which is an empty object. I assume it used to contain the CIRCLE_ environment variables but I'm not familiar with this API.

Perhaps this is a CircleCI API change?

It may be possible to use .workflows.job_name instead.

I can't find API documentation that covers either field (or even the /tree/ endpoint).

I discovered this issue with an older version of the orb, 1.0.110. In that version the null return causes the step to fail, which was fortunate really or else I would not have noticed. It might be good to introduce a sanity check in the latest version.

Expected behavior

It should correctly calculate the oldest job and block as needed.

Workarounds

Set block-workflow and live with the reduced concurrency.

Script should avoid using two different versions of CircleCI's API

Orb version

1.5.0

What happened

Script uses two different versions of CircleCI's API (v1.1 and v2). I am not sure if we're using two different APIs because of a lack of feature parity between them... But if there is feature parity for this orb's intents and purposes, we should use a single API!

Expected behavior

We should be using one API in the script to avoid potential API token bugs where the supplied API token is invalid for one API version but valid for another API version.

How to fix

Make sure either the personal token or project token works on both APIs, then just use that API.

[semver:minor] Reduce payload from CircleCI recent-builds API to improve performance

RELEASE CANDIDATE READY

➡️ Potential fix for this issue is ready ⬅️

For anyone willing to validate on non-critical workloads, please use eddiewebb/queue@dev:506 and report any issues on #112

Is your feature request related to a problem? Please describe.

Some users noted the Recent Builds for project API returning HTTP 400 at times (intermittent).

I noted the CircleCI V1 API is used:
https://circleci.com/docs/api/v1/index.html#recent-builds-for-a-single-project

Additionally, the shallow=true query parameter is recommended to reduce the returned payload, and possibly improve performance and reliability of the API response.

I noted from the Orb command's script that we are really only checking the following attributes from this API's response:

Attribute Evidence

.build_num

circleci-queue/src/commands/until_front_of_line.yml

Line 131 in 988b09b

  jq "[ .[] | select((.build_num | . == \"${CIRCLE_BUILD_NUM}\") or (.vcs_tag | (. != null and test(\"${tag_pattern}\"))) ) ]" /tmp/jobstatus.json >/tmp/jobstatus_tag.json 

.vcs_tag

circleci-queue/src/commands/until_front_of_line.yml

Line 131 in 988b09b

  jq "[ .[] | select((.build_num | . == \"${CIRCLE_BUILD_NUM}\") or (.vcs_tag | (. != null and test(\"${tag_pattern}\"))) ) ]" /tmp/jobstatus.json >/tmp/jobstatus_tag.json 

.workflow.workflow_id

circleci-queue/src/commands/until_front_of_line.yml

Line 140 in 988b09b

  for workflow in `jq -r ".[] | .workflows.workflow_id //empty" /tmp/augmented_jobstatus.json | uniq` 

Additionally, I can confirm the .build_num, .vcs_tag and .workflow.workflow_id attributes are returned even when shallow=true.
For example:
https://circleci.com/api/v1.1/project/github/kelvintaywl/whale-of-a-time?shallow=true

As such, I think we can leverage the shallow=true query parameter for this Orb indeed.
Hopefully, this can reduce or alleviate the randomly-intermittent HTTP 400 from CircleCI too?
If anything, it should help to improve the performance of the script 🙏

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[1.6.2] Fails with exit code 22

Orb version

1.6.2

What happened

Right after upgrading to 1.6.2, queue step failed with the following output (staging is the branch in question):

staging queueable

Exited with code exit status 22
CircleCI received exit code 22

Config:

- queue/until_front_of_line:
          time: '30'
          circleci-api-key: CIRCLE_TOKEN

Expected behavior

Queue step should not fail

Full logs

#!/bin/bash -eo pipefail

load_variables(){
  # just confirm our required variables are present
  : ${CIRCLE_BUILD_NUM:?"Required Env Variable not found!"}
  : ${CIRCLE_PROJECT_USERNAME:?"Required Env Variable not found!"}
  : ${CIRCLE_PROJECT_REPONAME:?"Required Env Variable not found!"}
  : ${CIRCLE_REPOSITORY_URL:?"Required Env Variable not found!"}
  : ${CIRCLE_JOB:?"Required Env Variable not found!"}
  # Only needed for private projects
  if [ -z "${CIRCLE_TOKEN}" ]; then
    echo "CIRCLE_TOKEN not set. Private projects will be inaccessible."
  else
    fetch "https://circleci.com/api/v2/me" "/tmp/me.cci"
    me=$(jq -e '.id' /tmp/me.cci)
    echo "Using API key for user: ${me}"
  fi
  VCS_TYPE="github"
}

fetch(){
  url=$1
  target=$2
  http_response=$(curl -f -s -X GET -H "Circle-Token:${CIRCLECI_API_KEY}" -o "${target}" -w "%{http_code}" "${url}")
  if [ $http_response != "200" ]; then
      echo "ERROR: Server returned error code: $http_response"
      cat ${target}
  else
      echo "DEBUG: API Success"
  fi
}

fetch_filtered_active_builds(){
  if [ "true" != "true" ];then
    echo "Orb parameter 'consider-branch' is false, will block previous builds on any branch." 
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
  elif [ -n "${CIRCLE_TAG:x}" ] && [ "" != "" ]; then
    # I'm not sure why this is here, seems identical to above?
    echo "CIRCLE_TAG and orb parameter tag-pattern is set, fetch active builds"
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
  else
    : ${CIRCLE_BRANCH:?"Required Env Variable not found!"}
    echo "Only blocking execution if running previous jobs on branch: ${CIRCLE_BRANCH}"
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/tree/${CIRCLE_BRANCH}?filter=running"
  fi

  if [ ! -z $TESTING_MOCK_RESPONSE ] && [ -f $TESTING_MOCK_RESPONSE ];then
    echo "Using test mock response"
    cat $TESTING_MOCK_RESPONSE > /tmp/jobstatus.json
  else
    echo "Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLE_TOKEN  is set."
    fetch "$jobs_api_url_template" "/tmp/jobstatus.json"
    if [ -n "${CIRCLE_TAG:x}" ] && [ "" != "" ]; then
      jq '[ .[] | select(.vcs_tag | . != null) | select(.vcs_tag | test("") ) ]' /tmp/jobstatus.json > /tmp/jobstatus_tag.json
      mv /tmp/jobstatus_tag.json /tmp/jobstatus.json
    fi
    echo "API access successful"
  fi
}

fetch_active_workflows(){
  cp /tmp/jobstatus.json /tmp/augmented_jobstatus.json
  for workflow in `jq -r ".[] | .workflows.workflow_id //empty" /tmp/augmented_jobstatus.json | uniq`
  do
    echo "Checking time of workflow: ${workflow}"
    workflow_file=/tmp/workflow-${workflow}.json
    if [ ! -z $TESTING_MOCK_WORKFLOW_RESPONSES ] && [ -f $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json ]; then
      echo "Using test mock workflow response"
      cat $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json > ${workflow_file}
    else
      fetch "https://circleci.com/api/v2/workflow/${workflow}" "${workflow_file}"
    fi
    created_at=`jq -r '.created_at' ${workflow_file}`
    echo "Workflow was created at: ${created_at}"
    cat /tmp/augmented_jobstatus.json | jq --arg created_at "${created_at}" --arg workflow "${workflow}" '(.[] | select(.workflows.workflow_id == $workflow) | .workflows) |= . + {created_at:$created_at}' > /tmp/augmented_jobstatus-${workflow}.json
    #DEBUG echo "new augmented_jobstatus:"
    #DEBUG cat /tmp/augmented_jobstatus-${workflow}.json
    mv /tmp/augmented_jobstatus-${workflow}.json /tmp/augmented_jobstatus.json
  done
}

update_comparables(){     
  fetch_filtered_active_builds

  fetch_active_workflows

  load_current_workflow_values

  # falsey parameters are empty strings, so always compare against 'true' 
  if [ "false" = "true" ] ;then
    echo "Orb parameter block-workflow is true."
    echo "This job will block until no previous workflows have *any* jobs running."
    oldest_running_build_num=`jq 'sort_by(.workflows.created_at)| .[0].build_num' /tmp/augmented_jobstatus.json`
    oldest_commit_time=`jq 'sort_by(.workflows.created_at)| .[0].workflows.created_at' /tmp/augmented_jobstatus.json`
  else
    echo "Orb parameter block-workflow is false."
    echo "Only blocking execution if running previous jobs matching this job: ${CIRCLE_JOB}"
    oldest_running_build_num=`jq ". | map(select(.workflows.job_name==\"${CIRCLE_JOB}\")) | sort_by(.workflows.created_at)|  .[0].build_num" /tmp/augmented_jobstatus.json`
    oldest_commit_time=`jq ". | map(select(.workflows.job_name==\"${CIRCLE_JOB}\")) | sort_by(.workflows.created_at)|  .[0].workflows.created_at" /tmp/augmented_jobstatus.json`
  fi
  if [ -z "$oldest_commit_time" ]; then
    echo "API Error - unable to load previous job timings. Report to developer."
    exit 1
  fi
  echo "Oldest job: $oldest_running_build_num"
  if [ -z $oldest_commit_time ];then
    echo "API Call for existing jobs failed, failing this build.  Please check API token"
    echo "All running jobs:"
    cat /tmp/jobstatus.json || exit 0
    echo "All running jobs with created_at:"
    cat /tmp/augmented_jobstatus.json || exit 0
    echo "All worfklow details."
    cat /tmp/workflow-*.json
    exit 1
  fi
}

load_current_workflow_values(){
   my_commit_time=`jq '.[] | select( .build_num == '"${CIRCLE_BUILD_NUM}"').workflows.created_at' /tmp/augmented_jobstatus.json`
}

cancel_current_build(){
  echo "Cancelleing build ${CIRCLE_BUILD_NUM}"
  cancel_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BUILD_NUM}/cancel?circle-token=${CIRCLE_TOKEN}"
  curl -s -X POST $cancel_api_url_template > /dev/null
}



#
# We can skip a few use cases without calling API
#
if [ ! -z "$CIRCLE_PR_REPONAME" ]; then
  echo "Queueing on forks is not supported. Skipping queue..."
  # It's important that we not fail here because it could cause issues on the main repo's branch
  exit 0
fi
if [ "*" = "*" ] || [ "*" = "${CIRCLE_BRANCH}" ]; then
  echo "${CIRCLE_BRANCH} queueable"
else
  echo "Queueing only happens on * branch, skipping queue"
  exit 0
fi

#
# Set values that wont change while we wait
# 
load_variables
max_time=30
echo "This build will block until all previous builds complete."
echo "Max Queue Time: ${max_time} minutes."
wait_time=0
loop_time=11
max_time_seconds=$((max_time * 60))

#
# Queue Loop
#
confidence=0
while true; do
  update_comparables
  echo "This Workflow Timestamp: $my_commit_time"
  echo "Oldest Workflow Timestamp: $oldest_commit_time"
  if [[ ! -z "$my_commit_time" ]] && [[ "$oldest_commit_time" > "$my_commit_time" || "$oldest_commit_time" = "$my_commit_time" ]] ; then
    # API returns Y-M-D HH:MM (with 24 hour clock) so alphabetical string compare is accurate to timestamp compare as well
    # recent-jobs API does not include pending, so it is posisble we queried in between a workfow transition, and we;re NOT really front of line.
    if [ $confidence -lt 1 ];then
      # To grow confidence, we check again with a delay.
      confidence=$((confidence+1))
      echo "API shows no previous jobs/workflows, but it is possible a previous workflow has pending jobs not yet visible in API."
      echo "Rerunning check ${confidence}/1"
    else
      echo "Front of the line, WooHoo!, Build continuing"
      break
    fi
  else
    # If we fail, reset confidence
    confidence=0
    echo "This build (${CIRCLE_BUILD_NUM}) is queued, waiting for build number (${oldest_running_build_num}) to complete."
    echo "Total Queue time: ${wait_time} seconds."
  fi

  if [ $wait_time -ge $max_time_seconds ]; then
    echo "Max wait time exceeded, considering response."
    if [ "false" == "true" ];then
      echo "Orb parameter dont-quit is set to true, letting this job proceed!"
      exit 0
    else
      cancel_current_build
      sleep 10 # wait for API to cancel this job, rather than showing as failure
      exit 1 # but just in case, fail job
    fi
  fi

  sleep $loop_time
  wait_time=$(( loop_time + wait_time ))
done

staging queueable

Exited with code exit status 22
CircleCI received exit code 22

[3.0.0] Document breaking changes

Orb version

3.0.0+

What happened

Updating to 3.1.1, there are number of parameter changes that break previous usage and require delving into the code (4c639c8) to determine what they are.

Expected behavior

Update the changelog or other documentation with the specifics of the breaking change - e.g. my-pipeline is now required and various parameters have changed their names (e.g. time is now max-wait-time_. Even if the changelog mentioned these changes in general linked someone off to the specific commit, then that would make updating easier.

support for serializing workflows by name

Is your feature request related to a problem? Please describe.
I'd like to serialize a workflow of a specific name without other workflows preventing it from running. Currently we wait for all workflows to complete before starting a workflow.

Describe the solution you'd like
Can we check only for existing workflows by workflow name (similar to what we do for serializing jobs)?

Describe alternatives you've considered
None

Additional context

[semver:patch] tag-pattern condition syntax error

Orb version

1.6.0

What happened

using the tag-pattern param, getting an from erroneous syntax

environment: line 23: [: missing `]'

caused by the line

[ -n "${CIRCLE_TAG:x}" && "<<parameters.tag-pattern>>" != "" ]

Expected behavior

the condition check shouldn't error

I'm going to open a PR to fix this shortly and will update this description with a link to the PR.

Block running single workflow parallel for any branch

Is your feature request related to a problem? Please describe.

We are doing Salesforce development, and we have different workflows used by different teams, we want different workflows can be running independently, but within single workflow, we want all commits at any branches running serially since each workflow has only one sandbox environment to run.
Right now seem we can only block workflows on the same branches or on all branches, neither meets our requirement.

[semver:patch] Boolean parameters not working

Orb version

3.0.0

What happened

Trying to set any of the boolean parameters does not work.
Affected parameters (All boolean parameters):

block-workflow
dont-quit
force-cancel-previous
include-debug
this-branch-only

Orb parameter dont-quit is set to true, letting this job proceed! is always shown even if dont-quit is set to false explicitly

Seems like CircleCI's booleans are passed as 0/1, so when trying to cast the boolean parameters as string, if [ "${DONT_QUIT}" != "false" ];then will always be true
Ex. when dont-quit: true, I see DONT_QUIT=1 in the job's logs, and vice versa for dont-quit: false, I'm seeing DONT_QUIT=0

Default block-workflow is true but Block: 0 is shown in the logs.

As a result, when a build exceeds the max wait time, it will always fail as force-cancel-previous is not yet implemented, instead of cancelling the build.

Expected behavior

Boolean parameters are passed and working properly. By default, dont-quit should be false and builds should be cancelling instead of failing if the max-wait-time is exceeded.

No fallback if job committer_date is null/empty

RELEASE CANDIDATE READY

➡️ Potential fix for this issue is ready ⬅️

For anyone willing to validate on non-critical workloads, please use eddiewebb/queue@dev:506 and report any issues on #112

Orb version

2.2.1

What happened

I am using a bitbucket repo for my project, and it looks like the committer_date field, which is used to determine which jobs are oldest is always empty. It would be good if there was an option/fallback to use build number in this case - especially since I also have uses cases to queue jobs that are manually triggered with parameters, i.e. without a new commit push

Expected behavior

Should still be able to queue jobs when committer_date is empty

Make this a first-party rather than third-party orb?

Is your feature request related to a problem? Please describe.

My organization is security-sensitive and the one-liner at line 48 for the actual execution is really hard to audit.

I want to include this in our pipeline to mutex AWS deployments, but will need admin approval. Not sure if we can justify it.

Describe the solution you'd like

I'd like circleci-queue to be part of the core CircleCI product so I don't have to ask for permission

Describe alternatives you've considered

Looking for other orbs
Copy-Pasta the contents of your orb into our yaml (with attribution somehow? TBD)

Additional context

Does `queue/until_front_of_line` guarantee a correct deployment order?

Hi!

Thank you for this project!

However, I have a doubt that prevents me from using queue/until_front_of_line for the use case that you describe in the docs:

Used to ensure that a only single job (deploy) is not run concurrently.
By default will only queue if the same job from previous workflows is running on the same branch.
This allows safe jobs like build/test to overlap, minimizing overall queue times

Imagine that you have a repo with a single workflow that runs 2 jobs sequentially:

test
deploy

The "test" job can take various amounts of time, let's say from 10 to 60 seconds.

The "deploy" job one has queue/until_front_of_line as the first step.

Now let's say we push 2 commits to the master branch of this repo, commit A first and then commit B, 10 seconds later.

This time the "test" job took 30 seconds for commit A but just 5 seconds for commit B.

My question is: which "deploy" job will run first, for commit A or commit B?

[3.1.1] Bitbucket support seems currently broken

Orb version

3.1.1

What happened

When using bitbucket, the orb isn't working. The regular expression is not identifying in the bash script something like this:
[email protected]:deedee/deeedee.git as a bitbucket address

Current code:

 VCS_TYPE="github"
    if [[ "$CIRCLE_REPOSITORY_URL" =~ "*bitbucket.org*" ]]; then
        VCS_TYPE="bitbucket"
    fi

But instead, it's assuming the default github VCS so consequently failing to reach to the project URL correctly

Should change to something like the following, so it could accept any:

Expected behavior


VCS_TYPE="github"
if [[ "$CIRCLE_REPOSITORY_URL" =~ .*bitbucket\.org.* ]]; then
    VCS_TYPE="bitbucket"
fi

Orb not working properly for BitBucket

I am trying to use this as the first step in one of my jobs.

queue/until_front_of_line:
  vcs-type: bitbucket

I triggered 2 workflows that utilized the same job.
I expected it not to continue but it still did with the following output.

queues queueable
This build will block until all previous builds complete.
Max Queue Time: 10 minutes.
Only blocking execution if running previous jobs on branch: queues
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY is set.
API access successful
Orb parameter block-worflow is false.
Only blocking execution if running previous jobs matching this job: build_release-1
Oldest job: 
This Workflow Timestamp: ""
Oldest Workflow Timestamp: ""
Front of the line, WooHoo!, Build continuing

The This Workflow Timestamp: "" line looked suspicious so I decided to dig into the code.
I found that when calling the API, I was seeing this for all jobs:

"all_commit_details" : [ {
    "committer_date" : "",
    "body" : "",
    "branch" : "queues",
    "author_date" : "2019-02-05T19:25:55+00:00",
    ...

Seeing that this was likely the cause, I went and copied all the code and swapped out committer_date with author_date and also tried queued_at. Both allowed this step to work as intended.
Is it possible to get a new version of this orb that utilizes author_date or queued_at in the case where committer_date is not populated?

[semver:minor] Support Server Customer with an option to specify the hostname

Is your feature request related to a problem? Please describe.
Server Customer can not use this orb.

Describe the solution you'd like
It would be great if there would be an option to set a custom host so that server customer can a also use this orb.

Describe alternatives you've considered
Fork this repository and change the hostname from circleci.com to your own custom host, then deploy the orb within the server.

To clarify what kind of API key is required

The documentation says:

This orb requires the project to have an API key in order to query build states.
It requires a single environment variable CIRCLECI_API_KEY which can be created in account settings - https://circleci.com/account/api.

In Circle CI there are 2 kinds of API Tokens: Personal API tokens and project-specific API tokens.
I've tried project-specific API tokens, doesn't work.

As I understand the orb requires a Personal API token, at least it works for me:).

Describe the solution you'd like
To update the doc and explicitly mention that the orb requires a Personal API token.

What the heck is going on with versions?

After the 1.8.4 release a bug was introduced into the publishing process that resulted in superfluous publishing of the orb. All versions published were reviewed and approved, but the process sometimes published them in duplicate or triplicate!

Releases prior to 1.8.4 are stated accurately in log, and releases 2.2.1 forward are valid.

I apologize that 2.xx was introduced this way, it was intended to be staged, not published.

The underlying bug has been fixed, and if you are a Bitbucket user who has issue with 2.2.1 please notify me.

circleci-queue doesn't support workflow isolation

As per our discussion, right now if multiple workflows are queued at the same time, it's possible that the circleci-queue job doesn't block the workflow until other workflows are stopped. This is because within a workflow, job ids are not assigned at start time, but at runtime.
Visual example if you queue two workflows right after the other, where A=the block_workflow job, the job ids will look like:
A'(2) B' C'
A(1) B C

Since A(1) is at the front of the queue, it will finish successfully instantly, and then begin executing B, which now gets the id B(3). Next, since A'(2) is now at the front of the queue, it will also finish successfully, and queue up B' which now gets the id B'(4). At this point you will have two B jobs concurrently running.

[semver:minor] Detect on-hold pipelines

Is your feature request related to a problem? Please describe.
I've noticed when using the queue, Circle CI doesn't list a pipeline as running when using the running filter if the pipeline is using the hold job (ex: https://circleci.com/docs/workflows#holding-a-workflow-for-a-manual-approval), so subsequent pipelines may kick off while in that waiting period. (I think this might be a circle limitation but wanted to see if you had any ideas)

Describe the solution you'd like
The queue would be able to block until all running or on_hold pipelines were finished.

Describe alternatives you've considered
I'm trying to think of workarounds, and that might be ultimately what is needed, but wanted to create an issue to see if this was a known problem or if other people have found workarounds.

Additional context

[2.2.1/1.12.0] queue step exits with status code 35

Orb version

2.2.1/1.12.0

What happened

https://circleci.com/gh/determined-ai/determined/1329921
https://circleci.com/gh/determined-ai/determined/1274126

After a DEBUG: Making API Call to ... line, the queue step exited with status code 35.

I think the status code is from curl, for which 35 represents SSL connect error. The SSL handshaking failed.. This failure only happens intermittently; I guess there could be some transient flakiness in CircleCI's network or servers.

Expected behavior

The queue step should not fail. Perhaps curl runs should all be done with retries to work around these intermittent issues, either with a shell loop or using curl's own retry mechanisms (though, last I checked, the version available was too old to support --retry-all-errors, which I think is what you would want).

[2.2.1] gitlab - does not support CIRCLE_PROJECT_REPONAME

Orb version

2.2.1

What happened

circleci does not support CIRCLE_PROJECT_REPONAME for gitlab, there is a workaround to define it in your circleci script as

environment:
      CIRCLE_PROJECT_REPONAME: << pipeline.trigger_parameters.gitlab.repo_name >>

but this does not work with orbs, could you add a new setting to provide repo name from outside

Expected behavior

the orb should fetch repo name from a parameter in case of gitlab

[semver:patch] Setting block-workflow as false in until_front_of_line doesn't works as expected

RELEASE CANDIDATE READY

➡️ Potential fix for this issue is ready ⬅️

For anyone willing to validate on non-critical workloads, please use eddiewebb/queue@dev:506 and report any issues on #112

Orb version

eddiewebb/[email protected]

What happened

I'm using the until_front_of_line inside the job as the command and passing the block-workflow as false (also tried removing it as it is by default false) after setting this when I created multiple commits in a short succession . Both the commits CI's runs together.
You can check the Demo project where I have tested it here

Expected behavior

It should block the second commits CI until the first commit CI finishes.
After checking the code here I found that is says echo "Only blocking execution if running previous jobs matching this job: ${JOB_NAME}"
It seems it should block the second commit CI because both are running same Job.. but it didn't work as expected.

I have also shared the first and second commit where I have tested it
first : slayer321/Demo_project@82382fe
second: slayer321/Demo_project@3bf7e99

Feature Request: Parameterize CIRCLECI_API_KEY env var

Minor Feature request, would you be interested in a PR that makes CIRCLECI_API_KEY an env-var parameter (with CIRCLECI_API_KEY as default) so we could pass in a different one? Our team's contexts use a different API key env var so it would be nice to just pass that in instead of having it to change in all our scripts!

Thanks for the good orb, we really need this ability in circleci by default!

The orb seems to be broken

The orb is always using the MOCK response

Max Queue Time: 10 minutes.
Using test mock response
Only blocking execution if running previous jobs matching: deploy-dev
Oldest job:

Here is my config:

      - queue/until_front_of_line:
          consider-job: true
          consider-branch: true
          dont-quit: true

And I do not have the TESTING_MOCK_RESPONSE variable set

Example in readme shows time as a number but seems to be required to be string

The example in the readme shows the time parameter as a YAML number, but it's actually required to be a string. I got an error with an unquoted number in my CI config:

# Error calling command: 'queue/until_front_of_line'
# Type error for argument time: expected type: string, actual value: \"60\" (type integer)

It worked fine with time: "60"

Happy to send a PR if it's the docs that need updating.

[1.7.0] Orb not working properly when pipeline uses `path-filtering` orb

Orb version

orbs:
  queue: eddiewebb/[email protected]

What happened

We are using the path-filtering orb, which under the neath utilizes the continuation
orb, and dispatches workflows through CircleCI API based on paths of changed files.

Ultimately, for those not familiar with it, in practice, that works as follows:

A commit triggers a new CircleCI pipeline which starts with an initial workflow (ie generate-config).
The generate-config workflow, uses the path-filtering orb, and is responsible to dispatch the creation of some follow up
workflows based on files changed, with the continuation orb.
The dispatched workflows take place, and run until completion/failure under the same pipeline context.

A nice example can be found here as well.

It seems that for a workflow to be blocked, comparison happens based on this expression:

if [[ ! -z "$my_commit_time" ]] && [[ "$oldest_commit_time" > "$my_commit_time" || "$oldest_commit_time" = "$my_commit_time" ]] ;

where,

oldest_commit_time variable points to the creation time of the oldest running workflow - and not really to the actual git commit
my_commit_time points to this workflow creation time

Normally, this works fine with workflows that are triggered upon a pipeline is being created.
However, that does not work well with pipelines that rely on the path-filtering and continuation orb, since in that case,
workflows are dispatched afterwards, and their creation time does not really reflect the "received at" time for the pipeline trigger.

As a result, while a path-filtering is still in progress and not completed yet - new workflows not dispatched yet, another commit (or more) can trigger a new pipeline in between, which creates a new generate-config workflow before the creation of the dispatched workflows of the previous pipeline.
So, despite the fact that this workflow belongs to newer pipeline, it wins on comparison with workflows that belong to older pipelines/builds but have been created afterwards and is not being blocked.

It seems that this logic introduced long time ago, where orb switched to use the workflow created_at field for the comparison, instead of the committer_date field.
As explained above, in cases where workflows are dispatched afterwards, this field does not really correlate with the time that a pipeline got triggered.

To help you understand better the problem, imagine the following scenario, which summaries the race condition mentioned above:

Our goal is to use the `circleci-queue` orb, and make sure that the `generate-config` workflow is properly 
queued/blocked until there are no workflows in progress, from previous pipelines. 
End goal is to have CircleCI/path-filtering orb dispatching any set of follow up workflows based on their 
commit order, and having only one commit with these workflows running at any time 
(a commit dispatches workflows based on path filtering only when workflows from the previous commit have been completed)

For simplicity reasons, let's imagine that there are no other pipelines/workflows in progress at that time, and events 
are ordered below based on their time.

* Commit `abcd` triggers a CircleCI pipeline with a `generate-config` workflow at `2023-03-21T00:00:00.000Z`. 
There are no other running workflows, so workflow is not being blocked from orb. 
At that time, both `oldest_commit_time` and `my_commit_time` point to `2023-03-21T00:00:00.000Z`.

* Commit `wxyz` triggers a CircleCI pipeline with a `generate-config` workflow at `2023-03-21T00:01:00.000Z`. 
At that time, `oldest_commit_time` points to `2023-03-21T00:00:00.000Z` and `my_commit_time` points to `2023-03-21T00:01:00.000Z`. 
This workflow will be blocked up until the above workflow, and any other workflow with older creation time is finished.

* The `generate-config` workflow, triggered by `abcd` commit, has completed and a new set of workflows are created at `2023-03-21T00:02:00.000Z`. 
At that time, `oldest_commit_time` points to `2023-03-21T00:01:00.000Z` and `my_commit_time` points to `2023-03-21T00:02:00.000Z`.

* The `generate-config` workflow, triggered by `wxyz` commit, is being unblocked, completes and dispatches another new set of workflows at `2023-03-21T00:04:00.000Z`. 
Normally, we would expect that the this workflow would be blocked up until the dispatched workflows from `abcd` commit are completed. 
However, this is not the case, since at that time, both `oldest_commit_time` and `my_commit_time` point to `2023-03-21T00:01:00.000Z`.

Expected behavior

The queue orb is able to block workflows, based on their commit order and does not rely on the workflow creation time, which can lead to race conditions as described above.

I am curious if we can fallback to another metadata field for the oldest_commit_time variable.

Also, currently, the name of this variable does not really reflect its real value, since it does not hold any information about the commit time, or the time around when a pipeline got triggered, and instead points to the workflow creation time.

I see that the motivation for this implementation was this issue: #22,
where some issues reported for Orb not working properly on BitBucket when using the committer_date attribute, but maybe makes sense to re-investigate whether another metadata field exists that can be used and satisfy both cases.

Thoughts?

"Ghost" jobs (with SSH session open) keeps queue always blocked

Orb version

What happened

Trying out this orb for the first time, for some reason I see that, within my builds, the number 380 was skipped (probs a CircleCI bug? ), and this number is detected by the queue orb as if it was still running, rendering the next deployments useless since they THINK there's a build still running.

All branches and everyone's pipelines are selected, so there are no filters that could hide that missing 380

Expected behavior

380 doesn't exist or it is not running, therefore the queue shouldn't be blocked!

I wonder if this is an uncaught error that could be solved from this orb's end

1.4 does not work with project tokens

Orb version

Upgrade from 1.1.2 to 1.4.0

What happened

Wanted the confidence param, tried the upgrade, got back:

master queueable
This build will block until all previous builds complete.
Max Queue Time: 1000000 minutes.
Only blocking execution if running previous jobs on branch: master
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY is set.
API access successful
Checking time of workflow: 64c04663-fa40-4a8e-a4ee-75da3bc7bc46
Exited with code exit status 22
CircleCI received exit code 22

Expected behavior

it works! queues!

Forked testing fails for lack of token

I think we can fix with an approval->context flow that Ill test out.

Cannot override VCS_TYPE: Cannot index object with number

The script fails for me, maybe circleci changed their api?

This build will block until all previous builds complete.
Max Queue Time: 5 minutes.
No Job variable, blocking on any runnin jobs for this project.
jq: error (at <stdin>:2): Cannot index object with number
API Call for existing jobs failed, failing this build.  Please check API token
Exited with code 1

Feature request: only-on-branch accept multiple values

On my project, I have 3 production branches. On each of it if tests pass successfully - deployment process starts, which I need to queue to prevent multiple parallel deployments processes.

Mu current circleci config looks like

workflows:
  version: 2.1
  test_and_deploy:
    jobs:
      - queue/block_workflow:
          block-workflow: true
          consider-branch: true
          only-on-branch: branch-1
          time: '10'
      - test
      - deploy:
          requires:
            - test
          filters:
            branches:
              only:
                - branch-1
                - branch-2
                - branch-3

what I propose:

workflows:
  version: 2.1
  test_and_deploy:
    jobs:
      - queue/block_workflow:
          block-workflow: true
          consider-branch: true
          only-on-branch: 
            - branch-1
            - branch-2
            - branch-3
          time: '10'
      - test
      - deploy:
          requires:
            - test
          filters:
            branches:
              only:
                - branch-1
                - branch-2
                - branch-3

until_front_of_line doesn't block on Alpine images

Perhaps I'm doing something wrong, but I can't get until_front_of_line to work. The job config looks like this:

deploy-step:
    docker:
      - image: alpine:3.9
    steps:
      - checkout
      - run:
          name: Install curl and jq
          command: |
            apk add --no-cache curl jq
      - queue/until_front_of_line:
          time: '10'
          only-on-branch: develop
          block-workflow: true
      - run:
          command: # some deploy command

Then I push two commits to develop with ~10 seconds between them. The log from the first job:

develop queueable
This build will block until all previous builds complete.
Max Queue Time: 10 minutes.
Only blocking execution if running previous jobs on branch: develop
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY is set.
API access successful
Orb parameter block-workflow is true.
This job will block until no previous workflows have *any* jobs running.
Oldest job: 
This Workflow Timestamp: "2019-09-04T12:33:17+01:00"
Oldest Workflow Timestamp: "2019-09-04T12:33:17+01:00"
Front of the line, WooHoo!, Build continuing

and the second one:

develop queueable
This build will block until all previous builds complete.
Max Queue Time: 10 minutes.
Only blocking execution if running previous jobs on branch: develop
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY is set.
API access successful
Orb parameter block-workflow is true.
This job will block until no previous workflows have *any* jobs running.
Oldest job: 
This Workflow Timestamp: "2019-09-04T12:33:27+01:00"
Oldest Workflow Timestamp: "2019-09-04T12:33:17+01:00"
Front of the line, WooHoo!, Build continuing

As you can see, oldest workflow timestamp is earlier than this workflow timestamp, but the build continues. Any ideas?

Orb Uses Unauthenticated Dockerhub Pull

Orb version

1.5.0

What happened

Starting Nov 1st dockerhub is going to rate limit unauthenticated docker image pulls. CircleCI emits a warning when using un-auth'ed pulls:

Warning: No authentication provided, this pull may be subject to Docker Hub download rate limits.

Expected behavior

Probably add the ability to use authenticated pulls in the orb executor. or the ability to override the executer with a parameter.

Git tags

Would love for this to work with git tags. Git tag-triggered jobs don't have $CIRCLE_BRANCH set :/

[semver:minor] Expose an env variable that fails the workflow instead of cancel

Is your feature request related to a problem? Please describe.
I want to send a notification to slack whenever the orb cancels a workflow. However, currently it is not possible that the slack notifier works on cancel. There seems to be a difference between cancel and fail

Describe the solution you'd like
With the use of a parameter in the command allow to fail the workflow instead of cancel.

Describe alternatives you've considered
Get circleci to expose a on_cancel event.

Feature: force through by cancelling previous

I could see where this would be ideal in hung situations, but could also . have cascading effect if ill timed.

Isolate by job in workflow.

Job block-workflow doesn't use block-workflow parameter

Parameter block-workflow is not used at all. It works correctly because command until_front_of_line checks legacy parameter consider-job

Spotted when was trying to figure out error which was probably #26

[3.1.2] Error "Required MY_BRANCH not found!" when running on a pipeline triggered by a tag

Orb version

3.1.2

What happened

Running on a pipeline triggered by a tag (we use tags to trigger our release/deployment pipelines), so CIRCLE_BRANCH is empty, with the following params:

steps:
  - queue/until_front_of_line:
      tag-pattern: /^@my-org/my-app@\d+\.\d+\.\d+.*/
      my-pipeline: <<pipeline.number>>

Validation in load_variables is throwing the following:

Block: false
environment: line 29: MY_BRANCH: Required MY_BRANCH not found! This is likely a bug in orb, please report.

Exited with code exit status 127
CircleCI received exit code 127

Expected behavior

I guess I expected it to work? 😛
Seriously though, when CIRCLE_TAG is present, CIRCLE_BRANCH will always be empty, so maybe don't make the branch required in that case?

[1.7.0] CircleCI API 404 response kills the job

Orb version

1.7.0

What happened

The job exited unexpectedly upon receiving a 404 from the CircleCI API when making a call to get a workflow.

Full job logs

#!/bin/bash -eo pipefail
tag_pattern=""

# If a pattern is wrapped with slashes, remove them.
if [[ "$tag_pattern" == /*/ ]]; then
  tag_pattern=${tag_pattern:1:-1}
fi

fetch(){
  echo "DEBUG: Making API Call to ${1}"
  url=$1
  target=$2
  http_response=$(curl -f -s -X GET -H "Circle-Token:${CIRCLECI_API_KEY}" -o "${target}" -w "%{http_code}" "${url}")
  if [ $http_response != "200" ]; then
      echo "ERROR: Server returned error code: $http_response"
      cat ${target}
      exit 1
  else
      echo "DEBUG: API Success"
  fi
}

load_variables(){
  # just confirm our required variables are present
  : ${CIRCLE_BUILD_NUM:?"Required Env Variable not found!"}
  : ${CIRCLE_PROJECT_USERNAME:?"Required Env Variable not found!"}
  : ${CIRCLE_PROJECT_REPONAME:?"Required Env Variable not found!"}
  : ${CIRCLE_REPOSITORY_URL:?"Required Env Variable not found!"}
  : ${CIRCLE_JOB:?"Required Env Variable not found!"}
  # Only needed for private projects
  if [ -z "${CIRCLECI_API_KEY}" ]; then
    echo "CIRCLECI_API_KEY not set. Private projects will be inaccessible."
  else
    fetch "https://circleci.com/api/v2/me" "/tmp/me.cci"
    me=$(jq -e '.id' /tmp/me.cci)
    echo "Using API key for user: ${me}"
  fi
  VCS_TYPE="github"
}


fetch_filtered_active_builds(){
  if [ "false" != "true" ];then
    echo "Orb parameter 'consider-branch' is false, will block previous builds on any branch." 
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
  elif [ -n "${CIRCLE_TAG:x}" ] && [ "$tag_pattern" != "" ]; then
    # I'm not sure why this is here, seems identical to above?
    echo "CIRCLE_TAG and orb parameter tag-pattern is set, fetch active builds"
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
  else
    : ${CIRCLE_BRANCH:?"Required Env Variable not found!"}
    echo "Only blocking execution if running previous jobs on branch: ${CIRCLE_BRANCH}"
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/tree/${CIRCLE_BRANCH}?filter=running"
  fi

  if [ ! -z $TESTING_MOCK_RESPONSE ] && [ -f $TESTING_MOCK_RESPONSE ];then
    echo "Using test mock response"
    cat $TESTING_MOCK_RESPONSE > /tmp/jobstatus.json
  else
    echo "Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY  is set."
    fetch "$jobs_api_url_template" "/tmp/jobstatus.json"
    if [ -n "${CIRCLE_TAG:x}" ] && [ "$tag_pattern" != "" ]; then
      jq "[ .[] | select((.build_num | . == \"${CIRCLE_BUILD_NUM}\") or (.vcs_tag | (. != null and test(\"${tag_pattern}\"))) ) ]" /tmp/jobstatus.json >/tmp/jobstatus_tag.json
      mv /tmp/jobstatus_tag.json /tmp/jobstatus.json
    fi
    echo "API access successful"
  fi
}

fetch_active_workflows(){
  cp /tmp/jobstatus.json /tmp/augmented_jobstatus.json
  for workflow in `jq -r ".[] | .workflows.workflow_id //empty" /tmp/augmented_jobstatus.json | uniq`
  do
    echo "Checking time of workflow: ${workflow}"
    workflow_file=/tmp/workflow-${workflow}.json
    if [ ! -z $TESTING_MOCK_WORKFLOW_RESPONSES ] && [ -f $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json ]; then
      echo "Using test mock workflow response"
      cat $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json > ${workflow_file}
    else
      fetch "https://circleci.com/api/v2/workflow/${workflow}" "${workflow_file}"
    fi
    created_at=`jq -r '.created_at' ${workflow_file}`
    echo "Workflow was created at: ${created_at}"
    cat /tmp/augmented_jobstatus.json | jq --arg created_at "${created_at}" --arg workflow "${workflow}" '(.[] | select(.workflows.workflow_id == $workflow) | .workflows) |= . + {created_at:$created_at}' > /tmp/augmented_jobstatus-${workflow}.json
    #DEBUG echo "new augmented_jobstatus:"
    #DEBUG cat /tmp/augmented_jobstatus-${workflow}.json
    mv /tmp/augmented_jobstatus-${workflow}.json /tmp/augmented_jobstatus.json
  done
}

update_comparables(){     
  fetch_filtered_active_builds

  fetch_active_workflows

  load_current_workflow_values
  
  JOB_NAME="${CIRCLE_JOB}"
  if [ "^validate-controllers$" ] ;then
    JOB_NAME="^validate-controllers$"
  fi

  # falsey parameters are empty strings, so always compare against 'true' 
  if [ "false" = "true" ] ;then
    echo "Orb parameter block-workflow is true."
    echo "This job will block until no previous workflows have *any* jobs running."
    oldest_running_build_num=`jq 'sort_by(.workflows.created_at)| .[0].build_num' /tmp/augmented_jobstatus.json`
    oldest_commit_time=`jq 'sort_by(.workflows.created_at)| .[0].workflows.created_at' /tmp/augmented_jobstatus.json`
  else
    echo "Orb parameter block-workflow is false."
    echo "Only blocking execution if running previous jobs matching this job: ${JOB_NAME}"
    oldest_running_build_num=`jq ". | map(select(.workflows.job_name | test(\"${JOB_NAME}\";\"sx\"))) | sort_by(.workflows.created_at)|  .[0].build_num" /tmp/augmented_jobstatus.json`
    oldest_commit_time=`jq ". | map(select(.workflows.job_name | test(\"${JOB_NAME}\";\"sx\"))) | sort_by(.workflows.created_at)|  .[0].workflows.created_at" /tmp/augmented_jobstatus.json`
  fi
  if [ -z "$oldest_commit_time" ]; then
    echo "API Error - unable to load previous job timings. Report to developer."
    exit 1
  fi
  echo "Oldest job: $oldest_running_build_num"
  if [ -z $oldest_commit_time ];then
    echo "API Call for existing jobs failed, failing this build.  Please check API token"
    echo "All running jobs:"
    cat /tmp/jobstatus.json || exit 0
    echo "All running jobs with created_at:"
    cat /tmp/augmented_jobstatus.json || exit 0
    echo "All worfklow details."
    cat /tmp/workflow-*.json
    exit 1
  fi
}

load_current_workflow_values(){
   my_commit_time=`jq '.[] | select( .build_num == '"${CIRCLE_BUILD_NUM}"').workflows.created_at' /tmp/augmented_jobstatus.json`
}

cancel_current_build(){
  echo "Cancelleing build ${CIRCLE_BUILD_NUM}"
  cancel_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BUILD_NUM}/cancel?circle-token=${CIRCLECI_API_KEY}"
  curl -s -X POST $cancel_api_url_template > /dev/null
}



#
# We can skip a few use cases without calling API
#
if [ ! -z "$CIRCLE_PR_REPONAME" ]; then
  echo "Queueing on forks is not supported. Skipping queue..."
  # It's important that we not fail here because it could cause issues on the main repo's branch
  exit 0
fi
if [ "*" = "*" ] || [ "*" = "${CIRCLE_BRANCH}" ]; then
  echo "${CIRCLE_BRANCH} queueable"
else
  echo "Queueing only happens on * branch, skipping queue"
  exit 0
fi

#
# Set values that wont change while we wait
# 
load_variables
max_time=20
echo "This build will block until all previous builds complete."
echo "Max Queue Time: ${max_time} minutes."
wait_time=0
loop_time=11
max_time_seconds=$((max_time * 60))

#
# Queue Loop
#
confidence=0
while true; do
  update_comparables
  echo "This Workflow Timestamp: $my_commit_time"
  echo "Oldest Workflow Timestamp: $oldest_commit_time"
  if [[ ! -z "$my_commit_time" ]] && [[ "$oldest_commit_time" > "$my_commit_time" || "$oldest_commit_time" = "$my_commit_time" ]] ; then
    # API returns Y-M-D HH:MM (with 24 hour clock) so alphabetical string compare is accurate to timestamp compare as well
    # recent-jobs API does not include pending, so it is posisble we queried in between a workfow transition, and we;re NOT really front of line.
    if [ $confidence -lt 1 ];then
      # To grow confidence, we check again with a delay.
      confidence=$((confidence+1))
      echo "API shows no previous jobs/workflows, but it is possible a previous workflow has pending jobs not yet visible in API."
      echo "Rerunning check ${confidence}/1"
    else
      echo "Front of the line, WooHoo!, Build continuing"
      break
    fi
  else
    # If we fail, reset confidence
    confidence=0
    echo "This build (${CIRCLE_BUILD_NUM}) is queued, waiting for build number (${oldest_running_build_num}) to complete."
    echo "Total Queue time: ${wait_time} seconds."
  fi

  if [ $wait_time -ge $max_time_seconds ]; then
    echo "Max wait time exceeded, considering response."
    if [ "false" == "true" ];then
      echo "Orb parameter dont-quit is set to true, letting this job proceed!"
      exit 0
    else
      cancel_current_build
      sleep 10 # wait for API to cancel this job, rather than showing as failure
      exit 1 # but just in case, fail job
    fi
  fi

  sleep $loop_time
  wait_time=$(( loop_time + wait_time ))
done

wf-963-cluster-autoscaler-tag-fix queueable
DEBUG: Making API Call to https://circleci.com/api/v2/me
DEBUG: API Success
Using API key for user: "2eddcc82-ce3a-478e-bf5c-a2f9fe456784"
This build will block until all previous builds complete.
Max Queue Time: 20 minutes.
Orb parameter 'consider-branch' is false, will block previous builds on any branch.
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY  is set.
DEBUG: Making API Call to https://circleci.com/api/v1.1/project/github/appvia/wayfinder?filter=running
DEBUG: API Success
API access successful
Checking time of workflow: 6fe3a27b-8e02-4cdb-845d-1de09c2cced1
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/6fe3a27b-8e02-4cdb-845d-1de09c2cced1
DEBUG: API Success
Workflow was created at: 2022-04-27T11:39:52Z
Checking time of workflow: 5e49bff4-b61c-48f4-8fd2-8ec7da55769d
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/5e49bff4-b61c-48f4-8fd2-8ec7da55769d
DEBUG: API Success
Workflow was created at: 2022-04-27T11:40:24Z
Checking time of workflow: 6fe3a27b-8e02-4cdb-845d-1de09c2cced1
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/6fe3a27b-8e02-4cdb-845d-1de09c2cced1
DEBUG: API Success
Workflow was created at: 2022-04-27T11:39:52Z
Checking time of workflow: bf767bb5-f023-45c9-9432-29e8ac9088c6
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/bf767bb5-f023-45c9-9432-29e8ac9088c6
DEBUG: API Success
Workflow was created at: 2022-04-27T11:33:35Z
Checking time of workflow: 63242a50-754b-4d33-9c80-c7cc979aa6d3
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/63242a50-754b-4d33-9c80-c7cc979aa6d3
DEBUG: API Success
Workflow was created at: 2022-04-27T11:10:46Z
Checking time of workflow: 013b74f9-6aea-4c77-a0c9-5142b78514c4
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/013b74f9-6aea-4c77-a0c9-5142b78514c4
DEBUG: API Success
Workflow was created at: 2022-04-21T19:30:12Z
Checking time of workflow: f7cc0a81-2c84-46a2-8a0c-3d3fbd6cd87b
DEBUG: Making API Call to https://circleci.com/api/v2/workflow/f7cc0a81-2c84-46a2-8a0c-3d3fbd6cd87b

Exited with code exit status 22
CircleCI received exit code 22

Expected behavior

If the workflow is not found, it should be ignored and the process should continue onto the next workflow to check.

[semver:patch] Blocking Forever on NULL Job

Orb version

1.8.1

What happened

"Oldest job" and "Oldest Workflow Timestamp" are returning null, which then blocks forever. Could this possibly be related to #88?

fix/repo#3508 queueable
DEBUG: Making API Call to https://circleci.com/api/v2/me
DEBUG: API Success
Using API key for user: "xxx"
This build will block until all previous builds complete.
Max Queue Time: 10 minutes.
Only blocking execution if running previous jobs on branch: fix/repo#3508
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY  is set.
DEBUG: Making API Call to https://circleci.com/api/v1.1/project/github/xxx/xxx/tree/fix/repo#3508?filter=running
DEBUG: API Success
API access successful
Orb parameter block-workflow is true.
This job will block until no previous workflows have *any* jobs running.
Oldest job: null
This Workflow Timestamp: 
Oldest Workflow Timestamp: null
This build (2260) is queued, waiting for build number (null) to complete.
Total Queue time: 0 seconds.
Only blocking execution if running previous jobs on branch: fix/repo#3508
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLECI_API_KEY  is set.
DEBUG: Making API Call to https://circleci.com/api/v1.1/project/github/trade-platform/lib/tree/fix/repo#3508?filter=running
DEBUG: API Success
API access successful
Orb parameter block-workflow is true.
This job will block until no previous workflows have *any* jobs running.
Oldest job: null
This Workflow Timestamp: 
Oldest Workflow Timestamp: null
This build (2260) is queued, waiting for build number (null) to complete.
Total Queue time: 11 seconds.

My config.yaml looks like:

orbs:
  queue: eddiewebb/[email protected]

workflows:
  version: 2
  CircleCI:
    jobs:
      - queue/block_workflow:
          time: '10'

Expected behavior

Things were working fine yesterday, but today it blocks until failure even if there is nothing else running.

[semver:patch] add "Content-Type: application/json" to all curl requests

Orb version

1.7.0

What happened

Circleci's api stopped working.

I.e. response:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: 4JEGQSiUPEc8KRujDuFT2XCMaHpV-1234567890==
</PRE>
<ADDRESS>
</ADDRESS>
* Connection #0 to host circleci.com left intact
</BODY></HTML>

Expected behavior

Lets add -H "Content-Type: application/json" to all curl requests in the src/ Circleci's api reponse should be visible

[3.1.2] Error when no commit message

Orb version

3.1.2

What happened

When force-pushing commits to a branch, they don't contain commit messages. This causes an error where CircleCI is indicating that arguments are unexpected, even though they are configured for the orb.

This same commit worked successfully when a commit message was present.

Expected behavior

The absence of a commit message should not cause a failure. It should continue processing regardless of a message.

Race condition during workflow transitions

If a queued job happens to hit API at just the right time, it will not find any running Jobs.

I think calling the workflow api differently can solve this.

Original issue:

So I have a multistep job workflow (as follows)

queue
build
deploy
and I have definitely seen the queue step allow a transition on Workflow 2 when going from 2 to 3 on Workflow 1. What kind of data can I grab to help debug?

eddiewebb / circleci-queue Goto Github PK

circleci-queue's Introduction

CircleCI Concurrency Control Orb

Basic Usage

Screenshots / Examples

Setup

Note

circleci-queue's People

Contributors

Stargazers

Watchers

Forkers

circleci-queue's Issues

RELEASE CANDIDATE READY

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Workarounds

Orb version

What happened

Expected behavior

How to fix

RELEASE CANDIDATE READY

Orb version

What happened

Expected behavior

Full logs

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

RELEASE CANDIDATE READY

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

RELEASE CANDIDATE READY

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior

Orb version

What happened

Expected behavior