gruntwork-io / fetch Goto Github PK

Download files, folders, and release assets from a specific git commit, branch, or tag of public and private GitHub repos.

Home Page: https://www.gruntwork.io/

License: MIT License

Go 100.00%

downloader git github

fetch's Introduction

gruntwork.io website

This is the code for the Gruntwork website.

Gruntwork can help you get your entire infrastructure, defined as code, in about a day. You focus on your product. We'll take care of the Gruntwork.

Docker quick start

The fastest way to launch this site is to use Docker.

git clone this repo
docker compose up
Go to http://localhost:4000 to test
If you are going to be testing the checkout flow, you must login to Aperture at: https://aperture.dogfood-stage.com/.

The default Docker compose configuration supports hot-reloading of your local environment, meaning that as you edit files to change markup, text, images, etc, your local development server will pick up these changes and reload the latest version of the site for you. This makes it quick and convenient to develop on the site locally.

Manual quick start

git clone this repo
Install Jekyll
Just the first time: bundle install
Start Jekyll server: bundle exec jekyll serve --livereload
Go to http://localhost:4000
If you are going to be testing the checkout flow, you must login to Aperture at: https://aperture.dogfood-stage.com/.

Deploying

To deploy the site:

Create a PR with your code changes
After the PR has been approved, merge it into master
Create a new tag, you can do this manually via git or in the subsequent step on the releases page - be sure to increment the version number using semantic versioning
Go to the releases page and create a draft release with the relevant information (use the "Generate Release Notes" button to make your life easier)
Release it
The CI/CD pipeline will deploy it automatically

Technologies

Built with Jekyll. This website is completely static and we use basic HTML or Markdown for everything.
Preview environments are built with Netlify.
Hosted on Amazon S3, with CloudFront as a CDN. Using s3_website to automatically upload static content to S3.
We use Bootstrap and Less.
We're using UptimeRobot, Google Analytics, and HubSpot Traffic Analytics for monitoring and metrics.

Troubleshooting

Disabling the Jekyll Feed gem

The Gruntwork website uses a Ruby Gem called Jekyll Feed which generates a structured RSS feed of "posts" on the site. Unfortunately, in development this can significantly slow down the hot-reloading of the site, forcing you to wait upwards of a minute at a time to see minor text changes locally.

You'll know this is happening when you look at the STDOUT of your docker-compose process and the final count of seconds spent Generating feed for posts is greater than 5:

web_1  |       Regenerating: 1 file(s) changed at 2021-07-21 14:31:08
web_1  |                     _data/website-terms.yml
web_1  |        Jekyll Feed: Generating feed for posts
web_1  |                     ...done in 58.507850014 seconds.

As a temporary workaround, you can open the Gemfile in the root of the project directory and temporarily comment out the line that pulls in the Jekyll Feed dependency:

source 'https://rubygems.org'
gem 'jekyll', '~> 4.1'
gem 's3_website', '3.3.0'
group :jekyll_plugins do
  gem 'jekyll-redirect-from', '0.16.0'
  gem 'jekyll-sitemap', '1.4.0'
  gem 'jekyll-paginate', '1.1.0'
  gem 'therubyracer', '0.12.3'
  gem 'less', '2.6.0'
  gem 'jekyll-asciidoc'
  gem 'jekyll-toc'
  gem 'nokogiri', '1.11.0.rc4' # Addressing security issue in earlier versions of this library
#  gem 'jekyll-feed'
end

Important - Be sure that you don't end up committing this change because we do want the Jekyll Feed plugin to run for production!

I made changes locally but they're not being reflected in my hot-reloaded development environment

This can happen especially if you add or remove files from the website's working directory. When this occurs, terminate your docker-compose process and restart it to see your changes reflected.

License

See LICENSE.txt.

fetch's People

Contributors

Stargazers

Watchers

Forkers

digitaloptimizationgroup omnypay stream-aha itshosted pombredanne derrickwilliams girishkg opsgang kmaehashi tomdee fashionmgr juullabs therockstardba achirko dhirenshumsher jniesen waccapital jvkumar ikyeazige rezahorrieh dorfilikatu doesitscript cccsv joestein ardavis3259 rivy-go pjcrosbie coryvirok testthedocs git-ty-up dg-jrichardson pks-os vishalbelsare backwardn div1127 vedala banben aapaeste mundey ohlsont upenderadepu mroddy775 cdwbrad bonedaddy andreybleme ybudimirov gtrevg cloud-admin-wafd newamsterdamlabs savadev pete0emerson necroscope java4all niallrmurphy m-bal pathologywatch forkkit schysys hntrmrrs cheredia2k12 celthi bosconi isabella232 xxxmonkeydluffyxxx flydev-fr devops-forkeables mark-ship-it ramesharun attractions-io sivaatluri567 ekmixon fyqtian worthpointlabs tagsource ramesh2051 shakahl obourdon doflo-gruntwork-io antonellocaroli purvakdevops tnn-gruntwork-io iq-scm rajbkrish clement94310 wendymunyasi irphilli

fetch's Issues

missing code verification with GPG signatures or published checksums

its Best Practices to have code verification with GPG signatures or checksums
https://github.com/docker-library/official-images#security

can you add it to fetch please?

You should be able to pass in the GitHub token via an environment variable

For private repos, this script requires you to set the --github-oauth-token flag. It would be more secure to read the token in as an environment variable called GITHUB_OAUTH_TOKEN to avoid the token being stored in bash history.

Support tag and commit ID verification.

In #34, we added the ability to validate a release asset by its checksum. But what to do about files downloaded straight from the repo? It seems the best we can do here is to validate the commit itself versus individual files, so one possibility is that fetch could be updated so that you can pass in both --tag and --commit-id and if the two don't match, fetch will fail. This will inoculate users against git tags whose associated git commit has been changed.

New version of Fetch fails with Gruntwork Installer Fetch version

When using the latest version of gruntwork installer v0.0.24, it installs Fetch v0.3.2 as part of its install.

    ubuntu-ami: Installing gruntwork-install...
    ubuntu-ami: Installing fetch version v0.3.2 to /usr/local/bin/fetch
    ubuntu-ami: Downloading https://github.com/gruntwork-io/fetch/releases/download/v0.3.2/fetch_linux_amd64 to /tmp/gruntwork-bootstrap-download-9HOtyS
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-9HOtyS to /usr/local/bin/fetch
    ubuntu-ami: Installing gruntwork-install version v0.0.24 to /usr/local/bin/gruntwork-install
    ubuntu-ami: Downloading https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/v0.0.24/gruntwork-install to /tmp/gruntwork-bootstrap-download-mz7gtj
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-mz7gtj to /usr/local/bin/gruntwork-install
    ubuntu-ami: Creating /etc/user-data as a place to store scripts intended to be run in the User Data of an EC2 instance during boot
    ubuntu-ami: Success!

If I proceed to install the latest version of Fetch v0.3.7 and have additional gruntwork-install --module-name XYZ --repo https://github.com/gruntwork-io/xxxx --tag vN.N.N commands after installing Fetch, then it will fail with the error:

==> ubuntu-ami: ERROR: You specified the --branch flag but did not provide any value.

The above only happens when using --module-name and not --binary-name, binaries like gruntkms, ssh-grunt install fine and then it will fail on installing a module.

As a workaround, I installed Fetch v0.3.7 last in the list of GW modules and then everything works as expected in building the packer image. I require that the latest fetch is installed on the AMI, which is why I'm not leaving v0.3.2 in place.

This is my install-gruntwork-modules.sh that is called from the packer json.

#!/usr/bin/env bash
# Install the Gruntwork modules

set -e

readonly GRUNTWORK_INSTALL_VERSION="v0.0.24"
readonly BASH_COMMONS_VERSION="v0.1.2"
readonly GRUNTKMS_VERSION="v0.0.8"
readonly MODULE_SECURITY_VERSION="v0.27.1"
readonly MODULE_AWS_MONITORING_VERSION="v0.19.0"
readonly MODULE_STATEFUL_SERVER_VERSION="v0.8.1"
readonly METRICS_MODULES="--disk-path=/ --mem-util --mem-used --mem-avail --disk-space-util --auto-scaling"
readonly FETCH_VERSION="v0.3.7"

function get_aws_region {
    curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | awk -F\" '{print $4}'
}

function install_bash_commons {
  local -r bash_commons_version="$1"

  echo "Installing bash-commons version $bash_commons_version"
  gruntwork-install --module-name 'bash-commons' --repo https://github.com/gruntwork-io/bash-commons --tag ${bash_commons_version}
}

function install_security_packages {
  local -r kms_version="$1"
  local -r module_security_version="$2"

  echo "Installing Gruntwork Security Modules"
  gruntwork-install --binary-name 'gruntkms' --repo https://github.com/gruntwork-io/gruntkms --tag ${kms_version}
  gruntwork-install --binary-name 'ssh-grunt' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
  gruntwork-install --module-name 'auto-update' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
  gruntwork-install --module-name 'ntp' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
  gruntwork-install --module-name 'ip-lockdown' --repo https://github.com/gruntwork-io/module-security --tag ${module_security_version}
}

function install_monitoring_packages {
  local -r module_aws_monitoring_version="$1"
  local -r metrics_modules="$2"
  local -r aws_region=$(get_aws_region)

  echo "Installing Gruntwork Monitoring Modules"
  gruntwork-install --module-name 'logs/cloudwatch-log-aggregation-scripts' --repo https://github.com/gruntwork-io/module-aws-monitoring --tag ${module_aws_monitoring_version} --module-param aws-region="$aws_region"
  gruntwork-install --module-name 'metrics/cloudwatch-memory-disk-metrics-scripts' --repo https://github.com/gruntwork-io/module-aws-monitoring --tag ${module_aws_monitoring_version}  --module-param "metrics-to-monitor=$metrics_modules"
  gruntwork-install --module-name 'logs/syslog' --repo https://github.com/gruntwork-io/module-aws-monitoring --tag ${module_aws_monitoring_version}
}

function install_stateful_server_packages {
  local -r module_server_version="$1"

  echo "Installing Gruntwork Stateful Server Modules"
  gruntwork-install --module-name 'persistent-ebs-volume' --repo 'https://github.com/gruntwork-io/module-server' --tag ${module_server_version}
  gruntwork-install --module-name 'route53-helpers' --repo 'https://github.com/gruntwork-io/module-server' --tag ${module_server_version}
}

function install_fetch {
  local -r fetch_version="$1"

  echo "Installing Fetch"
  gruntwork-install --binary-name 'fetch' --repo https://github.com/gruntwork-io/fetch --tag ${fetch_version}

}
function assert_env_var_not_empty {
  local -r var_name="$1"
  local -r var_value="${!var_name}"

  if [[ -z "$var_value" ]]; then
    echo "ERROR: Required environment variable $var_name not set."
    print_usage
    exit 1
  fi
}

function install_gruntwork_modules {
  assert_env_var_not_empty "GITHUB_OAUTH_TOKEN"

  curl -LsS https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash /dev/stdin --version ${GRUNTWORK_INSTALL_VERSION}

  install_bash_commons ${BASH_COMMONS_VERSION}
  install_fetch ${FETCH_VERSION}
  install_security_packages ${GRUNTKMS_VERSION} ${MODULE_SECURITY_VERSION}
  install_monitoring_packages ${MODULE_AWS_MONITORING_VERSION} ${METRICS_MODULES}
  install_stateful_server_packages ${MODULE_STATEFUL_SERVER_VERSION}
}

install_gruntwork_modules

If the install_fetch ${FETCH_VERSION} is called directly after install_bash_commons ${BASH_COMMONS_VERSION} and prior to other modules, it will fail to install any subsequent modules, but if the install function is called last, after install_stateful_server_packages ${MODULE_STATEFUL_SERVER_VERSION} then it works.

Here is a full error output:

==> ubuntu-ami: Provisioning with shell script: /Users/john/code/infra-live-modules/terraform/aws/services/monitor-pm2/packer/install-gruntwork-modules.sh
    ubuntu-ami: Installing gruntwork-install...
    ubuntu-ami: Installing fetch version v0.3.2 to /usr/local/bin/fetch
    ubuntu-ami: Downloading https://github.com/gruntwork-io/fetch/releases/download/v0.3.2/fetch_linux_amd64 to /tmp/gruntwork-bootstrap-download-9HOtyS
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-9HOtyS to /usr/local/bin/fetch
    ubuntu-ami: Installing gruntwork-install version v0.0.24 to /usr/local/bin/gruntwork-install
    ubuntu-ami: Downloading https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/v0.0.24/gruntwork-install to /tmp/gruntwork-bootstrap-download-mz7gtj
    ubuntu-ami: Got expected status code 200
    ubuntu-ami: Moving /tmp/gruntwork-bootstrap-download-mz7gtj to /usr/local/bin/gruntwork-install
    ubuntu-ami: Creating /etc/user-data as a place to store scripts intended to be run in the User Data of an EC2 instance during boot
    ubuntu-ami: Success!
    ubuntu-ami: Installing bash-commons version v0.1.2
==> ubuntu-ami: 2020-03-27 07:46:17 [INFO] [gruntwork-install] Installing from bash-commons...
    ubuntu-ami: Downloading tag "v0.1.2" of https://github.com/gruntwork-io/bash-commons ...
==> ubuntu-ami: 2020-03-27 07:46:17 [INFO] [gruntwork-install] Downloading module bash-commons from https://github.com/gruntwork-io/bash-commons
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Executing /tmp/gruntwork-script-modules/bash-commons/install.sh
    ubuntu-ami: Extracting files from <repo>/modules/bash-commons to /tmp/gruntwork-script-modules/bash-commons ...
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [install.sh] Starting install of bash-commons...
    ubuntu-ami: Download and file extraction complete.
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [install.sh] Successfully installed bash-commons!
    ubuntu-ami: Installing Fetch
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Success!
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Installing fetch...
    ubuntu-ami: Downloading release asset fetch_linux_amd64 to /tmp/gruntwork-script-modules/fetch_linux_amd64
    ubuntu-ami: Download of release assets complete.
==> ubuntu-ami: 2020-03-27 07:46:18 [INFO] [gruntwork-install] Moving /tmp/gruntwork-script-modules/fetch_linux_amd64 to /usr/local/bin/fetch and setting execute permissions
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Success!
    ubuntu-ami: Installing Gruntwork Security Modules
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Installing gruntkms...
    ubuntu-ami: Downloading release asset gruntkms_linux_amd64 to /tmp/gruntwork-script-modules/gruntkms_linux_amd64
    ubuntu-ami: Downloaded /tmp/gruntwork-script-modules/gruntkms_linux_amd64
    ubuntu-ami: Download of release assets complete
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Moving /tmp/gruntwork-script-modules/gruntkms_linux_amd64 to /usr/local/bin/gruntkms and setting execute permissions
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Success!
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.
==> ubuntu-ami: 2020-03-27 07:46:19 [INFO] [gruntwork-install] Installing ssh-grunt...
    ubuntu-ami: Downloading release asset ssh-grunt_linux_amd64 to /tmp/gruntwork-script-modules/ssh-grunt_linux_amd64
    ubuntu-ami: Downloaded /tmp/gruntwork-script-modules/ssh-grunt_linux_amd64
    ubuntu-ami: Download of release assets complete
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Moving /tmp/gruntwork-script-modules/ssh-grunt_linux_amd64 to /usr/local/bin/ssh-grunt and setting execute permissions
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Success!
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Repository is not public. GITHUB_OAUTH_TOKEN environment variable is required.
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Installing from auto-update...
==> ubuntu-ami: 2020-03-27 07:46:20 [INFO] [gruntwork-install] Downloading module auto-update from https://github.com/gruntwork-io/module-security
==> ubuntu-ami: ERROR: You specified the --branch flag but did not provide any value.
==> ubuntu-ami: Terminating the source AWS instance...

How do you compile from source?

My operating system doesn't match any of the available binary releases. How do I compile from source using go build?

When user passes an empty value for --branch, fetch returns the wrong error

When fetch is called with --branch but no value is given for --branch, fetch returns an inaccurate error message:

ERROR: You must specify exactly one of --tag, --commit, or --branch. Run "fetch --help" for full usage info.

Feature: port architecture and OS introspection support from gruntwork-install

gruntwork-install has the ability to introspect the OS and architecture of the system to determine which binary asset to install.

I can see why this is not in fetch, which is more generic and thus cannot rely on our naming conventions, but it would be nice if that functionality was baked into fetch itself. The primary use case for this would be Windows support. gruntwork-install is implemented in bash, which means it can't be used in most Windows environments. It would be nice if we can port that functionality into fetch, so that we can support Windows too.

Notes:

I am not sure how to make this more generic. Perhaps we can assume our naming convention, and then if it doesn't match have an interactive prompt to select the correct one, sorting the list by name introspection?
Alternatively, we can reimplement gruntwork-install to be in python or Go so that it is portable.

Certain repos fail silently

Works with other repos but the following fails/freezes:

$ fetch --repo="https://github.com/Jannomag/Yaru-Colors" --ref="20.10.1" --source-path="/Themes" /tmp/yaru-themes
Downloading git reference "20.10.1" of https://github.com/Jannomag/Yaru-Colors ...

It fails also with --commit, --tag, --branch etc.
Is there any way to debug what is happening?

Does not work on Raspberry Pi 3 running Ubuntu Server 20.04

Can't download every release assets and/or by wildcard

Unless I'm missing something, there is no ability to download every assets for a particular release tag, ie: everything inside https://github.com/gruntwork-io/fetch/releases/tag/v0.1.1

This could happen when the --release-asset is not specified or it is with the star "*" symbol.

Also, it would be nice to be able to specify wildcard/s as part of the --release-asset, and download those assets matching the expression.

Fetching asset files with + signs causes them not to be found.

For a repo+release that has a file named pipeline-assembly-0.1.0+1-6d47f7c7.jar,
--release-asset=pipeline-assembly* works fine but --release-asset=pipeline-assembly-0.1.0+1-6d47f7c7.jar says Could not find assets matching pipeline-assembly-0.1.0+1-6d47f7c7.jar in release v0.1.0+1-6d47f7c7.

My assumption is there's some sort of url encoding issue going on with the "+" when querying the github api for the assets on the release.

Incorrect error: Tag does not exist.

When I run fetch as follows:

fetch --tag '~>v0.0.1' --repo https://github.com/gruntwork-io/module-security --source-path="/modules/auto-update" /tmp

I get the following error:

ERROR: Error occurred while computing latest tag that satisfies version contraint expression: -1 - Tag does not exist

This is odd because v0.0.1 exists on this repo and changing the tag constraint expression to >v0.0.1 makes it work. This is clearly a bug.

Progress info

Nice work here in this repo, thank you.

I would like to propose a feature: to add a flag to add some kind of progress indicator.
It is especially important when downloading big files.

When the tag expression has no acceptable value, fetch defaults to the first tag.

Given:

The repo has tags: v0.0.1,v0.0.2,v0.0.3
fetch requested the repo with tag expression ~>0.0.4

Then:

fetch will download tag v0.0.1 of the repo

Instead, fetch should error out and complain that no valid tag was found.

Add support for resolving symlinks

I attempted to download files from a repo that looked like this:

...
lib
- script.sh
modules
- run-x
  - hello.txt
  - lib --> ../lib/


where `/modules/lib` is a symlink to `/lib/`.

I'd like `fetch` to inclue all the files in `/lib/*` (e.g. `/lib/script.sh`) in my downloads, but instead, `fetch` only includes the path of the folder to which the symlink points.

A helpful enhancement to fetch would be to optionally resolve symlinks. This could result in unexpected behavior if symlinks reference files outside the repo (but present on a local filesystem), and would represent a backward-incompatible change, so this feature should be enabled as an option, but not by default.

No Bug, but need Help

Hello Thank you for this much promising tool.

Unfortunately I cannot get the programm to run.

Here my cmdline output:

Can you please tell me what is actually wrong? When I am able to clone the same repo, why I am not able to run your fetcher in same context?

I want to use it, to save my all github releases and ther assets locally as fallback.

fetch no longer works with just --branch

It looks like adding --ref in #79 may have broken the --branch parameter.

With fetch version v0.3.9:

gruntwork-install \
  --module-name "gcp-helpers" \
  --repo "https://github.com/gruntwork-io/terraform-google-ci" \
  --branch "gcp-helpers"

Works as follows:

2021-02-01 14:10:38 [INFO] [gruntwork-install] Installing from gcp-helpers...
2021-02-01 14:10:38 [INFO] [gruntwork-install] Downloading module gcp-helpers from https://github.com/gruntwork-io/terraform-google-ci
Downloading latest commit from branch "gcp-helpers" of https://github.com/gruntwork-io/terraform-google-ci ...
Extracting files from <repo>/modules/gcp-helpers to /tmp/gruntwork-script-modules/gcp-helpers ...
Download and file extraction complete.
2021-02-01 14:10:39 [INFO] [gruntwork-install] Setting GRUNTWORK_INSTALL_TAG to 
2021-02-01 14:10:39 [INFO] [gruntwork-install] Setting GRUNTWORK_INSTALL_BRANCH to gcp-helpers
2021-02-01 14:10:39 [INFO] [gruntwork-install] Executing /tmp/gruntwork-script-modules/gcp-helpers/install.sh

But with fetch version v0.3.13:

2021-02-01 13:31:02 [INFO] [gruntwork-install] Installing from gcp-helpers...
2021-02-01 13:31:02 [INFO] [gruntwork-install] Downloading module gcp-helpers from https://github.com/gruntwork-io/terraform-google-ci
Downloading git reference "v0.1.0" of https://github.com/gruntwork-io/terraform-google-ci ...
Extracting files from <repo>/modules/gcp-helpers to /tmp/gruntwork-script-modules/gcp-helpers ... 0 files extracted
Download and file extraction complete.
2021-02-01 13:31:02 [ERROR] [gruntwork-install] No files were downloaded. Are you sure "gcp-helpers" is a valid Script Module in https://github.com/gruntwork-io/terraform-google-ci (tag = , branch = gcp-helpers)?

I'm guessing the way GitRef is set here to latestTag is causing the code to go down the wrong code path.

Possible build issue: fetch_linux_amd64 v0.3.9

When executing fetch_linux_amd64 0.3.9:

./fetch --help
./fetch: 1: --2020-05-04: not found
./fetch: 2: Syntax error: "(" unexpected

Prior versions execute successfully. I tried downloading a second time, same result.

Bitbucket support?

I'm interested to use fetch with Bitbucket instead of Github Enterprise; and would be interested to take a stab at implementing.

Please provide some guidance on which part of the code I should focus on to support this. Thanks!

Support fetching private repos with SSH keys

Using SSH keys makes secrets management easier in certain environments (e.g., Docker with --mount=type=ssh) as it allows avoiding multistage builds and dancing with environment variables.

ARM compiled binary

Hello,

I will start by saying nice work @gruntwork-ci and all contributors, I really like the simplicity.

Would it be possible to offer pre-compiled binary for arm based platforms?
Raspberry Pi community, and probably Apple M1 will probably appreciate it!

I can compile it for myself but sadly not everyone has the patience or knowledge to do it.

One-Time Segmentation Fault

In our automated tests, we encountered a "segmentation fault" when running fetch:

TestPersistentEbsVolume 2016/05/20 15:23:27 1463757807,,ui,message,    
gruntwork-persistent-ebs-volume-example-ubuntu: /usr/local/bin/gruntwork-install: line 69:  2444
 Segmentation fault      fetch --repo="$SCRIPT_MODULES_REPO" --tag="$tag" --branch="$branch" 
--github-oauth-token="$github_token" "/modules/$module_name" "$download_path/$module_name" 
> /dev/null

This is the only time we've seen this, so I'll close this issue given that we don't intend to pursue this, but if anyone encounters this again, please re-open this issue.

Does fetch support git-lfs?

Some large files in github are used git-lfs and will the fetch download those files with the real content instead of the pointer file?

Update fetch to allow downloading binaries from GitHub releases.

It's a pain to download private GitHub binaries as part of a release. Fetch could handle this pretty easily.

Fetching Using --branch Gives Tag Error

Using fetch_linux_amd64 to fetch a copy of my repo by branch name using OAuth token. So the command I am using is the below:

./fetch_linux_amd64 --repo="https://github.com/AAA/REPO_A" --branch="master" ./LOCAL_REPO_A

The error that it's returning is:
ERROR: Error occurred while computing latest tag that satisfies version contraint expression: -1 - Malformed version: testing_fix-multi-netif-ctrl-tab_07_06

"testing_fix-multi-netif-ctrl-tab_07_06" in the error is a pre-release tag that I had created on the repo. I am confused as to why it's even computing/checking tag names because I want to fetch the repo by branch name and had specified using --branch option. When I tried other branch names besides master I got the same error.

Then I tried the original command and substituted --branch with --tag, see below:
./fetch_linux_amd64 --repo="https://github.com/AAA/REPO_A" --tag="master" ./LOCAL_REPO_A

It seems that the new command was able to fetch the master branch of my repo. I am confused as to why the --tag would work but --branch didn't work. This seems like a bug.

Handle tags that do not follow semantic versioning

Your documentation indeed points out that tags need to follow SemVer:

Note that fetch assumes git tags are specified according to Semantic Versioning principles.

However, git tags that are not semantically versioned are valid tags, and if one exists in the source repository, the tool breaks, even if you are not requesting a --tag.

I suggest that tags should not be required to be semantically versioned unless a Tag Contraint Expression is specified.

This would allow the tool to work in repositories where non-SemVar tags exist.

To replicate:

fetch $ git rev-parse HEAD
83ed55048791ec5385d4143a78798e2a6c82b9f8

fetch $ ./fetch --repo=https://github.com/pete0emerson/demo --branch=main demo
Downloading latest commit from branch "main" of https://github.com/pete0emerson/demo ...
Extracting files from <repo>/ to demo ...
Download and file extraction complete.

demo $ git tag banana
demo $ git push --tags
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/pete0emerson/demo.git
 * [new tag]         banana -> banana

fetch $ ./fetch --repo=https://github.com/pete0emerson/demo --branch=main demo
ERROR: Error occurred while computing latest tag that satisfies version contraint expression: -1 - Malformed version: banana

How to add all release files that end with file extension

Can you use the release-asset regex to get many files that match a file extension?

fetch --repo="https://github.com/foo/bar" --tag="0.1.5" --release-asset="*.tar.gz" /tmp

similar to #58, but for any other files that match

Use better log formatting

I was debugging an issue in a call to gruntwork-install, and my underlying issue was actually with the command I passed to fetch. But the logs output by fetch don't make it obvious that it's fetch throwing the error. Here's an example:

    ubuntu-ami: 2017-11-14 20:47:53 [INFO] [gruntwork-install] Success!
    ubuntu-ami: 2017-11-14 20:47:53 [INFO] [gruntwork-install] Installing from install-kafka...
    ubuntu-ami: 2017-11-14 20:47:53 [INFO] [gruntwork-install] Downloading module install-kafka from https://github.com/gruntwork-io/package-kafka
    ubuntu-ami: ERROR: You must specify exactly one of --tag, --commit, or --branch. Run "fetch --help" for full usage info.

The statement Run "fetch --help" for full usage info. is an indicator this is a fetch error, but the log message should be prefixed with the fetch name, as gruntwork-install logs are in the example above.

How to download Source code (tar.gz) asset of release

Sorry, I not understand how to download release source code (tar.gz)?

# fetch --repo="https://github.com/hashicorp/go-version" --tag="v1.2.0"  --release-asset="v1.2.0.tar.gz" /tmp
ERROR: Could not find assets matching v1.2.0.tar.gz in release v1.2.0

fetch doesn't work with "SUB" domain in repo URL

fetch is not working if there are any subdomain in the "--repo" option
fetch --repo="https://example.github.com/foo/bar" --tag="0.1.5" --release-asset="foo.exe" /tmp

The error is:

[root@jenkins01 ~]# fetch --repo="https://example.github.com/foo/bar" --tag="0.1.5" --release-asset="foo.exe" /tmp
ERROR: Error occurred while getting tags from GitHub repo: -1 - 300 - GitHub Repo URL https://example.github.com/foo/bar could not be parsed correctly

"example" SUB domain is used here for demonstration purpose.

Please specify code license

First of all, great thanks for sharing your work!

Would you mind to add an explicit copyright licence to your code? Base on your other repose I assume you wanted to use MIT license for this porject as well :-)

Thanks in advance!

Cannot install on FreeBSD

this fetch program cannot be installed on BSD because of FreeBSD's preinstalled program similar to wget or curl, also called fetch. Placing both in $PATH can cause ports (and maybe other programs) to malfunction.

Use of --source-path can include unwanted files in downloads.

I attempted to download files from a repo that looked like this:

- ...
- modules
  - run-x
    - hello.txt
  - run-x-y
  - run-x-y-z

using this fetch command:

fetch --repo <repo> --branch <branch> --source-path "/modules/run-x/" /tmp/

At this point, /tmp should have only these files:

hello.txt

But instead, it has these files:

-x-y
-x-y-z
hello.txt

If a repo has no tags, fetch exits with an error

When using fetch with --branch or --commit, you should be able to use a repo that has no tags. But if you do that, you get this error:

Received an HTTP 404 Response when attempting to query the repo for its tags.

In some cases, files downloaded but with empty content.

Given the following:

./fetch --repo="https://github.com/gruntwork-io/packer-modules" --tag="0.0.1" --github-oauth-token="<igthub-auth-token" /modules/base/ubuntu1604 /tmp/packer-modules/base/ubuntu1604

The output is as follows:

$ ls -al /tmp/packer-modules/base/ubuntu1604/
total 0
drwxr-xr-x  4 josh  wheel  136 Apr 27 13:49 .
drwxr-xr-x  3 josh  wheel  102 Apr 27 13:49 ..
-rw-r--r--  1 josh  wheel    0 Apr 27 13:49 README.md
-rw-r--r--  1 josh  wheel    0 Apr 27 13:49 ubuntu1604.sh

Note that ubuntu1604.sh should not have size 0.

Two Digit Versions

Hi All,

While SemVer requires X.Y.Z it would be great to support X.Y as well as this is also a common version format used in GitHub.

To illustrate:
fetch --repo="https://github.com/dnephin/dobi" --tag=">=0.9" --release-asset="dobi-windows.exe" %CMDER_ROOT%\bin
Fails, while
fetch --repo="https://github.com/dnephin/dobi" --tag="v0.9" --release-asset="dobi-windows.exe" %CMDER_ROOT%\bin
Works but the 'v' is required to make this work.

Use a proper logger

fetch currently writes directly to stdout and stderr, but we should use a proper logger. That way, we'd have log levels, logs would go to stderr, we could swap in a logger more easily that captures the output in tests, and so on.

Write file to stdout

Describe the solution you'd like
When grabbing a file, it would be nice to write the file to stdout. This would be useful when downloading a script and wanting to pipe it to another command.

Describe alternatives you've considered
Specifying /dev/stdout as the filename. It's ugly but has worked.

Additional context
None given

Can't download anything from private repo

So I've tried this so many times in so many different ways but the program says that it succeeded but doesn't actually download into the /tmp dir next to the exe. I don't think I am doing anything wrong but here's what I got setup atm.

@echo off
SET GITHUB_OAUTH_TOKEN=123
fetch.exe --repo="https://github.com/User/repo" --branch="master" --source-path="/Folder/File.txt" /tmp
PAUSE

I tried it in cmd and in a bat file, the same happens thing both times.

Downloading latest commit from branch "master" of https://github.com/User/repo ...
Extracting files from <repo>/Folder/Text.txt to /tmp ...
Download and file extraction complete.
Press any key to continue . . .

Any help would be appreciated.

fetch doesn't work with "SUB" domain in repo URL

fetch is not working if there are any subdomain in the "--repo" option
fetch --repo="https://example.github.com/foo/bar" --tag="0.1.5" --release-asset="foo.exe" /tmp

The error is:

[root@jenkins01 ~]# fetch --repo="https://example.github.com/foo/bar" --tag="0.1.5" --release-asset="foo.exe" /tmp
ERROR: Error occurred while getting tags from GitHub repo: -1 - 300 - GitHub Repo URL https://example.github.com/foo/bar could not be parsed correctly

"example" SUB domain is used here for demonstration purpose.

Unable to fetch from repositories that have tags of format 0.1.2.deadbeef

Unable to fetch from repositories that have tags of format 0.1.2.deadbeef
It is because latestTag variable is, in this case, empty.
I've made a workaround there, but I am not sure whether it should get merged or not.

Feature Request: Ability to `rename` and`chmod` resultant file

what

I wish that we could specify the octal mode of the file download from the github release
I wish that we could specify the destination name

why

So that I can set the execution bit (E.g. 0755) without calling chmod explicitly
Extremely common usage pattern is to download binary releases of apps (e.g. fetch) and then run them

example interface

fetch --repo https://github.com/gruntwork-io/fetch \
      --tag=v0.2.0 \
      --release-asset=fetch_darwin_amd64 \
      --asset-name=fetch \
      --asset-file-mode=0755 \
        /usr/local/bin

Support git refs

Right now fetch supports --commit, --branch, and --tag, but it requires you to know which type you are passing in. This can be limiting especially in CI where the logic to know if the build is on a tag, branch, or commit sha can be quite complex depending on the platform.

Propose a unified --ref flag that can be used to pass in any valid git ref and fetch does the right thing.

No warning when fetch downloads 0 files.

Sometimes, fetch will download no files at all, for example when the --source-dir property specified a file/folder prefix for which there are no files. This isn't an error, but it'd be nice if Fetch gave a warning that explicitly said no files were downloaded.

Panic when repo has no existing tags

I ran fetch against a repo that didn't have any tags and it panics:

goroutine 1 [running]:
panic(0x7f8a00, 0xc820010030)
    /usr/local/go/src/runtime/panic.go:464 +0x3e6
main.getLatestAcceptableTag(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/josh/go/src/github.com/gruntwork-io/fetch/tag.go:28 +0x6f1
main.runFetch(0xc820090a00, 0x0, 0x0)
    /Users/josh/go/src/github.com/gruntwork-io/fetch/main.go:102 +0xaba
main.runFetchWrapper(0xc820090a00)
    /Users/josh/go/src/github.com/gruntwork-io/fetch/main.go:48 +0x30
github.com/gruntwork-io/fetch/vendor/github.com/codegangsta/cli.(*App).Run(0xc820084840, 0xc82000a0e0, 0x7, 0x7, 0x0, 0x0)
    /Users/josh/go/src/github.com/gruntwork-io/fetch/vendor/github.com/codegangsta/cli/app.go:201 +0x13ce
main.main()
    /Users/josh/go/src/github.com/gruntwork-io/fetch/main.go:43 +0x517

The cause is this code:

if tagConstraint == "" {
  tagConstraint = versions[len(versions)-1].String()
}

The code should check that versions is not empty.

Prevent silent fail

Hi!,

We are using this command in our CI/CD pipeline:

fetch --repo="https://github.com/..." --tag="1.0.9" --github-oauth-token="****" --release-asset="example-v1.0.9.zip" tmp-path

Yesterday this command fails because of a GitHub authorization error, however, it only logs the error instead of exiting with non-zero code this causes the pipeline to continue then exist on another step that depends on what has been downloaded by fetch.

Please we need an option to exit on error.

Add fetch in brew

Any change to add fetch as a brew formulae?

HTTPS and Self-Signed Certificate support

Hi,

I added support for Self-Signed Certificate with InsecureSkipVerify: true settable with a parameter, does a PR is welcome for this feature ?

My use case is quite specific. I have a custom software which the goal is to update a set of executables embedded on clients in the wild, then I use fetch to download releases from private repo on Github.
The issue I had is the following. The clients are behind firewalls where only our backend server is whitelisted, so a connection to Github was impossible.
I then used NGINX as a reverse proxy to forward requests from our own domain to Github, eg: https://github.example.com/MyAccount/MyRepo/ and now fetch let me download the release from this private repo using this custm URL 👍

fetch is an awesome tool ✌

Cannot fetch file from GitHub enterprise by commit

I'm using fetch v0.3.2 against our GH Enterprise instance. The initial logging indicates Enteprise is detected

Assuming GitHub Enterprise since the provided url (<enterprise url>/<redacted repo>) does not appear to be for GitHub.com

but then later on there is an error
ERROR: Error occurred while downloading zip file from GitHub repo: 500 - Failed to download file at the url https://api.github.com/repos/<redacted repo>

It should be using a URL beginning with https://<enterprise url>/api/v3/<repo>

Fetch version constraints tag does not support paging in tags list

If you are referencing a sufficiently old release tag in a version constraints tag, fetch will fail.

Example:

fetch --repo 'https://github.com/gruntwork-io/module-ecs' --tag '~>v0.3.2' foo

fails with:

ERROR: Error occurred while computing latest tag that satisfies version contraint expression: -1 - Tag does not exist

gruntwork-io / fetch Goto Github PK

fetch's Introduction

gruntwork.io website

Docker quick start

Manual quick start

Deploying

Technologies

Troubleshooting

Disabling the Jekyll Feed gem

I made changes locally but they're not being reflected in my hot-reloaded development environment

License

fetch's People

Contributors

Stargazers

Watchers

Forkers

fetch's Issues

The error is:

The error is:

what

why

example interface

Recommend Projects

Recommend Topics

Recommend Org