dpb587 / metalink-repository-resource Goto Github PK

View Code? Open in Web Editor NEW

1.0 3.0 3.0 9.33 MB

A Concourse resource for managing versions/files in a Metalink repository.

License: MIT License

Go 98.68% Dockerfile 1.32%

concourse-resource metalink

metalink-repository-resource's Introduction

metalink-repository-resource

A Concourse resource for managing versions/files in a Metalink repository.

Source Configuration

uri - location of the repository
signature_trust_store - identities and keys used for signature verification
skip_hash_verification - skip hash verification of files
skip_signature_verification - skip signature verification of files
version - a supported version constraint (e.g. ^4.1)
filters - a list of supported filters to limit the discovered metalinks
options - a hash of supported options, depending on the repository type
- for git repositories
  - private_key - a SSH private key for git+ssh URIs
  - rebase - number of rebase attempts when pushing (default 3)
- for s3 repositories
  - access_key - access key for private S3 endpoints
  - secret_key - secret key for private S3 endpoints
  - role_arn - role arn for private S3 endpoints when using AssumeRole
include_files - a list of file globs to match when downloading a version's files (used by in)
exclude_files - a list of file globs to skip when downloading a version's files (used by in)
url_handlers - a list of URL handlers for custom download/upload configurations
- type - handler type (i.e. s3)
- include - a list of URIs that should use this handler (regex'd)
- exclude - a list of URIs that should avoid this handler (regex'd)
- options - a hash of supported options, depending on type
  - for s3:
    - access_key - access key for private S3 endpoints
    - secret_key - secret key for private S3 endpoints
    - role_arn - role arn for private S3 endpoints when using AssumeRole
mirror_files - a list of mirror configurations for mirroring files (used by out)
- destination - the mirror URI for uploading files (templated; Name, Version, SHA1, SHA256, SHA512, MD5)
- location - the ISO3166-1 alpha-2 country code for the geographical location (embedded in the metalink)
- priority - a priority for the file (embedded in the metalink)

Operations

`check`

Check for new versions in the repository.

Metadata:

version - semantic version (e.g. 4.1.2)

`in`

Download and verify the referenced file(s).

.resource/metalink.meta4 - metalink data used when downloading the file
.resource/version - version downloaded (e.g. 4.1.2)
* - the downloaded file(s) from the metalink

Parameters:

include_files - a list of file globs to match when downloading files (intersects with include_files from source configuration, when present)
skip_download - do not download blobs (only metalink.meta4 and version will be available)

`out`

Publish a metalink file to the repository.

Parameters:

metalink - path to the metalink file (one of metalink or files must be configured)
files - a list of glob paths for files to create a metalink from (one of metalink or files must be configured; requires version)
version - path to a file with the version number (only effective with files)
rename - publish the metalink file with a different file name (templated; Version)
rename_from_file - path to a file whose content is the metalink file name (alternative to rename)
options - a hash of supported options, depending on the repository type
- for git repositories
  - author_name, author_email - the commit author
  - message - the commit message

Usage

To use this resource type, you should configure it in the resource_types section of your pipeline.

- name: metalink-repository
  type: docker-image
  source:
    repository: dpb587/metalink-repository-resource

URL Credentials

When working with authenticated URLs (for either upload or download), configure the url_handlers option of the resource:

url_handlers:
- type: s3
  options:
    access_key: AKIAA1B2C3...
    secret_key: a1b2c3d4e5...

When using multiple URLs which require different configurations, use the include or exclude options to restrict usage:

url_handlers:
- type: s3
  include:
  - s3://[^/]+/org1-bucket-name/
  options:
    access_key: AKIAA1B2C3...
    secret_key: a1b2c3d4e5...
- type: s3
  include:
  - s3://[^/]+/org2-bucket-name/
  options:
    access_key: AKIAB2C3D4...
    secret_key: b2c3d4e5f6...
	mirror_files:
- destination: s3://s3-external-1.amazonaws.com/org1-bucket-name/my-private-blobs/{{.Version}}/{{.Name}}
- destination: s3://s3-external-1.amazonaws.com/org2-bucket-name/my-private-blobs/{{.Version}}/{{.Name}}

Filters

The fileversion and repositorypath filters are supported.

filters:
- repositorypath: prefix-*.meta4
- fileversion: 27.x              # equivalent to using source version

License

MIT License

metalink-repository-resource's People

Contributors

Stargazers

Watchers

Forkers

luan luis-nagaki ystros

metalink-repository-resource's Issues

Support creating metalink files

Currently, a previous task must create the metalink file before this can be resource can be used. Often, that task runs the same sort of import-file/file-upload commands. This gets redundant across multiple pipelines and repositories, and can probably be made simpler.

Alternatively, allow put operations to receive a list of files from which to generate a metalink using import-file. For the file-upload, it should probably receive a list of upload targets. Currently upload configuration can only be specified by environment variables.

A future put/out interface might look like...

metalink - path to the metalink file (one of metalink or files must be specified)
files - an array of file glob paths to create a new metalink file from (one of metalink or files must be specified)
upload_files - an array of hashes to upload files when creating a metalink with files
- to - a supported upload path with basic template interpolation support (e.g. s3://hostname/bucketname/{{.Name}}; version is available with {{.Version}}, and checksums are available, e.g. {{.SHA1}})
- env - a hash of configuration to specify through environment variables for the upload (e.g. AWS_ACCESS_KEY_ID)
version_file - a path to a file whose contents have a version number to include when creating a metalink with files
rename - publish the metalink file with a different file name (only applicable with metalink)
options - a hash of supported options, depending on the repository type
- for git repositories
  - author_name, author_email - the commit author
  - message - the commit message

With an example pipeline usage being...

jobs:
- name: build
  plan:
  - get: repo
  - get: version
    params:
      bump: rc
  - task: build
      file: repo/ci/tasks/build/config.yml
    - put: repo-dev-artifacts
      params:
        files:
        - ssoca-*
        version_file: version/version
        upload_files:
        - to: "s3://s3-external-1.amazonaws.com/my-release/releases/{{.Version}}/{{.Name}}"
          env:
            AWS_ACCESS_KEY_ID: ((assets_s3_access_key))
            AWS_SECRET_ACCESS_KEY: ((assets_s3_secret_key))
resources:
- name: repo-dev-artifacts
  type: metalink-repository
  source:
    uri: git+ssh://[email protected]:my/release.git//dev#artifacts
    options:
      private_key: ((git_private_key))

This does result in a fairly complex put configuration, though. In theory, the upload_files could (should?) become part of the resource configuration and be renamed to upload_options/upload/mirrors? This would help ensure consistency across puts to the resource, and reduce the amount of configuration needed for resource operations. If termed mirrors, it would also allow mirroring existing metalinks to other targets as they are being added to a repository.

Perhaps something more like...

jobs:
- name: build
  plan:
  - get: repo
  - get: version
    params:
      bump: rc
  - task: build
      file: repo/ci/tasks/build/config.yml
    - put: repo-dev-artifacts
      params:
        files:
        - ssoca-*
        version_file: version/version
resources:
- name: repo-dev-artifacts
  type: metalink-repository
  source:
    uri: git+ssh://[email protected]:my/release.git//dev#artifacts
    options:
      private_key: ((git_private_key))
    mirrors:
    - uri: "s3://s3-external-1.amazonaws.com/my-release/dev/{{.Version}}/{{.Name}}"
      env:
        AWS_ACCESS_KEY_ID: ((assets_s3_access_key))
        AWS_SECRET_ACCESS_KEY: ((assets_s3_secret_key))

A few more examples

This concourse resource looks pretty interesting, but I can't really tell how it works from the first look of it. At some point I'll have to experiment with setting up a simple demo on my own time, but still:

Could you post a few more examples / use-cases in the README when/if you get a chance?

Thanks for your time.

Doesn't seem possible to use a S3 repo for storing metalinks (just file mirroring).

See issue filed here: dpb587/metalink#4

I'm getting in: bad repository: load: Unmarshaling: Unmarshaling XML: EOF when specifying an S3 repo to store the metalink.

Just dropping an issue here as well since it was one of the first places I looked to see if anyone else was having this problem. 😄

Allow for private bucket usage

Once dpb587/metalink#2 or similar is implemented, this resource should also receive something like the following to support the option:

diff --git a/factory/url_loader_factory.go b/factory/url_loader_factory.go
index cc66e88..0acadec 100644
--- a/factory/url_loader_factory.go
+++ b/factory/url_loader_factory.go
@@ -3,13 +3,13 @@ package factory
 import (
        "fmt"

+       "github.com/dpb587/metalink-repository-resource/api"
+       "github.com/dpb587/metalink/file/url"
        fileurl "github.com/dpb587/metalink/file/url/file"
        ftpurl "github.com/dpb587/metalink/file/url/ftp"
        httpurl "github.com/dpb587/metalink/file/url/http"
        s3url "github.com/dpb587/metalink/file/url/s3"
        "github.com/dpb587/metalink/file/url/urlutil"
-       "github.com/dpb587/metalink/file/url"
-       "github.com/dpb587/metalink-repository-resource/api"
 )

 func GetURLLoader(handlers []api.HandlerSource) url.Loader {
@@ -40,6 +40,15 @@ func GetURLLoader(handlers []api.HandlerSource) url.Loader {
                                opts.SecretKey = valStr
                        }

+                       if val, ok := handlerSource.Options["private"]; ok {
+                               valBool, ok := val.(bool)
+                               if !ok {
+                                       panic("unsupported handler option: s3: private: expected bool")
+                               }
+
+                               opts.Private = valBool
+                       }
+
                        handlerLoader = s3url.NewLoader(opts)
                default:
                        panic(fmt.Errorf("unsupported handler: %s", handlerSource.Type))

Not sending a PR yet because of the vendor race, but figured we'd create a tracking issue.

Support Parallel Upload/Download Operations

For metalinks with multiple large files it would be very helpful to be able to upload and download in parallel. Probably add a parallel option to source with a default of 3.