puppetlabs / pdksync Goto Github PK

View Code? Open in Web Editor NEW

13.0 18.0 29.0 416 KB

Manage large sets of modules with automation built on top of PDK

License: Apache License 2.0

Ruby 100.00%

puppet pdk modules git hacktoberfest

pdksync's Issues

NoMethodError: undefined method `error' for PdkSync::Logger:Class

Describe the Bug

Error logging code explodes.

NoMethodError: undefined method `error' for PdkSync::Logger:Class
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync/utils.rb:146:in `rescue in push_staged_files'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync/utils.rb:143:in `push_staged_files'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync.rb:247:in `block in main'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync.rb:158:in `each'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync.rb:158:in `main'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync/rake_tasks.rb:10:in `block in <top (required)>'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'

Expected Behavior

Error to be logged correctly

add a rake task to update module dependencies

Use Case

pdksync provides a method to "globally" manage requirement, it would be great to be able to manage dependencies in a similar way.

Describe the Solution You Would Like

bundle exec rake 'pdksync:update_dependencies[puppetlabs/stdlib,version_requirement,>= 6.0.0 < 9.0.0]'

Describe Alternatives You've Considered

extending the update_requirements task

Regression in recent `git` gem releases

There appears to be a regression in the recent versions of the git dependency. If I don't pin to 1.14.0 I get the following failure in my pdksync pipeline.

Looks like the checkout command is invalid?? (If I run the git command manually and don't throw away the stderr, I get error: pathspec 'pdksync_pdksync_tags/hl-latest-0-g76a98ec' did not match any file(s) known to git)

I'm using gitlab instead of the more common github, but as this is a failure with a local git command, I think this isn't relevant.

$ bundle exec rake pdksync
To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
INFO - PdkSync: Commit branch_name=pdksync_{ref} commit_message=pdksync_{ref}
INFO - PdkSync: PR title = pdksync_{ref}
INFO - PdkSync: mymodule, 
INFO - PdkSync: delete module directory
INFO - PdkSync: cloned
WARN - PdkSync: (WARNING) Using pdk on PATH not '/opt/puppetlabs/pdk/bin/pdk'
INFO - PdkSync: pdk update
rake aborted!
Git::FailedError: git '--git-dir=/builds/example/puppet-platform/pdk/pdksync-config/modules_pdksync/mymodule/.git' '--work-tree=/builds/example/puppet-platform/pdk/pdksync-config/modules_pdksync/mymodule' '-c' 'core.quotePath=true' '-c' 'color.ui=false' 'checkout' 'pdksync_pdksync_tags/hl-latest-0-g76a98ec'  2>&1
status: pid 1843 exit 1
stderr: ""
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/lib.rb:1215:in `command'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/lib.rb:834:in `checkout'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/base.rb:359:in `checkout'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/branch.rb:28:in `checkout'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/bundler/gems/pdksync-25b4d3031c11/lib/pdksync/utils.rb:329:in `checkout_branch'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/bundler/gems/pdksync-25b4d3031c11/lib/pdksync/utils.rb:195:in `create_commit'

GITHUB_TOKEN not needed when GITLAB_TOKEN provided

Describe the Bug

GITHUB_TOKEN shouldn't be necessary if using GITLAB_TOKEN

❯ export GITLAB__TOKEN=XXXXXXXXXXXXX
❯ bundle exec rake git:clone_managed_modules
rake aborted!
ArgumentError: Git platform access token for Github not set - use 'export GITHUB_TOKEN="<your token>"' to set

FATAL output

I don't think the FATAL output here is actually correct:

❯ bundle exec rake 'pdksync:pdk_validate'
INFO - PdkSync: puppet-profile_base,
INFO - PdkSync:


FATAL - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk validate -a': pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.4.1
pdk (INFO): Running all available validators...
pdk (INFO): Validator 'puppet-epp' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-profile_base'. No files matching '["**/*.epp"]' found to validate.
pdk (INFO): Validator 'task-metadata-lint' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-profile_base'. No files matching '["tasks/*.json"]' found to validate.

INFO - PdkSync: validated
INFO - PdkSync: puppet-git_version,
INFO - PdkSync:


FATAL - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk validate -a': pdk (WARN): This module is compatible with an older version of PDK. Run `pdk update` to update it to your version of PDK.
pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.4.1
pdk (INFO): Running all available validators...
pdk (INFO): Validator 'puppet-syntax' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-git_version'. No files matching '["**/*.pp"]' found to validate.
pdk (INFO): Validator 'puppet-lint' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-git_version'. No files matching '["**/*.pp"]' found to validate.
pdk (INFO): Validator 'puppet-epp' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-git_version'. No files matching '["**/*.epp"]' found to validate.

INFO - PdkSync: validated
INFO - PdkSync: puppet-monitoring_scripts,
INFO - PdkSync:


FATAL - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk validate -a': pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.4.1
pdk (INFO): Running all available validators...
pdk (INFO): Validator 'puppet-epp' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-monitoring_scripts'. No files matching '["**/*.epp"]' found to validate.
pdk (INFO): Validator 'task-name' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-monitoring_scripts'. No files matching '["tasks/**/*"]' found to validate.
pdk (INFO): Validator 'task-metadata-lint' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-monitoring_scripts'. No files matching '["tasks/*.json"]' found to validate.

INFO - PdkSync: validated

nonexistant branch referened in readme

Describe the Change You Would Like

A clear and concise description of what you would like changed.

README.md references the pdksync-externalisation-fixes branch... there is no such branch. Please update the section about what to put in your Gemfile.

clone_managed_modules produces multiple warnings

Describe the Bug

The two warnings below are repeated many many times while running the task

╔ ☕️  beanm1:~/repos/dio-pdksync (main ✘) ✭
╚ᐅ bundle exec rake git:clone_managed_modules
INFO - PdkSync: ploperations-account,
WARNING: `Faraday::Connection#authorization` is deprecated; it will be removed in version 2.0.
While initializing your connection, use `#request(:authorization, ...)` instead.
See https://lostisland.github.io/faraday/middleware/authentication for more usage info.
/Users/geneliverman/repos/dio-pdksync/.bundle/gems/ruby/2.7.0/gems/json_pure-2.1.0/lib/json/common.rb:156: warning: Using the last argument as keyword parameters is deprecated

Expected Behavior

I expect to run the task and not see a bunch of notices about things that need to be addressed

Environment

Version 0.6.0

gem install doesnt seem to pickup the newly released 0.6.0 version

Describe the Bug

while following the pdksync setup using the README in the repository, there are 2 problems

attachmentgenie-pdksync ❯ bundle install --path .bundle/gems/ main
[DEPRECATED] The --path flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use bundle config set path '.bundle/gems/', and stop using this flag
Fetching https://github.com/puppetlabs/pdksync
fatal: Needed a single revision
Git error: command git rev-parse --verify pdksync-externalisation-fixes in directory

for which i raised a PR (#156), the second problem, is that after fixing the gem file, bundler install the 0.5.0 version instead of the latest released version.

attachmentgenie-pdksync ❯ bundle install --path .bundle/gems/
....
Using pdk 2.2.0
Using pdksync 0.5.0 from https://github.com/puppetlabs/pdksync (at master@3b17e32)
Bundler attempted to update pdksync but its version stayed the same
Bundle updated!

Expected Behavior

have the latest released version installed (v0.6.0)

Steps to Reproduce

Steps to reproduce the behavior:

follow the How It Works Tutorial

Environment

Version ruby 2.7.0
Platform ubuntu 20.04

Run in parallel

Updating a single module takes several minutes to complete. I would like to see this process use threads to speedup the update job.

concurrent-ruby would make it easier to work with threading models. If using AASM library for #107 concurrent-ruby would be available as a dependency.

pdksync exits 0 despite fatal errors being logged

Describe the Bug

There seem to be lots of scenarios where an error running pdksync still results in the rake task exiting with a success return code.

eg. Here I'm trying to run pdksync in a gitlab pipeline. Despite my pipeline not working correctly yet, the job passes as bundle exec rake pdksync must have returned a zero exit code.

FATAL - PdkSync: Unable to run `pdk update`: pdk (INFO): Updating example-mylib using the template at https://gitlab.com/example/puppet-platform/pdk/pdk-templates.git, from tags/latest to latest@1b549a0
pdk (FATAL): Fetching https://gitlab.com/example/puppet-platform/puppet-lint-plugins/roles_and_profiles-check.git
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
Git error: command `git clone
'https://gitlab.com/example/puppet-platform/puppet-lint-plugins/roles_and_profiles-check.git'
"/root/.pdk/cache/ruby/2.7.0/cache/bundler/git/roles_and_profiles-check-5196364372340f4d47abfd168675b7421bd0362c"
--bare --no-hardlinks --quiet` in directory
/builds/example/puppet-platform/pdk/pdksync-config/modules_pdksync/mylib
has failed.
pdk (FATAL): Unable to resolve default Gemfile dependencies.
Cleaning up project directory and file based variables
Job succeeded

Expected Behavior

If log messages at level FATAL (and maybe ERROR too??) are logged, the whole task should probably exit with a non zero code.

allow hooks to be called

Use Case

I need to generate a .fixtures file from the Puppetfile found in the module (among other tasks). I would like a script to be called post pdk update or convert via a hook method that calls a script I define in the configuration.

Example workflow:

rake pdksync
pdk update
pdksync calls hooks
hook calls script like generate_fixtures.rb
files are committed and submited in PR.

Describe the Solution You Would Like

A post pdk update completion hook that is called before a commit is made

Describe Alternatives You've Considered

Have to create manually

Rake tasks are missing usage info

Describe the Bug

Running bundle exec rake -T produces output that is of very little use

Expected Behavior

I expect the output of bundle exec rake -T to be well formatted and readable. I also expect it to tell me what each task does.

Steps to Reproduce

╔ ☕️  beanm1:~/repos/dio-pdksync (main ✘) ✭
╚ᐅ bundle exec rake -T
rake gem_testing[additional_title,gem_to_test,gem_line,gem_sha_finder,gem_sha_replacer,gem_version_finder,gem_version_replacer,gem_branch_finder,gem_branch_replacer]  # ...
rake git:clean_branches[branch_name]                                                                                                                                   # ...
rake git:clone_gem[gem_name]                                                                                                                                           # ...
rake git:clone_managed_modules                                                                                                                                         # ...
rake git:create_commit[branch_name,commit_message]                                                                                                                     # ...
rake git:create_pr[pr_title,label]                                                                                                                                     # ...
rake git:push                                                                                                                                                          # ...
rake pdksync[additional_title]                                                                                                                                         # ...
rake pdksync:add_platform_to_metadata[os,version]                                                                                                                      # ...
rake pdksync:add_provision_list[key,provisioner,images]                                                                                                                # ...
rake pdksync:fetch_test_results_locally                                                                                                                                # ...
rake pdksync:gem_file_update[gem_to_test,gem_line,gem_sha_finder,gem_sha_replacer,gem_version_finder,gem_version_replacer,gem_branch_finder,gem_branch_replacer]       # ...
rake pdksync:generate_vmpooler_release_checks[puppet_version]                                                                                                          # ...
rake pdksync:multi_gem_testing[gem_name,version_file,build_gem,gem_path,gemfury_username]                                                                              # ...
rake pdksync:multigem_file_update[gem_name,gemfury_username]                                                                                                           # ...
rake pdksync:normalize_metadata_supported_platforms                                                                                                                    # ...
rake pdksync:pdk_convert                                                                                                                                               # ...
rake pdksync:pdk_validate                                                                                                                                              # ...
rake pdksync:remove_platform_from_metadata[os,version]                                                                                                                 # ...
rake pdksync:run_a_command[command,option]                                                                                                                             # ...
rake pdksync:run_tests_jenkins[jenkins_server_url,github_branch,test_framework,github_user]                                                                            # ...
rake pdksync:run_tests_locally[provision_type,puppet_collection]                                                                                                       # ...
rake pdksync:show_config                                                                                                                                               # ...
rake pdksync:test_results_jenkins[jenkins_server_url]                                                                                                                  # ...
rake pdksync:update_os_support                                                                                                                                         # ...
rake pdksync:update_requirements[name,key,value]                                                                                                                       # ...

In a terminal that is not absolutely huge, the text output above ends up looking like this:

Environment

Version 0.6.0
Platform macOS

Describe GITHUB_TOKEN

Describe the Change You Would Like

Would it be possible to give an quick guide of where to get a github token ( eg which type ) and what is the minimum permissions it needs.

Add a finite state machine

Use Case

Usage of a state machine allows us to model the workflow better and show how events flow for a particular action. Additionally, this would keep unwanted events from occurring and future additions to be easily added.

Describe the Solution You Would Like

I would like to see some clear definitions of defined states that pdksync can be in. Currently, it looks like

setting_up_pdk
cloning modules
running_pdk_update
running_pdk_convert
validating_module
creating_branch
creating_commit
pushing_branch
creating_pr
notifying_group
cleaning_up

Each of these states would call actions like create_commit or create_pr. A failure in a states forces a secondary run to know try and recreate things.

The state machine would also provide an easy way to log output during a transition from one state to the next. Essentially this is already be done in the form of puts statement and sequential line.

Below are two libraries I like to use:
https://github.com/aasm/aasm (pulls in ruby-concurrent)
https://github.com/geekq/workflow (no external libs)

Overall, this would greatly cleanup the pdksync.rb file which currently contains a method called "main" with 140 lines of code.

pdksync:pdk_convert errors on every module

Describe the Bug

I get this for every module in my managed modules list:

╔ ☕️  beanm1:~/repos/dio-pdksync (main ✘) ✭
╚ᐅ bundle exec rake pdksync:pdk_convert
INFO - PdkSync: ploperations-account,
INFO - PdkSync:

------------Files to be added-----------
/Users/geneliverman/repos/dio-pdksync/modules_pdksync/ploperations-account/hiera.yaml
/Users/geneliverman/repos/dio-pdksync/modules_pdksync/ploperations-account/data/common.yaml

----------Files to be modified----------
/Users/geneliverman/repos/dio-pdksync/modules_pdksync/ploperations-account/metadata.json

----------------------------------------

You can find a report of differences in convert_report.txt.



ERROR - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk convert --force --template-url=https://github.com/puppetlabs/pdk-templates.git --template-ref=2.2.0': pdk (FATAL):
/opt/puppetlabs/pdk/private/ruby/2.5.9/bin/ruby: invalid option -:  (-h will show valid options) (RuntimeError)

pdk (FATAL): Unable to resolve Gemfile dependencies.

INFO - PdkSync: converted

Expected Behavior

I expect it to successfully run pdk convert on each module.

Environment

Version 0.6.0

uninitialized constant PDK::VERSION (NameError) running rake task

Describe the Bug

Attempting to run any pdk command using bundle exec rake pdksync:* results in:

(FAILURE) Unable to run `pdk update`: /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/pdk-1.14.0/lib/pdk/cli/update.rb:28:in `block (2 levels) in <module:CLI>': uninitialized constant PDK::VERSION (NameError)
        from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/cri-2.15.9/lib/cri/command.rb:360:in `run_this'
        from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/cri-2.15.9/lib/cri/command.rb:296:in `run'
        from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/cri-2.15.9/lib/cri/command.rb:314:in `run'
        from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/pdk-1.14.0/lib/pdk/cli.rb:56:in `run'
        from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/pdk-1.14.0/exe/pdk:6:in `<top (required)>'
        from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/bin/pdk:23:in `load'
        from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/bin/pdk:23:in `<main>'

Expected Behavior

To actually run the PDK command

Steps to Reproduce

I have been reproducing this locally as follows:

docker run -it --rm -v ${pwd}:/pdk ruby:2.4.1 bash

Inside the container setting up my ssh key and then:

bundle install --path .bundle/gems/
bundle exec rake pdksync

Environment

latest gem (0.5.0) rather than github location
Platform - Docker on windows running ruby:2.4.0

Additional Context

My Gemfile contains just

gem 'pdksync'
gem 'rake'

and my Rakefile contains just

require 'pdksync/rake_tasks'

Just to note I love that this exists and will be very pleased when I can get it working. Thanks!

Allow management of arbitrary files like modulesync does

PDKSync should allow for this as it is a function of modulesync that I no longer have access to due to the PDK also using a file names .sync.yaml for its configuration.

I use this functionality to maintain a templetized version of files not edited by the PDK.

_{Sent with GitHawk}

Add proper logger

A proper logger allows the user to select the amount of output dumped to STDOUT. Currently, the code is littered with a bunch of puts statements when it should use the ruby logger class for this.

Add a new logger class that wraps the Ruby logger
Replace all puts statements with logger.info, logger.warn logger.crit, logger.debug statements depending on the severity.

pdksync:show_config has unexpected output

Describe the Bug

I think some kind of terminal formatting attempt failed here and results in nearly illegible output:

Expected Behavior

Easy to read output

Environment

Version 0.6.0
Platform macOS using zsh

Document *_platform_to_metadata

Describe the Change You Would Like

Document in the README:

add_platform_to_metadata
remove_platform_to_metadata

puppetlabs / pdksync Goto Github PK

pdksync's Issues

Describe the Bug

Expected Behavior

Use Case

Describe the Solution You Would Like

Describe Alternatives You've Considered

Describe the Bug

Describe the Change You Would Like

Describe the Bug

Expected Behavior

Environment

Describe the Bug

Expected Behavior

Steps to Reproduce

Environment

Describe the Bug

Expected Behavior

Use Case

Describe the Solution You Would Like

Describe Alternatives You've Considered

Describe the Bug

Expected Behavior

Steps to Reproduce

Environment

Describe the Change You Would Like

Use Case

Describe the Solution You Would Like

Describe the Bug

Expected Behavior

Environment

Describe the Bug

Expected Behavior

Steps to Reproduce

Environment

Additional Context

Describe the Bug

Expected Behavior

Environment

Describe the Change You Would Like

Recommend Projects

Recommend Topics

Recommend Org