puppetlabs / pdksync Goto Github PK
View Code? Open in Web Editor NEWManage large sets of modules with automation built on top of PDK
License: Apache License 2.0
Manage large sets of modules with automation built on top of PDK
License: Apache License 2.0
Error logging code explodes.
NoMethodError: undefined method `error' for PdkSync::Logger:Class
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync/utils.rb:146:in `rescue in push_staged_files'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync/utils.rb:143:in `push_staged_files'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync.rb:247:in `block in main'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync.rb:158:in `each'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync.rb:158:in `main'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/pdksync-0.6.0/lib/pdksync/rake_tasks.rb:10:in `block in <top (required)>'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Error to be logged correctly
pdksync provides a method to "globally" manage requirement, it would be great to be able to manage dependencies in a similar way.
bundle exec rake 'pdksync:update_dependencies[puppetlabs/stdlib,version_requirement,>= 6.0.0 < 9.0.0]'
extending the update_requirements task
There appears to be a regression in the recent versions of the git
dependency. If I don't pin to 1.14.0
I get the following failure in my pdksync pipeline.
Looks like the checkout command is invalid?? (If I run the git command manually and don't throw away the stderr, I get error: pathspec 'pdksync_pdksync_tags/hl-latest-0-g76a98ec' did not match any file(s) known to git
)
I'm using gitlab
instead of the more common github
, but as this is a failure with a local git command, I think this isn't relevant.
$ bundle exec rake pdksync
To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
INFO - PdkSync: Commit branch_name=pdksync_{ref} commit_message=pdksync_{ref}
INFO - PdkSync: PR title = pdksync_{ref}
INFO - PdkSync: mymodule,
INFO - PdkSync: delete module directory
INFO - PdkSync: cloned
WARN - PdkSync: (WARNING) Using pdk on PATH not '/opt/puppetlabs/pdk/bin/pdk'
INFO - PdkSync: pdk update
rake aborted!
Git::FailedError: git '--git-dir=/builds/example/puppet-platform/pdk/pdksync-config/modules_pdksync/mymodule/.git' '--work-tree=/builds/example/puppet-platform/pdk/pdksync-config/modules_pdksync/mymodule' '-c' 'core.quotePath=true' '-c' 'color.ui=false' 'checkout' 'pdksync_pdksync_tags/hl-latest-0-g76a98ec' 2>&1
status: pid 1843 exit 1
stderr: ""
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/lib.rb:1215:in `command'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/lib.rb:834:in `checkout'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/base.rb:359:in `checkout'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/gems/git-1.17.0/lib/git/branch.rb:28:in `checkout'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/bundler/gems/pdksync-25b4d3031c11/lib/pdksync/utils.rb:329:in `checkout_branch'
/builds/example/puppet-platform/pdk/pdksync-config/vendor/bundle/ruby/2.7.0/bundler/gems/pdksync-25b4d3031c11/lib/pdksync/utils.rb:195:in `create_commit'
GITHUB_TOKEN shouldn't be necessary if using GITLAB_TOKEN
❯ export GITLAB__TOKEN=XXXXXXXXXXXXX
❯ bundle exec rake git:clone_managed_modules
rake aborted!
ArgumentError: Git platform access token for Github not set - use 'export GITHUB_TOKEN="<your token>"' to set
I don't think the FATAL output here is actually correct:
❯ bundle exec rake 'pdksync:pdk_validate'
INFO - PdkSync: puppet-profile_base,
INFO - PdkSync:
FATAL - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk validate -a': pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.4.1
pdk (INFO): Running all available validators...
pdk (INFO): Validator 'puppet-epp' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-profile_base'. No files matching '["**/*.epp"]' found to validate.
pdk (INFO): Validator 'task-metadata-lint' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-profile_base'. No files matching '["tasks/*.json"]' found to validate.
INFO - PdkSync: validated
INFO - PdkSync: puppet-git_version,
INFO - PdkSync:
FATAL - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk validate -a': pdk (WARN): This module is compatible with an older version of PDK. Run `pdk update` to update it to your version of PDK.
pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.4.1
pdk (INFO): Running all available validators...
pdk (INFO): Validator 'puppet-syntax' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-git_version'. No files matching '["**/*.pp"]' found to validate.
pdk (INFO): Validator 'puppet-lint' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-git_version'. No files matching '["**/*.pp"]' found to validate.
pdk (INFO): Validator 'puppet-epp' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-git_version'. No files matching '["**/*.epp"]' found to validate.
INFO - PdkSync: validated
INFO - PdkSync: puppet-monitoring_scripts,
INFO - PdkSync:
FATAL - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk validate -a': pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.4.1
pdk (INFO): Running all available validators...
pdk (INFO): Validator 'puppet-epp' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-monitoring_scripts'. No files matching '["**/*.epp"]' found to validate.
pdk (INFO): Validator 'task-name' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-monitoring_scripts'. No files matching '["tasks/**/*"]' found to validate.
pdk (INFO): Validator 'task-metadata-lint' skipped for '/Users/cdenneen/src/gitlab/puppet/pdksync/modules_pdksync/puppet-monitoring_scripts'. No files matching '["tasks/*.json"]' found to validate.
INFO - PdkSync: validated
A clear and concise description of what you would like changed.
README.md references the pdksync-externalisation-fixes
branch... there is no such branch. Please update the section about what to put in your Gemfile.
The two warnings below are repeated many many times while running the task
╔ ☕️ beanm1:~/repos/dio-pdksync (main ✘) ✭
╚ᐅ bundle exec rake git:clone_managed_modules
INFO - PdkSync: ploperations-account,
WARNING: `Faraday::Connection#authorization` is deprecated; it will be removed in version 2.0.
While initializing your connection, use `#request(:authorization, ...)` instead.
See https://lostisland.github.io/faraday/middleware/authentication for more usage info.
/Users/geneliverman/repos/dio-pdksync/.bundle/gems/ruby/2.7.0/gems/json_pure-2.1.0/lib/json/common.rb:156: warning: Using the last argument as keyword parameters is deprecated
I expect to run the task and not see a bunch of notices about things that need to be addressed
while following the pdksync setup using the README in the repository, there are 2 problems
attachmentgenie-pdksync ❯ bundle install --path .bundle/gems/ main
[DEPRECATED] The--path
flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please usebundle config set path '.bundle/gems/'
, and stop using this flag
Fetching https://github.com/puppetlabs/pdksync
fatal: Needed a single revision
Git error: commandgit rev-parse --verify pdksync-externalisation-fixes
in directory
for which i raised a PR (#156), the second problem, is that after fixing the gem file, bundler install the 0.5.0 version instead of the latest released version.
attachmentgenie-pdksync ❯ bundle install --path .bundle/gems/
....
Using pdk 2.2.0
Using pdksync 0.5.0 from https://github.com/puppetlabs/pdksync (at master@3b17e32)
Bundler attempted to update pdksync but its version stayed the same
Bundle updated!
have the latest released version installed (v0.6.0)
Steps to reproduce the behavior:
Updating a single module takes several minutes to complete. I would like to see this process use threads to speedup the update job.
concurrent-ruby would make it easier to work with threading models. If using AASM library for #107 concurrent-ruby would be available as a dependency.
There seem to be lots of scenarios where an error running pdksync still results in the rake task exiting with a success return code.
eg. Here I'm trying to run pdksync in a gitlab pipeline. Despite my pipeline not working correctly yet, the job passes as bundle exec rake pdksync
must have returned a zero exit code.
FATAL - PdkSync: Unable to run `pdk update`: pdk (INFO): Updating example-mylib using the template at https://gitlab.com/example/puppet-platform/pdk/pdk-templates.git, from tags/latest to latest@1b549a0
pdk (FATAL): Fetching https://gitlab.com/example/puppet-platform/puppet-lint-plugins/roles_and_profiles-check.git
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.com/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
Git error: command `git clone
'https://gitlab.com/example/puppet-platform/puppet-lint-plugins/roles_and_profiles-check.git'
"/root/.pdk/cache/ruby/2.7.0/cache/bundler/git/roles_and_profiles-check-5196364372340f4d47abfd168675b7421bd0362c"
--bare --no-hardlinks --quiet` in directory
/builds/example/puppet-platform/pdk/pdksync-config/modules_pdksync/mylib
has failed.
pdk (FATAL): Unable to resolve default Gemfile dependencies.
Cleaning up project directory and file based variables
Job succeeded
If log messages at level FATAL
(and maybe ERROR
too??) are logged, the whole task should probably exit with a non zero code.
I need to generate a .fixtures file from the Puppetfile found in the module (among other tasks). I would like a script to be called post pdk update or convert via a hook method that calls a script I define in the configuration.
Example workflow:
generate_fixtures.rb
A post pdk update completion hook that is called before a commit is made
Have to create manually
Running bundle exec rake -T
produces output that is of very little use
I expect the output of bundle exec rake -T
to be well formatted and readable. I also expect it to tell me what each task does.
╔ ☕️ beanm1:~/repos/dio-pdksync (main ✘) ✭
╚ᐅ bundle exec rake -T
rake gem_testing[additional_title,gem_to_test,gem_line,gem_sha_finder,gem_sha_replacer,gem_version_finder,gem_version_replacer,gem_branch_finder,gem_branch_replacer] # ...
rake git:clean_branches[branch_name] # ...
rake git:clone_gem[gem_name] # ...
rake git:clone_managed_modules # ...
rake git:create_commit[branch_name,commit_message] # ...
rake git:create_pr[pr_title,label] # ...
rake git:push # ...
rake pdksync[additional_title] # ...
rake pdksync:add_platform_to_metadata[os,version] # ...
rake pdksync:add_provision_list[key,provisioner,images] # ...
rake pdksync:fetch_test_results_locally # ...
rake pdksync:gem_file_update[gem_to_test,gem_line,gem_sha_finder,gem_sha_replacer,gem_version_finder,gem_version_replacer,gem_branch_finder,gem_branch_replacer] # ...
rake pdksync:generate_vmpooler_release_checks[puppet_version] # ...
rake pdksync:multi_gem_testing[gem_name,version_file,build_gem,gem_path,gemfury_username] # ...
rake pdksync:multigem_file_update[gem_name,gemfury_username] # ...
rake pdksync:normalize_metadata_supported_platforms # ...
rake pdksync:pdk_convert # ...
rake pdksync:pdk_validate # ...
rake pdksync:remove_platform_from_metadata[os,version] # ...
rake pdksync:run_a_command[command,option] # ...
rake pdksync:run_tests_jenkins[jenkins_server_url,github_branch,test_framework,github_user] # ...
rake pdksync:run_tests_locally[provision_type,puppet_collection] # ...
rake pdksync:show_config # ...
rake pdksync:test_results_jenkins[jenkins_server_url] # ...
rake pdksync:update_os_support # ...
rake pdksync:update_requirements[name,key,value] # ...
In a terminal that is not absolutely huge, the text output above ends up looking like this:
Would it be possible to give an quick guide of where to get a github token ( eg which type ) and what is the minimum permissions it needs.
Usage of a state machine allows us to model the workflow better and show how events flow for a particular action. Additionally, this would keep unwanted events from occurring and future additions to be easily added.
I would like to see some clear definitions of defined states that pdksync can be in. Currently, it looks like
Each of these states would call actions like create_commit
or create_pr
. A failure in a states forces a secondary run to know try and recreate things.
The state machine would also provide an easy way to log output during a transition from one state to the next. Essentially this is already be done in the form of puts statement and sequential line.
Below are two libraries I like to use:
https://github.com/aasm/aasm (pulls in ruby-concurrent)
https://github.com/geekq/workflow (no external libs)
Overall, this would greatly cleanup the pdksync.rb file which currently contains a method called "main" with 140 lines of code.
I get this for every module in my managed modules list:
╔ ☕️ beanm1:~/repos/dio-pdksync (main ✘) ✭
╚ᐅ bundle exec rake pdksync:pdk_convert
INFO - PdkSync: ploperations-account,
INFO - PdkSync:
------------Files to be added-----------
/Users/geneliverman/repos/dio-pdksync/modules_pdksync/ploperations-account/hiera.yaml
/Users/geneliverman/repos/dio-pdksync/modules_pdksync/ploperations-account/data/common.yaml
----------Files to be modified----------
/Users/geneliverman/repos/dio-pdksync/modules_pdksync/ploperations-account/metadata.json
----------------------------------------
You can find a report of differences in convert_report.txt.
ERROR - PdkSync: Unable to run command '/opt/puppetlabs/pdk/bin/pdk convert --force --template-url=https://github.com/puppetlabs/pdk-templates.git --template-ref=2.2.0': pdk (FATAL):
/opt/puppetlabs/pdk/private/ruby/2.5.9/bin/ruby: invalid option -: (-h will show valid options) (RuntimeError)
pdk (FATAL): Unable to resolve Gemfile dependencies.
INFO - PdkSync: converted
I expect it to successfully run pdk convert
on each module.
Attempting to run any pdk command using bundle exec rake pdksync:* results in:
(FAILURE) Unable to run `pdk update`: /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/pdk-1.14.0/lib/pdk/cli/update.rb:28:in `block (2 levels) in <module:CLI>': uninitialized constant PDK::VERSION (NameError)
from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/cri-2.15.9/lib/cri/command.rb:360:in `run_this'
from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/cri-2.15.9/lib/cri/command.rb:296:in `run'
from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/cri-2.15.9/lib/cri/command.rb:314:in `run'
from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/pdk-1.14.0/lib/pdk/cli.rb:56:in `run'
from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/gems/pdk-1.14.0/exe/pdk:6:in `<top (required)>'
from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/bin/pdk:23:in `load'
from /pdk/puppet-module-sync/.bundle/gems/ruby/2.4.0/bin/pdk:23:in `<main>'
To actually run the PDK command
I have been reproducing this locally as follows:
docker run -it --rm -v ${pwd}:/pdk ruby:2.4.1 bash
Inside the container setting up my ssh key and then:
bundle install --path .bundle/gems/
bundle exec rake pdksync
My Gemfile contains just
gem 'pdksync'
gem 'rake'
and my Rakefile contains just
require 'pdksync/rake_tasks'
Just to note I love that this exists and will be very pleased when I can get it working. Thanks!
PDKSync should allow for this as it is a function of modulesync that I no longer have access to due to the PDK also using a file names .sync.yaml
for its configuration.
I use this functionality to maintain a templetized version of files not edited by the PDK.
Sent with GitHawk
A proper logger allows the user to select the amount of output dumped to STDOUT. Currently, the code is littered with a bunch of puts statements when it should use the ruby logger class for this.
Document in the README:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.