Git Product home page Git Product logo

lapidist's Introduction

Lapidist: synchronized gem releases

tests

Scope

Certain projects are dependent on the interoperability of a web of gems. Changes to dependent gems, especially ones closer to the root, can cause many of the downstream gems to need upgrading.

For example, a release of Metanorma spans over 50 gems. Changes to an underlying gem like metanorma will require all downstream gems to re-run all tests.

This software is created to simplify the release procedure of dependent gems. It provides methods to:

  • Define this dependent web of gems

  • Synchronized gem testing among this dependent web of gems

  • Synchronized release of these gems (with version increments)

Terms and definitions

root gem

a gem that is no other software is dependent on

dependent gem

a gem that other gems depend on

leaf gem

a gem that does not depend on any other gem in the defined gem web

gem net

a coherent set of gems that needs to interoperate and that requires synchronized releasing

caught gem

gem under the release control of the releaser

release repository

git repository or directory used for the releasing process

Requirements

  • hub must be installed

Features

  1. Integration testing

    • all downstream gems can test against the work-in-progress upstream gems

    • management of Gemfile and gemspec version control

  2. Release process

    • automatic version bumps for gems

    • synchronized pushes of feature branches across gems

    • creation of PRs across gems

    • synchronized release of gems

Flow

Start release process

It all starts with the start subcommand.

Note
If you wish to run the Git version of lapidist, use bundler and a Gemfile; you will want to replace the command lapidist with lapidist for all steps below.

Go to the release repository:

lapidist start -p .. -b 'feature/some-feature' -g isodoc,metanorma-iso,metanorma,metanorma-cli
-p

path to .gitmodules file

-b

branch name for feature, the same name will be used across all gems

-g

comma-separated list of gems which will be included in feature (if missed default all gems included)

Finally this command will ask you to push changes to remote.

Under the hood this command does the following:

  • use bundler to set local path to gems

  • generate Gemfile.devel to keep CI “green”

Updating code and integrated testing

  1. Go to your release repository.

Note
If you have missed some gem, you can run lapidist start …​ for it separately but keep branch name the same.
  1. Do the necessary work in the submodule’d gem in the gem net (within the release repository) i.e. isodoc or metanorma-iso.

  2. Run a script that performs tests on all the gems at once using the newly created feature branches:

lapidist rake -p .. -b 'feature/some-feature'
  1. If you want Travis and Appveyor to test for you, and/or if you want to create PRs, run a script that pushes gems with the feature branches:

lapidist push -p .. -b 'feature/some-feature'
  1. When all the gems pass locally, create PRs for feature/some-featuremaster

  2. Before merge finish feature with:

lapidist finish -p .. -b 'feature/some-feature'
  1. (alt) this will lead for short time breaks on CI till gems will be actually released

    1. Merge PRs by hand or by script (into master or a release branch)

Releasing

When all PRs merged, run a script to:

  1. Re-run tests against master (or release) branch to ensure integration of PRs has not introduced problems, and that any external gem releases you are not responsible for have been taken into account (e.g. metanorma stack depending on relaton stack). This test set will involve a new integration testing branch, forked from master (or release):

lapidist start -p .. -b 'test/date' -g isodoc,metanorma-iso,metanorma,metanorma-cli
lapidist rake -p .. -b 'test/date'
lapidist push -p .. -b 'test/date'
lapidist finish -p .. -b 'test/date'

Any needed fixes will be against this new integration testing branch, and would trigger new PRs.

Once integration testing is done, run a script to:

  1. Update gemspecs dependencies version of recently released gems

  2. Run this to release passed gems:

lapidist release -p .. -v patch -g isodoc,metanorma-iso,metanorma,metanorma-cli
  1. The release command will do:

    1. version bump

    2. commit bump

    3. do tag

    4. push tag to git

    5. build gem

    6. release gem to rubygems.org

Ideally, we want to update the leaf gems first, then the immediately dependent gems, and so forth towards the root gems to ensure that the builds always pass.

Installation

$ gem install lapidist

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/metanorma/lapidist. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

Code of Conduct

Everyone interacting in the Lapidist project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.

Origin of the name

A “lapidist” is someone who cuts, polishes, or engraves gems. Geddit?

Note
The noun form of “lapidary” is now more popular than “lapidist” in this meaning, but the current usage also conflates the “person” (“the lapidary”), and the relation to gem processing (“a lapidary engraving”).

Release flow diagrams

Slide1
Slide2
Slide3
Slide4
Slide5
Slide6

lapidist's People

Contributors

camobap avatar dependabot[bot] avatar opoudjis avatar ronaldtse avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lapidist's Issues

Support YAML configuration

A YAML file called "lapidist.yaml" can be placed in a release repo:

  • to define which gems need to integrate with which ones (i.e. dependency graph)
  • potentially auto-detect the dependencies
  • Custom gem order #4
  • each gem:
    • define gem name
    • define gem source code location (and branches)

In the README there's an example of:
-g isodoc,metanorma-iso,metanorma,metanorma-cli
which is really long to type. It would be much better if either the system auto-detects the changes that need to be released, or allow manual definition in the YAML file.

lapidist run as UTF-8

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

is set on Gemfile. However, lapidist is still running with default encoding of ASCII. As a result, sometimes SimpleCov crashes; e.g. for metanorma-nist:

Coverage report generated for RSpec to /Users/nickn/Documents/Arbeit/upwork/ribose/metanorma-nist/coverage. 1447 / 1608 LOC (89.99%) covered.
SimpleCov failed with exit 1Traceback (most recent call last):
	31: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/exe/rspec:4:in `<main>'
	30: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/runner.rb:45:in `invoke'
	29: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/runner.rb:71:in `run'
	28: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/runner.rb:87:in `run'
	27: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/runner.rb:110:in `run_specs'
	26: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/reporter.rb:76:in `report'
	25: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/reporter.rb:166:in `finish'
	24: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/reporter.rb:186:in `close_after'
	23: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/reporter.rb:170:in `block in finish'
	22: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/reporter.rb:200:in `notify'
	21: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/reporter.rb:200:in `each'
	20: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/reporter.rb:201:in `block in notify'
	19: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/base_text_formatter.rb:32:in `dump_failures'
	18: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/notifications.rb:113:in `fully_formatted_failed_examples'
	17: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/notifications.rb:113:in `each_with_index'
	16: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/notifications.rb:113:in `each'
	15: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/notifications.rb:114:in `block in fully_formatted_failed_examples'
	14: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/notifications.rb:200:in `fully_formatted'
	13: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:78:in `fully_formatted'
	12: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:86:in `fully_formatted_lines'
	11: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:240:in `formatted_message_and_backtrace'
	10: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:34:in `colorized_message_lines'
	 9: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:149:in `failure_lines'
	 8: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:149:in `tap'
	 7: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:150:in `block in failure_lines'
	 6: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:163:in `failure_slash_error_lines'
	 5: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/exception_presenter.rb:218:in `read_failed_lines'
	 4: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/snippet_extractor.rb:30:in `extract_expression_lines_at'
	 3: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/formatters/snippet_extractor.rb:18:in `source_from_file'
	 2: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-core-3.8.2/lib/rspec/core/world.rb:150:in `source_from_file'
	 1: from /Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-support-3.8.2/lib/rspec/support/source.rb:12:in `from_file'
/Users/nickn/.rvm/gems/ruby-2.5.1/gems/rspec-support-3.8.2/lib/rspec/support/source.rb:12:in `read': "\xC2" on US-ASCII (Encoding::InvalidByteSequenceError)

The crash happens because the output of rspec is citing native UTF-8 characters in the diffs between expected and actual output (such as em-dashes), and the output encoding is clearly still set to ASCII.

How to pass Travis and Appveyor tests?

Tell me please what did I wrong. I've:

  • created a feature branch new_xml_convertor in all relaton gems
  • changed relaton-bib and relaton-cli
  • committed the changed gems
  • pushed all new branches

Then I tried to run in relaton-release dir:

bundle exec lapidist rake -p .gitmodules -b 'new_xml_convertor'

but nothing happened.
Then I made PRs in changed gems (relaton-bib and relaton-cli) and tried to run:

bundle exec lapidist finish -p .gitmodules -b 'new_xml_convertor'

and got:

[?] please confirm that all PRs for [new_xml_convertor] are 'green' - press 'y' to continue:

Does it mean all Travis and Appveyor test should pass? Hmm... If so how it could be done? The relaton-bib doesn't depend on over relaton gems and pass Travis/Appveyor tests. But relaton-cli depends on relaton-bib and can't pass these tests before the dependence is released, because it needs the changes to pass the tests. Am I wrong?

Implement Lapidist YAML to determine which gems to release and for dependency tracking

I am getting errors in Travis metanorma-standoc which I am not getting in local compilation: the Travis compilation is ignoring post-release updates to relaton-iso-bib: https://github.com/relaton/relaton-iso-bib/releases

I note from the Gemfile.devel being run in lapidist the following:

gem 'isodoc', :github => '/isodoc', :branch => 'test/20190923'
gem 'relaton', :github => '/relaton', :branch => 'test/20190923'
gem 'html2doc', :github => '/html2doc', :branch => 'test/20190923'
gem 'metanorma', :github => '/metanorma', :branch => 'test/20190923'

This seems to me to indicate that the change in relaton-iso-bib is not being inherited through to the Travis run.

With these Gemfile.devel github pulls, they have to explore the inheritance trail exhaustively: if relaton is pulled, the entire stack of relaton needs to be pulled. The same applies for any other gems. This is the Gemfile.devel for metanorma-gb

gem 'metanorma-iso', :github => '/metanorma-iso', :branch => 'test/20190923'
gem 'isodoc', :github => '/isodoc', :branch => 'test/20190923'
gem 'metanorma', :github => '/metanorma', :branch => 'test/20190923'

metanorma-iso inherits from metanorma-standoc. Therefore, metanorma-standoc must also be pulled from github.

Lapidist delete branches

lapidist -start pushes the test branch. lapidist -finish does not delete the test branch. This is polluting the branch space, especially if no changes have been pushed. There needs to be a command to delete a nominated branch against nominated gems as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.