Merge appup.pre.src && appup.post.src files into the appup file after .appup is generated by 'appup generate' about rebar3_appup_plugin HOT 10 CLOSED

The idea sounds interesting, since a lot of the functionality is already present in this plugin and some of it can be re-used i think this validation aspect of release upgrades could also live in this plugin. Can you open a new issue where we can continue the discussion? Meanwhile i'll go about squashing and merging this one. thanks!

from rebar3_appup_plugin.

This is actually a good question why this functionality is needed since appup generate should create the appup file with all the required contents for the upgrade.

However it can't figure out all needed steps automatically. For example as part of the upgrade we have to call some conversion on a riak bucket or some other tasks which could be called manually before/after calling the upgrade.

If the required steps are the same for all version upgrades those steps can be automatically merged this way into the .appup file from those other two .pre.appup.src/.post.appup.src files.

You are right, this can be done by an escript as well, which calls the upgrade and also calls the other manual steps before/after the upgrade, but it would be probably simpler to give the user this option.

Also I know generating the appup file from the .appup.src with rebar3 appup compile is similar, but after calling rebar3 appup generate those contents generated by appup compile are lost. Basically this feature is only combining appup compile and appup generate together.

from rebar3_appup_plugin.

Out of curiosity, don't you expect the appup.pre file upgrades to contain the opposite steps to the appup.post file downgrades?

Since this is moving from "one file is all ordered" into " each app now has two priorities (pre/post)", I have to ask the following questions to dig into the design approach: would there be a need for more priorities than just two? How would you reconcile the presence of .pre and/or .post files with an existing appup file? Is this a third priority or is there anything else that should happen?

from rebar3_appup_plugin.

No, the only thing we need is just to merge the .pre.appup.src/.post.appup.src files with the generated .appup file during calling appup generate. The .pre and .post steps are not opposite steps. Some steps should be called before the generated commands, and some steps should be called after them.

Let's assume the following. A module deals with CRDT maps stored in riak. In the next release a new field is added to the map, and the code will fail if the field is not there. In this case the database must be converted first (add the new field to the db), then the module can be reloaded. On the contrary if the field is not needed anymore then the module should be updated first and after that the db conversion can be called.

In the first case the conversion is called from the .pre steps and in the second case it can be called from the .post file.

An other example would be to notify a connected java application about the start and stop of an upgrade. The notification about the start of the upgrade would go into the .pre file, and the stop into the .post.appup.src.

If the .pre and/or .post files don't exist that part of the .appup after merging would be empty.
If none of them exists both the .pre and .post part will be empty, so the final .appup file will contain exactly the same steps what was there in .appup before the merge, if only one them is not there only the corresponding part should be empty.

For example:

%% appup generated for relapp by rebar3_appup_plugin (2018/01/03 09:06:17)
{ "1.0.16",
    [{ "1.0.15",
        [
          %% comes from the .pre.appup.src if the file exists
          {apply, io, format, ["Upgrading from 1.0.15 to 1.0.16"]},
          %% contents from  the original appup files
          {load_module,relapp_m1,brutal_purge,brutal_purge,[]},
          %% .post.appup.src does not exist, so this part is empty
        ] }],
     [{ "1.0.15",
        [
           %% comes from the .pre.appup.src if the file exists
           {apply, io, format, ["Downgrading from 1.0.16 to 1.0.15"]},
           %% contents from  the original appup files
           {load_module,relapp_m1,brutal_purge,brutal_purge,[]},
           %% .post.appup.src does not exist, so this part is empty
      ] }]
}.

from rebar3_appup_plugin.

I think this could be a valuable addition to the plugin, @tothlac would you be interested in proposing a PR with the change? i'd be happy to help with tests or debugging

from rebar3_appup_plugin.

Sure, I will try to implement it.

from rebar3_appup_plugin.

I've created the PR: #30
I wanted to add some eunit tests, and added a new release for relapp1 in order to test the feature.
Maybe we need a bit more rebar_api:debug calls just to make debugging bit easier.

from rebar3_appup_plugin.

Thanks for reviewing the PR. Those changes are fine. In my opinion it can be squashed.

There is an other requirement we are thinking about.

In a lot of companies there are checks implemented validating whether an upgrade between two specific versions are fine. For example:

calls between applications follow the start order specified in the .rel file. It should not be possible for an application to call into an application which is started later. There can be exceptions though which can be described using exception rules.

It should be possible to define the interface modules of an application. The checks should validate if only those modules are called from outside. It should also ensure that no interface functions between two releases are deleted. Think about possible rpc calls from an other node still running an old release.

we've just started thinking about other possible useful checks. It looks like it is possible to implement these checks in a rebar plugin, which will be very similar to the appup plugin. Inside appup it could be a command 'appup check-release'. The list of actual checks could be described in a list containing callback module names, so it would be actually configurable what kind of checks the command will be running.

Do you think that such a feature would fit the functionalities of the appup plugin and would it be interesting for you?

from rebar3_appup_plugin.

closing via #31

from rebar3_appup_plugin.

Thanks for the quick update. I will create a new ticket for the other feature.

from rebar3_appup_plugin.