Git Product home page Git Product logo

rebar3_appup_plugin's Introduction

rebar3 appup plugin

Build Status hex.pm version gitter

A rebar3 plugin for handling release upgrades. Supports the following features:

  • Automatic generation of the .appup file containing instructions necessary to upgrade and downgrade from one release to the other. More info
  • Validation of any .appup.src files that might be present, these are scripts that can contain Erlang code. They are templated, evaluated and their results and written to an .appup file that is then moved to the target dir. More info
  • Handles any dependency .appup.src files maintained by your application. More info
  • Automatic code injection for gen_server state record conversion between versions. More info
  • Automatically generated module dependencies. More info

Demo

Generating a release upgrade

gif

Upgrading a release

gif

Build

$ rebar3 compile

Configure

Add the plugin to your project's rebar.config:

{plugins, [rebar3_appup_plugin]}.

and the provider hooks:

  {provider_hooks, [
      {pre, [{tar, {appup, tar}}]},
      {post, [{compile, {appup, compile}},
              {clean, {appup, clean}}]}
  ]}.

Application upgrade generation

By generating two releases, the one that is live right now and the one that we want to end up in. We can invoke the plugin and have it generate a special .appup file that OTP knows how to process in order to generate a release upgrade file (relup). This file contains low level Erlang VM instructions that will transition the Erlang application from one version to another without any downtime. More info.

Using an .appup.src file

You can generate the .appup file every time you pack a release with the rebar3 appup generate call. However when there is the need to add custom data or instructions to the .appup it's useful to have it in source control alongside your .app.src file. The .appup.src file can contain Erlang code and it's result should be a valid appup Erlang term. The plugin will look for any files ending in .appup.src, perform template variables substitution, evaluate them and have their results written to an .appup file that will then be used to generate the relup. More info.

Code Injection

When doing relups an issue that comes up often is the problem of ugprading the state record of your gen_server code. Most likely you have made some change that needs a new record structure alongside the code that handles it. OTP offers you the gen_server:code_change/3 call that enables state migration, this implies however that the new code must somehow know about the old record structure to migrate from. The plugin will automatically, by specifying a -state_record directive, inject the necessary code into the gen_server's .beam file that will take care of the migration. More info.

Automatically generated module dependencies

OTP allows the developer to manually specify the dependencies for each module in the .appup file, that is, which modules should be loaded before that one. The plugin takes care of this for you, by using xref it discovers which modules are being used and adds them to the relevant .appup entries. More info.

Command line options

    Appup generator
    Usage: rebar3 appup generate

      -p, --previous    location of the previous release
      -c, --current     location of the current release
      -t, --target_dir  target dir in which to generate the .appups to
      -t, --target_dir  target dir in which to generate the .appups to
      -g, --purge       per module purge instructions, format is
                          [default | Module]=Purge | PrePurge/PostPurge
                        where Purge, PrePurge, PostPurge is either soft or brutal. Module is the name of a module in the .appup file, default is a reserved name to apply purge options
                        to unspecified modules.
rebar3 appup generate [--previous /path/to/previous/release] 
[--previous_version version] [--current /path/to/current/release] 
[--target_dir /path/to/target/dir] [--purge ]
  • previous and current options should be full path names to the release (ie. the folder that contains lib/, releases/)
  • current is optional and defaults to _build/<profile>/rel
  • target_dir is optional and defaults to _build//rel//lib//ebin

Compile

rebar3 appup compile

Clean

rebar3 appup clean

Copyright and License

Copyright (c) 2016 Luis Rascão

rebar3_appup_plugin source code is licensed under Apache 2.0.

rebar3_appup_plugin's People

Contributors

benbro avatar choptastic avatar falkevik avatar getong avatar joaohf avatar kyle-neal avatar lrascao avatar soranoba avatar tothlac avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rebar3_appup_plugin's Issues

Appup file generated by rebar3_appup_generate contains all files if only the compile options change

The plugin uses beam_lib:cmp_dirs/2 to get the list of changed files. It works well if the beam files in the two releases
were compiled using the same erl_opts flag. Unfortunately for instance if I add option deterministic to erl_opts the plugin thinks
all files were changed.

Unfortunately, we have experienced this problem between two releases. Only two files have been changed but hundreds of files were added to the appup file.

Using a modified version of beam_lib:cmp_dirs which filters out some attributes from the abstract code should be able to fix this problem.

Can't make relup

===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: function_clause
===> Stack trace to the error location:
[{rebar3_appup_generate,select_behaviour,
[[gen_server,poolgirl_worker]],

Follow behaviour parents when generating appup files

My problem is the following:

As far as I know rebar3 appup generate currently generates {upgrade, Mod, ...} instructions into the .appup file only if the module has been changed and it implements an OTP behaviour.

I've implemented a behaviour, let's call it my_behaviour and the module using this behaviour my_behaviour_impl.erl.

The module my_behaviour implements gen_server, and MyBehaviourState being part of the gen_server's state is passed from my_behaviour.erl to the modules implementing this behaviour.

It would be a good feature if appup generate could automatically figure out that the behavioural parent of my_behaviour_impl.erl is my_behaviour.erl and because it implements an otp behaviour, it should automatically add {update, my_behaviour, ...} instructions to the generated .appup file. So the following instructions in upgrade/downgrade sections of the .appup file:

[
  {load_module, my_behaviour_impl, ...}
]

should be converted into:

[
  {upgrade, my_behaviour, ...}
  {load_module, my_behaviour_impl, ...}
]

Do you think that adding such a feature would be doable in rebar3_appup_generate.erl and also would it be a nice place for inserting these additional instructions into the .appup file?

appup compile generates "git" into the generated appup file

when calling appup release, which calls appup compile on one of our applications it generates

{"git", UpgradeInstructions, DowngradeInstructions} instead of putting the correct version number there. The version number comes from this line:

Context = [{"vsn", rebar_app_info:original_vsn(AppInfo)}],

It gives back an atom 'git' instead of the version.
so basically there is a bad version number coming from the rebar state.
I also see that during earlier when rebar_app_info:original_vsn/2 is called the Vsn still stores the correct value, but right after that this atom comes out from the state. I can't see who sets it.
Can it be a rebar bug, or maybe in this case the plugin should read this information from somewhere else?

Overrides for dependency appups

Dependencies very rarely include appups. It often doesn't really make sense because you can only provide an upgrade between specific versions (or a regex match) and the modules that need to be unloaded/reloaded can vary a lot between 1.0 -> current and 2.0 -> current, etc.

So, usually what you have to do if you're using release upgrades is: fork the dependency and add your own custom appup.

So I was very excited to find this plugin, however, it can struggle with generating a correct appup, especially with complex supervision trees, so it can't really be trusted in an automated CI build situation. It's fine for your own apps because you can just commit the appups into your own project, but not so for deps (without forking).

So, yeah, this is mainly an issue where releases and upgrades are built by a ci job, as if you're running a release manually you can obviously edit the generated appups before tarring it.

One way to workaround this is for the parent project to be able to provide its own appups for a dependency that can be committed into its own source repo. In the past we have used something like this:

Set a hook in rebar.config

% can be used to inject appup files. script inherits lots of rebar env vars.
{pre_hooks, [
    {release, "./support/rebar_hooks/pre_release.sh"}
]}.

pre_release.sh:

#!/bin/bash -e

# Looks for .appup files in a folder called dep_appups and installs them
# in the rebar dep ebin before release
for appup in dep_appups/*.appup; do
    dep=$(basename $appup .appup)
    ebin="$REBAR_DEPS_DIR/$dep/ebin"
    target="$ebin/$dep.appup"
    test -d "$ebin" || continue
    echo "===> Installing $appup in $target"
    cp "$appup" "$target"
done

You still have to go in and generate an appup for each dep, so it might be nice if this plugin could generate the initial appup for you as a template to tweak.


Related to this, it might be nice to add the ability to specify a simpler appup generation algorithm for certain deps, where just loading all the modules and restarting the application would work fine.

For example, the place I encountered auto-generation failing was upgrading between this diff lpgauth/statsderl@v0.3.5...0.5.2

It ended up adding 8 generate_supervisor_child_instruction pairs for each of the dynamically generated children, but using the module name rather than their dynamic names. In this case an application restart would have been better. Let me know if I should file an issue for this.

Merge appup.pre.src && appup.post.src files into the appup file after .appup is generated by 'appup generate'

I would like to implement a feature which can be implemented in the appup plugin or separately in a different plugin.

It does not look difficult to implement it, probably it would not be a big change in rebar3_appup_generate.erl.

So, here is the feature we would like to have:

rebar3 appup generate plugin generates appup files using the following format:

> cat relapp.appup
%% appup generated for relapp by rebar3_appup_plugin (2018/01/03 09:06:17)
{ "1.0.16",
    [{ "1.0.15",
        [{load_module,relapp_m1,brutal_purge,brutal_purge,[]}] }],
     [{ "1.0.15",
        [{load_module,relapp_m1,brutal_purge,brutal_purge,[]}] }]
}.

The contents of relapp.appup.pre.src and relapp.appup.post.src files should be merged into the relapp.appup so that the commands from relapp.appup.pre.src should precede the commands defined in the generated appup file, and commands from relapp.appup.post.src should come after those commands.

The two files will have similar format:

> cat src/relapp.appup.pre.src
[
 { "1.0.16",
    [{ "1.0.15",
        [{apply, io, format, ["Upgrading from 1.0.15 to 1.0.16"]}] }],
     [{ "1.0.15",
        [{apply, io, format, ["Downgrading from 1.0.16 to 1.0.15"]}] }]
},
 { "1.0.17",
    [{ "1.0.16",
        [{apply, io, format, ["Upgrading from 1.0.16 to 1.0.17"]}] }],
     [{ "1.0.16",
        [{apply, io, format, ["Downgrading from 1.0.17 to 1.0.16"]}] }]
}
].

> cat src/relapp.appup.post.src
[
 { "1.0.16",
    [{ "1.0.15",
        [{apply, io, format, ["Upgrade completed from 1.0.15 to 1.0.16"]}] }],
     [{ "1.0.15",
        [{apply, io, format, ["Downgrade completed from 1.0.16 to 1.0.15"]}] }]
},
 { "1.0.17",
    [{ "1.0.16",
        [{apply, io, format, ["Upgrade completed from 1.0.16 to 1.0.17"]}] }],
     [{ "1.0.16",
        [{apply, io, format, ["Downgrade completed from 1.0.17 to 1.0.16"]}] }]
}
].

After merging the contents of the .appup && .pre.appup.src && .post.appup.src files, the appup file should like this:

%% appup generated for relapp by rebar3_appup_plugin (2018/01/03 09:06:17)
{ "1.0.16",
    [{ "1.0.15",
        [
          %% comes from the .pre.appup.src
          {apply, io, format, ["Upgrading from 1.0.15 to 1.0.16"]},
          %% contents from  the original appup files
          {load_module,relapp_m1,brutal_purge,brutal_purge,[]},
          %% comes from the .post.appup.src
          {apply, io, format, ["Upgrade completed from 1.0.15 to 1.0.16"]}
        ] }],
     [{ "1.0.15",
        [
           %% comes from the .pre.appup.src
           {apply, io, format, ["Downgrading from 1.0.16 to 1.0.15"]},
           %% contents from  the original appup files
           {load_module,relapp_m1,brutal_purge,brutal_purge,[]},
           %% comes from the .post.appup.src
           {apply, io, format, ["Downgrade completed from 1.0.16 to 1.0.15"]}
      ] }]
}.

In the version strings patterns should be allowed and comparing the pattern in the .pre.appup.src / .post.appup.src files against the version defined in the .appup file the commands should be merged only if the patterns are matching.

What do you think would it be easy to implement, and should it go into a different plugin?

Regards,
Laszlo

relup fail on second upgrade

I can do hot upgrade of a test app from version 0.1.0 to version 0.2.0 but generation of relup from version 0.2.0 to version 0.3.0 fails.

Steps to reproduce:
erlang/otp version 20.2
rebar3 version 3.5.0
rebar3_appup_plugin version 2.3.0

  1. Create project.
rebar3 new release mytest
  1. Add rebar3_appup_plguin and provider_hooks to rebar.config
{plugins, [rebar3_appup_plugin]}.

{provider_hooks, [
    {pre, [
        {tar, {appup, tar}}
    ]},
    {post, [
        {compile, {appup, compile}},
        {clean, {appup, clean}}
    ]}
]}.
  1. Generate first release.
rebar3 as prod release
  1. Change release in rebar.config and app.src to 0.2.0
  2. Create second release and upgrade.
rebar3 as prod release
rebar3 as prod appup generate
rebar3 as prod relup
rebar3 as prod tar
  1. Change release in rebar.config and app.src to 0.3.0
  2. Create third release and upgrade.
rebar3 as prod release
rebar3 as prod appup generate --previous_version 0.2.0
rebar3 as prod relup

Error:

===> Errors generating relup 
          No release upgrade script entry for mytest-0.3.0 to mytest-0.2.0 in file mytest/_build/prod/rel/mytest/lib/mytest-0.3.0/ebin/mytest.appup

no `debug_info` when release with otp24

I'm using trying appup with otp24, I got crash after I modify an erlang file and appup generate

I check the code(rebar3_appup_generate:module_dependencies/3) and find out beam haven't debug_info:

xref:add_directory(xref,"/tmp/test/_build/default/rel/test/lib/test-0.1.10/ebin").
Skipping /tmp/test/_build/default/rel/test/lib/test-0.1.10/ebin/test.beam (no debug information)
Skipping /tmp/test/_build/default/rel/test/lib/test-0.1.10/ebin/test_sup.beam (no debug information)
Skipping /tmp/test/_build/default/rel/test/lib/test-0.1.10/ebin/test_app.beam (no debug information)
{ok,[]}

then, I try to use otp22 and rebar3(3.11.0) and it work well

Old and buggy version in pkg

Hi!

Some weeks ago I've reported a problem #24

It has been already fixed by this commit: 7fc4766

The problem is that the current pkg version (2.3.0) still has this bug as it was released some weeks ago.

We've started to use the current git version in our projects, but the version we use in production still uses pkg, and unfortunately because of this we can't locally release that version when we want to upgrade.

Can you create a new pkg version containing this fix? It would be quite urgent because without this we can't release our new version.

Option to generate simpler appups

From #13

The plugin can struggle to generate correct appups when there are complex supervision trees involved.

It might be nice to add the ability to specify a simpler appup generation algorithm for certain deps, where just loading all the modules and restarting the application would work fine.

For example, I encountered auto-generation failing when upgrading between this diff lpgauth/[email protected]

It ended up adding 8 generate_supervisor_child_instruction pairs for each of the dynamically generated children, but using the module name rather than their dynamic names. In this case an application restart would have been better.

In order to enable consistent appups for a dependency during a CI build, this would probably need to be configured in rebar.config somehow, for example:

{appup, [
    {application_restart, [name_of_dep1, name_of_dep2]}
]}.

Or something better.

Modules changed that implements more than one behaviour are not supported

==> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {case_clause,{undefined,[application,supervisor]}}
===> Stack trace to the error location:
[{rebar3_appup_generate,get_behavior,1,
                        [{file,"_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
                         {line,516}]},
 {rebar3_appup_generate,generate_instruction,4,
                        [{file,"_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
                         {line,450}]},
 {lists,map,2,[{file,"lists.erl"},{line,1237}]},
 {lists,map,2,[{file,"lists.erl"},{line,1237}]},
 {rebar3_appup_generate,generate_appup_files,6,
                        [{file,"_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
                         {line,317}]},
 {lists,foreach,2,[{file,"lists.erl"},{line,1336}]},

rebar3 appup fails - new release overwriting old "previous version apps"

Environment

-bash-4.2$ rebar3 report

Rebar3 report
 version 3.5.3+build.4088.refe2e7f65
 generated at 2018-06-20T14:40:40+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: 
Entered as:
  
-----------------
Operating System: x86_64-unknown-linux-gnu
ERTS: Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [kernel-poll:false]
Root Directory: /usr/local/lib/erlang
Library directory: /usr/local/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.5.0
certifi: 2.0.0
cf: 0.2.2
common_test: 1.15.3
compiler: 7.1.4
crypto: 4.2
cth_readable: 1.4.1
dialyzer: 3.2.3
edoc: 0.9.2
erlware_commons: 1.1.0
eunit: 2.3.5
eunit_formatters: 0.5.0
getopt: 1.0.1
hipe: 3.17
inets: 6.4.5
kernel: 5.4.1
providers: 1.7.0
public_key: 1.5.2
relx: 3.24.4
sasl: 3.1.1
snmp: 5.2.9
ssl_verify_fun: 1.1.3
stdlib: 3.4.3
syntax_tools: 2.1.4
tools: 2.11.1

-----------------
Escript path: /usr/local/bin/rebar3
Providers:
  app_discovery as clean clean compile compile compile cover ct deps dialyzer do edoc escriptize eunit generate get-deps help install install_deps list lock new path pkgs release relup report shell state tar tar tree unlock update upgrade upgrade upgrade version xref 

Current behaviour

Attempting to generate an appup from an old release. First, I removed the whole _build directory. Then, on the most up-to-date version, 0.4.1, I ran rebar3 release:

... (building and compiling deps)
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /home/rauger/repos/backend_debug_client/_build/default/lib
          /usr/local/lib/erlang/lib
===> Resolved backend_debug_client-0.4.1
===> Including Erts from /usr/local/lib/erlang
===> release successfully created!
-bash-4.2$ git branch
* (detached from v0.4.1)
  master
  refactor_as_ct_suite
-bash-4.2$ rebar3 appup generate --previous_version "0.4.0"
===> can not find .rel file for version "0.4.0"

This makes sense - we removed the whole build directory, so there should be no release for "0.4.0". So now build that release:

-bash-4.2$ git checkout "v0.4.0"
Previous HEAD position was 3f62b97... v0.5.0
HEAD is now at 62b3bcd... Added continuous test
-bash-4.2$ rebar3 release
===> Verifying dependencies...
===> Compiling backend_debug_client
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /home/rauger/repos/backend_debug_client/_build/default/lib
          /usr/local/lib/erlang/lib
          /home/rauger/repos/backend_debug_client/_build/default/rel
===> Resolved backend_debug_client-0.4.0
===> Including Erts from /usr/local/lib/erlang
===> release successfully created!
-bash-4.2$ git checkout "v0.4.1"
Previous HEAD position was 62b3bcd... Added continuous test
HEAD is now at 3f62b97... v0.5.0
**-bash-4.2$ rebar3 appup generate --previous_version "0.4.0"
===> current ("0.4.0") and previous ("0.4.0") release versions are the same**
-bash-4.2$ rebar3 release
===> Verifying dependencies...
===> Compiling backend_debug_client
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /home/rauger/repos/backend_debug_client/_build/default/lib
          /usr/local/lib/erlang/lib
          /home/rauger/repos/backend_debug_client/_build/default/rel
===> Resolved backend_debug_client-0.4.1
===> Including Erts from /usr/local/lib/erlang
===> release successfully created!

-bash-4.2$ DEBUG=1 rebar3 appup generate --previous_version "0.4.0"                                                                                                                                                                                                                                                
===> Expanded command sequence to be run: []
===> Provider: {default,do}
===> Expanded command sequence to be run: [{appup,generate}]
===> Provider: {appup,generate}
===> opts: [{previous_version,"0.4.0"}]
===> release name: "backend_debug_client"
===> current release, name: "backend_debug_client", version: "0.4.1"
===> previous release: "/home/rauger/repos/backend_debug_client/_build/default/rel/backend_debug_client"
===> current release: "/home/rauger/repos/backend_debug_client/_build/default/rel/backend_debug_client"
===> target dir: undefined
===> previous release, name: "backend_debug_client", version: "0.4.0"

Here is where the unexpected behaviour comes in - the previous version apps should be 0.4.0, not 0.4.1.

===> previous version apps: [{backend_debug_client,"0.4.1"},
                                    {sasl,"3.1.1"},
                                    {lager,"3.2.2"},
                                    {goldrush,"0.1.9"},
                                    {syntax_tools,"2.1.4"},
                                    {compiler,"7.1.4"},
                                    {cqerl,"1.0.4"},
                                    {ssl,"8.2.3"},
                                    {public_key,"1.5.2"},
                                    {asn1,"5.0.4"},
                                    {pooler,"1.5.2"},
                                    {re2,"1.7.5"},
                                    {semver,"0.0.1"},
                                    {snappy,"1.1.1"},
                                    {lz4,"0.2.2"},
                                    {uuid,"1.7.2"},
                                    {quickrand,"1.7.2"},
                                    {crypto,"4.2"},
                                    {emqttc,"0.8.0"},
                                    {gen_logger,"1.2"},
                                    {sim_security,"0.1.0"},
                                    {stdlib,"3.4.3"},
                                    {kernel,"5.4.1"}]
===> current version apps: [{backend_debug_client,"0.4.1"},
                                   {sasl,"3.1.1"},
                                   {lager,"3.2.2"},
                                   {goldrush,"0.1.9"},
                                   {syntax_tools,"2.1.4"},
                                   {compiler,"7.1.4"},
                                   {cqerl,"1.0.4"},
                                   {ssl,"8.2.3"},
                                   {public_key,"1.5.2"},
                                   {asn1,"5.0.4"},
                                   {pooler,"1.5.2"},
                                   {re2,"1.7.5"},
                                   {semver,"0.0.1"},
                                   {snappy,"1.1.1"},
                                   {lz4,"0.2.2"},
                                   {uuid,"1.7.2"},
                                   {quickrand,"1.7.2"},
                                   {crypto,"4.2"},
                                   {emqttc,"0.8.0"},
                                   {gen_logger,"1.2"},
                                   {sim_security,"0.1.0"},
                                   {stdlib,"3.4.3"},
                                   {kernel,"5.4.1"}]
===> added: []
===> removed: []
===> upgraded: []
===> apps that already have .appups: ["stdlib","kernel","ssl",
                                             "compiler","asn1","sasl",
                                             "syntax_tools","public_key",
                                             "crypto"]
===> generating .appup for apps: []
===> appup opts: [{purge_opts,[[]]},
                         {plugin_dir,"/home/rauger/repos/backend_debug_client/_build/default/plugins/rebar3_appup_plugin"}]

No .appup is generated. It appears that when I run rebar3 release on any newer version, the old version is overwritten. How could this be/how can I prevent this?

Expected behaviour

Expect an appup file to be generated in my _build/default/lib/backend_debug_client/ebin/ directory.

Module names with capital names quoted with ' (single quote) are html esacped

Module names in appup like "'APOLLO-PDU-Descriptions'" are generated as "{load_module,'APOLLO-PDU-Descriptions',..." which can not be parsed by relup.

The reason seems to be that the generator uses templates and bbmustasch to render the appup.
And bbmustach does html escaping on single quotes.

in bbmustach.erl
%% @see escape/1
-spec escape_char(0..16#FFFF) -> binary().
escape_char($<) -> <<"<">>;
escape_char($>) -> <<">">>;
escape_char($&) -> <<"&">>;
escape_char($") -> <<""">>;
escape_char($') -> <<"'">>;
escape_char($/) -> <<"/">>;
escape_char($`) -> <<"`">>;
escape_char($=) -> <<"=">>;
escape_char(C) -> <<C:8>>.

appup generate creates appup files even if appup compile has already created them

List = proplists:delete(list_to_atom(First), UpgradedApps),

In the above mentioned line the proplists:delete tries to operate on the following structures:

First: '"wsa"'
UpgradedApps: '[{upgrade,dapp,
{"0.1.38+build.134.ref8cda303",
"0.1.40"}},
{upgrade,wsa,
{"0.0.28+build.98.refae382b5",
"0.0.29"}},
{upgrade,gwb,{"0.1.6","0.1.9"}},
{upgrade,urbis,{"0.1.6","0.1.9"}},
{upgrade,olog,{"0.0.7","0.0.8"}}]'

It will fail to delete the app from the list as it is not a proplist, so appup generate will create appup files for all modified applications which is not the expected behaviour.

Using
List = lists:keydelete(list_to_atom(First), 2, UpgradedApps),
in the mentioned line fixes the issue.

The new child does not start if a release is upgraded.

I add a new child under a supervisor process, the new module is added, but the child process does not start.
The spec in the supervisor is:

init([]) ->
    AServer = #{
                   id => a_server,
                   start => {a_server, start_link, []}
                   },
    BServer = #{
                   id => b_server,
                   start => {b_server, start_link, []}
                    },
    {ok, {{one_for_all, 10, 10}, [AServer, BServer]}}.

The b_server module is the new added module.
The children are:

supervisor:which_children(server_demo_sup).
[{b_server,undefined,worker,[b_server]},
 {a_server,<0.413.0>,worker,[a_server]}]

The b_server does not start, and its pid is undefined.
Is it the correct? Or do I miss something?

generated appup order

Hi

This is not a bug, but a possible feature request.

We have around 60 erlang repos. All of them use appup plugin by either generating appup from appup.src calling appup compile, or it just calls appup generate if there is no appup.src.

We have an other plugin which does a lot of checks on our erlang projects. For example we have a check (let's name it upgrade_order_validation) which checks the following:

  • if modA calls modB, but modB does not call modA, then our logic is that modB must be upgraded first, then modA.

During a downgrade it is the opposite: modA should be downgraded first. What is the reason?

If all api functions of a module are always backward compatible, so you never delete an existing exported function, but of course you can add a new modB:new/0, then if you upgrade modA first, there will be a small chance that modB is not upgraded yet, so calls to this added function will fail.

On the other hand if modB is upgraded first then the new function will already exist by the time modA is upgraded so it can call the new function. With a similar logic you can easily see during a downgrade you have to downgrade modA first since if you downgrade modB first, modA will still call the already purged function. If you don't follow this logic there will be failing requests during an upgrade.

Using xref we already have the entire calltree in memory, so what we do is that we reorder the generated appup file so it will follow our logic.

The question is whether you would be interested in moving this feature into the appup plugin?

Insert application version in appup

My application has a hardcoded .appup.src like this:

{%VSN%,
[{<<".">>,[{restart_application, MyApp}]}],
[{<<".">>,[{restart_application, MyApp}]}]
}.

I used to have a vsn.mk file which defined the current version, and my build system used this to replace %VSN% with the correct version string. When converting to rebar3 builds, I need an different way to insert the version string. In my app.src I use {vsn,git}.

I have (successfully) used the following appup.src:

_ = application:load(MyApp),
{ok,Vsn} = application:get_key(MyApp,vsn),
{Vsn,
[{<<".">>,[{restart_application, MyApp}]}],
[{<<".">>,[{restart_application, MyApp}]}]
}.

but I think that it might not be a good solution in all cases?

appups not generated for apps that need it but for which a previous version had appups

Description:

The steps I used to create this:

  • build a first release (rebar3 as prod do release, tar)
  • make some changes to the repo:
    • change a module
    • bump app version (0.2.0)
    • bump release version
  • build the new release and generate appups and relups (rebar3 as prod do release, appup generate, relup, tar)
  • make some changes to the repo:
    • change a module
    • bump app version (0.3.0)
    • bump release version
  • build a third release (rebar3 as prod do release, appup generate, relup, tar)

This third release will fail to create the appup for the 0.2.0 -> 0.3.0 transition because the file in _build/rel/lib/myapp-0.2.0/ebin/myapp.appup exists. I believe this is too wide of a scan in this code:

https://github.com/lrascao/rebar3_appup_plugin/blob/a439a59/src/rebar3_appup_generate.erl#L138-L145

    %% Get a list of any appup files that exist in the current release
    CurrentAppUpFiles = rebar3_appup_utils:find_files_by_ext(
                            filename:join([CurrentRelPath, "lib"]),
                            ".appup"),
    %% Convert the list of appup files into app names
    CurrentAppUpApps = [file_to_name(File) || File <- CurrentAppUpFiles],
    rebar_api:debug("apps that already have .appups: ~p",
        [CurrentAppUpApps]),

The file_to_name check drops the version information from the path, which makes the plugin mistakenly assume that an appup file exists for the current version when it only existed for an old one.


Temporary Workaround: remove the old appup file by hand

I don't know if this is bad usage on my end or if this is what could be a bug that needs tracking versions/paths better

SVN rebar3_appup_plugin

Hi ,

I am trying to use this plugin for SVN code base. But when we are doing general appup, the beam_lib:cmp_dirs is listing all the files as changed.

Is this due to path changes.

Using current version on github I have the following error

It does not happen with last stable version (2.2.1) which is available on hex.pm, but if I switch to the github/develop branch, this line

AppDeps = dict:fetch({parsed_deps,default},

will fail with current rebar3 producing this rebar3.crashdump. (this function was introduced recently : 9804f62)

[{dict,fetch,
[{parsed_deps,default},
{dict,22,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],
[[{locks,default},
{<<"certifi">>,
{pkg,<<"certifi">>,<<"1.2.1">>,
<<"C3904F192BD5284E5B13F20DB3CEAC9626E14EEACFBB492E19583CF0E37B22BE">>},
1},
{<<"cfg">>,
{git,"ssh://[email protected]:7999/core/cfg.git",
{ref,"b0a66ba373ab9c3a304b35fba26d1ef7e1a34ffa"}},
1},
{<<"cowboy">>,
{pkg,<<"cowboy">>,<<"1.0.4">>,
<<"A324A8DF9F2316C833A470D918AAF73AE894278B8AA6226CE7A9BF699388F878">>},
2},
...
[[relx,
{dev_mode,false},
{extended_start_script,true},
{include_erts,true},
{release,
{wsa,{semver,[]}},
[runtime_tools,kernel,sasl,stdlib,lager,cowboy,
cowboy_swagger,csi,olog,wsa]},
{sys_config,"config/sys.config"},
{vm_args,"config/vm.args"}]],
[],[],[],[],[],[],[]}}}],
[{file,"dict.erl"},{line,131}]},
{rebar3_appup_compile,'-process_app/2-fun-0-',2,
[{file,
"/Users/tothlac/otp/connector/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_compile.erl"},
{line,81}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
{rebar3_appup_compile,do,1,
[{file,
"/Users/tothlac/otp/connector/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_compile.erl"},
{line,58}]},
{rebar_core,do,2,
[{file,
"/Users/tothlac/rebar3/build/default/lib/rebar/src/rebar_core.erl"},
{line,154}]},
{rebar_hooks,run_provider_hooks
,6,
[{file,
"/Users/tothlac/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
{line,49}]},
{rebar_hooks,run_all_hooks,6,
[{file,
"/Users/tothlac/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
{line,16}]},
{rebar_prv_compile,compile,3,
[{file,
"/Users/tothlac/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl"},
{line,142}]}]

incorrect appup file getting generated

I'm generating appup file for my svn hosted project. I have two branches names rel_v0_1_0 and rel_v0_2_0. Both are identical except application version I have changed. Here is output of svn diff :

user@TH1574U:~/svn/erl_experiment/branches/rel_v0_2_0$ svn diff -r18
Index: lib/erl_experiment/src/erl_experiment.app.src
===================================================================
--- lib/erl_experiment/src/erl_experiment.app.src (.../rel_v0_1_0) (revision 18)
+++ lib/erl_experiment/src/erl_experiment.app.src (.../rel_v0_2_0) (working copy)
@@ -1,6 +1,6 @@
{application, erl_experiment,
[{description, "An OTP application"},

  • {vsn, "0.1.0"},
  • {vsn, "0.2.0"},
    {registered, []},
    {mod, { erl_experiment_app, []}},
    {applications,
    Index: rebar.config
    ===================================================================
    --- rebar.config (.../rel_v0_1_0) (revision 18)
    +++ rebar.config (.../rel_v0_2_0) (working copy)
    @@ -5,7 +5,7 @@
    { base_dir, "../_build"}.
    { out_dir, "../_build/default/lib/"}.

-{relx, [{release, {erl_experiment, "0.1.0"}, [ erl_experiment, kernel, stdlib, sasl ]},
+{relx, [{release, {erl_experiment, "0.2.0"}, [ erl_experiment, kernel, stdlib, sasl ]},

{dev_mode, false},
{include_erts, true},

To Generate appup file, I'm doing :
[in branch rel_v0_1_0 ]

user@TH1574U:~/svn/erl_experiment/branches/rel_v0_1_0$ ./rebar3 release
user@TH1574U:~/svn/erl_experiment/branches/rel_v0_1_0$ ./rebar3 tar

[in branch rel_v0_2_0 ]

user@TH1574U:~/svn/erl_experiment/branches/rel_v0_2_0$ ./rebar3 release
user@TH1574U:~/svn/erl_experiment/branches/rel_v0_2_0$ ./rebar3 appup generate --previous_version 0.1.0 --previous /home/user/svn/erl_experiment/branches/_build/default/rel/erl_experiment/

Appup file is getting generated successfully and here it is, (unfortunately its wrong.)

%% appup generated for erl_experiment by rebar3_appup_plugin (2018/03/06 17:02:57)
{ "0.2.0",
    [{ "0.1.0",
        [{load_module,erl_experiment_app,brutal_purge,brutal_purge,
                      [erl_experiment_sup]},
         {update,erl_experiment_sup,supervisor},
         {update,mynew_server,{advanced,[]},brutal_purge,brutal_purge,[]}] }],
    [{ "0.1.0",
        [{update,mynew_server,{advanced,[]},brutal_purge,brutal_purge,[]},
         {update,erl_experiment_sup,supervisor},
         {load_module,erl_experiment_app,brutal_purge,brutal_purge,
                      [erl_experiment_sup]}] }]
}.

The appup file shows changes in mynew_server module. But that module is not modified.

Can anyone clerify why ?

SVN Repo : erl_experiment
Log : appup_generate.log

start_erl.data is missing

The debug message:

~/e/gate ❯❯❯ DEBUG=1 rebar3 appup generate --previous /Users/suyejun/ema/gate/_build/prod/rel/gate/releases/0.1.0 --current /Users/suyejun/ema/gate/_build/default/rel/gate/releases/0.2.0
===> Expanded command sequence to be run: []
===> Expanded command sequence to be run: [{appup,generate}]
===> opts: [{previous,"/Users/suyejun/ema/gate/_build/prod/rel/gate/releases/0.1.0"},
                   {current,"/Users/suyejun/ema/gate/_build/default/rel/gate/releases/0.2.0"}]

===> release name: "gate"
===> previous release: "/Users/suyejun/ema/gate/_build/prod/rel/gate/releases/0.1.0"

===> current release: "/Users/suyejun/ema/gate/_build/default/rel/gate/releases/0.2.0"

===> target dir: undefined

===> /Users/suyejun/ema/gate/_build/prod/rel/gate/releases/0.1.0/releases/start_erl.data is missing

Then I read the README and changed previous to /Users/suyejun/ema/gate/_build/prod/rel/gate/releases/, then it reports another error:

~/e/gate ❯❯❯ DEBUG=1 rebar3 appup generate --previous /Users/suyejun/ema/gate/_build/prod/rel/gate/releases --current /Users/suyejun/ema/gate/_build/default/rel/gate/releases/0.2.0
===> Expanded command sequence to be run: []
===> Expanded command sequence to be run: [{appup,generate}]
===> opts: [{previous,"/Users/suyejun/ema/gate/_build/prod/rel/gate/releases"},
                   {current,"/Users/suyejun/ema/gate/_build/default/rel/gate/releases/0.2.0"}]

===> release name: "gate"
===> previous release: "/Users/suyejun/ema/gate/_build/prod/rel/gate/releases"

===> current release: "/Users/suyejun/ema/gate/_build/default/rel/gate/releases/0.2.0"

===> target dir: undefined

===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {case_clause,{error,enotdir}}
===> Stack trace to the error location: [{rebar3_appup_rel_utils,
                                                 get_permanent_version,1,
                                                 [{file,
                                                   "/Users/suyejun/ema/gate/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_rel_utils.erl"},
                                                  {line,11}]},
                                                {rebar3_appup_rel_utils,
                                                 get_rel_file_path,2,
                                                 [{file,
                                                   "/Users/suyejun/ema/gate/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_rel_utils.erl"},
                                                  {line,23}]},
                                                {rebar3_appup_rel_utils,
                                                 get_rel_release_info,2,
                                                 [{file,
                                                   "/Users/suyejun/ema/gate/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_rel_utils.erl"},
                                                  {line,38}]},
                                                {rebar3_appup_generate,do,1,
                                                 [{file,
                                                   "/Users/suyejun/ema/gate/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
                                                  {line,62}]},
                                                {rebar_core,do,2,
                                                 [{file,
                                                   "/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
                                                  {line,125}]},
                                                {rebar_prv_do,do_tasks,2,
                                                 [{file,
                                                   "/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_do.erl"},
                                                  {line,68}]},
                                                {rebar_core,do,2,
                                                 [{file,
                                                   "/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
                                                  {line,125}]},
                                                {rebar3,main,1,
                                                 [{file,
                                                   "/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar3.erl"},
                                                  {line,56}]}]

I don't understand how to specify the current and previous, can you give any point?

Thanks.

string:tokens/2 and string:strip/3 will be deprecated

I'm getting two errors when trying to use with erlang/otp master.
Is it possible to not fail on warnings?

_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_rel_utils.erl:36: string:tokens/2: deprecated; use string:lexemes/2 instead
_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_rel_utils.erl:37: string:strip/3: deprecated; use string:trim/3 instead

Should be:
https://github.com/lrascao/rebar3_appup_plugin/blob/develop/src/rebar3_appup_rel_utils.erl#L36

[_, Version] = string:lexemes(
                             string:trim(binary_to_list(DataBin), trailing, [$\n]),
                             " "),

error in rebar3_appup_generate

Hi. We are now using pkg 2.4.0, and we have the following error:

Error: function_clause
[{rebar3_appup_generate,select_behaviour,
[[]],
[{file,"/Users/tothlac/otp/kamarf/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
{line,564}]},
{rebar3_appup_generate,generate_instruction,4,
[{file,"/Users/tothlac/otp/kamarf/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
{line,476}]},
{lists,map,2,[{file,"lists.erl"},{line,1239}]},
{lists,map,2,[{file,"lists.erl"},{line,1239}]},
{rebar3_appup_generate,generate_appup_files,6,
[{file,"/Users/tothlac/otp/kamarf/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
{line,322}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
{rebar3_appup_generate,do,1,
[{file,"/Users/tothlac/otp/kamarf/_build/default/plugins/rebar3_appup_plugin/src/rebar3_appup_generate.erl"},
{line,161}]},
{rebar_core,do,2,
[{file,"/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
{line,153}]}]

It looks like this commit: 5fd1632

and specifically the drop_unknown_behaviours function takes all elements from the list, so select_behaviour is called with an empty list, hence we have the function_clause. Would it be enough to fix this problem to add:

  • select_behaviour([]) -> undefined;
    select_behaviour([B]) -> B;

Do "rebar3 appup generate" work with git?

Hi!
Do "rebar3 appup generate" work with git?
I rm -rf .git, and follow the UPGRADE_DOWNGRADE,
change the code, and modify the version of rebar.config

dcy@dcy-dcy:~/test/relapp1$ rebar3 appup generate
dcy@dcy-dcy:~/test/relapp1$ 

Nothing generated.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.