labzero / bootleg Goto Github PK

I'm trying to start run custom task via mix bootleg.invoke and getting nothing. So, this is the strategy:

1. I have the file in my projects lib, as in example:

# lib/bootleg/tasks/other.ex

defmodule Bootleg.Tasks.Other do
  use Bootleg.Task do
    task :other do
      Bootleg.UI.info("World?")
    end
  end
end

2. I've modified bootleg's source for debug purposes - like this:

  @spec invoke(atom) :: :ok
  def invoke(task) when is_atom(task) do
    IO.inspect("Invoking task #{inspect task}")
    invoke_task_callbacks(task, :before_hooks)

    module_name = module_for_task(task)
    IO.inspect("Module name is #{inspect module_name}")
    if Code.ensure_compiled?(module_name) do
      IO.inspect("Module is loaded")
      apply(module_name, :execute, [])
    else
      IO.inspect("Module is NOT loaded")
    end

    invoke_task_callbacks(task, :after_hooks)
  end

3. Now, I'm trying to call this task via invoke:

$ mix bootleg.invoke other
"Invoking task :other"
"Module name is Bootleg.DynamicTasks.Other"
"Module is NOT loaded"
$

4. Calling phoenix_digest from bootleg_phoenix package is working:

$ mix bootleg.invoke phoenix_digest
"Invoking task :phoenix_digest"
"Module name is Bootleg.DynamicTasks.PhoenixDigest"
"Module is loaded"
....
$

5. The task other is also working via iex shell:

iex(1)> Bootleg.Config.invoke :other
"Invoking task :other"
"Module name is Bootleg.DynamicTasks.Other"
"Module is loaded"
World?
:ok
iex(2)>

I'm totally missed in this reality, hope my debuggings will help you with fixing this bug.

I believe there is sufficient externally useful bits in UI now after @Virviil's prompt additions that it might be useful to have some documentation around the helpers available in UI. In fact, this might also be a good opportunity to break out the existing README docs for hooks/tasks/invoke/remote/ui into a separate doc file which could cover customization as a whole, maybe something like CUSTOMIZATION.md

Currently there is no way to tell exactly what tasks/hooks will get executed when a particular task is invoked, especially when another package is providing some of those tasks for you. The bootleg mix tasks would support a --dry-run flag, which would cause bootleg to load and parse its config files like normal, and then "run" the tasks with remote turned into a no-op (or maybe patch into SSHKit's dry run functionality). During the run, it would print out each task/hook as they are entered/exited.

The flexibility of the DSL is a bit of a disadvantage here, as things like invoke can be conditionalised on the results of a remote call. Doing AST traversal, it should be possible to detect those cases and maybe the output would reflect both branches? Or warn that the results are unknown (we don't want to follow too many conditional branches if the user has a bunch of conditional invokes).

This feature is probably going to need to rely on AST parsing instead of actual code execution to avoid side-effects like notifying APMs or touching files via system calls.

Example output:

$ mix bootleg.build --dry-run
Dry run...
task :build started
  task :compile started
    remote :build, MIX_ENV=prod mix deps.compile
    remote :build, MIX_ENV=prod mix compile
  task :compile ended
  task :generate_release started
    remote :build, MIX_ENV=prod mix release
  task :generate_release ended
task :build ended

task :update do
  invoke :build
  invoke :deploy
  invoke :start
end

If the app is already running, Distillery start will not stop it and start it again on its own.

Unlike restart, Distillery's start and stop will not throw errors if the running status is unexpected, so it's possible we can just add an invoke :stop in our :update task to fix the problem, but this may require some more work.

After upgrading to bootleg 0.5.0 the format of the output looks like the following. Notice the quotes look weird now. Some output has lines split across multiple lines of output as well, which may have been a problem before.

[MY_IP_ADDRESS ] removed �deps/postgrex/lib/postgrex/extensions/int4.ex�
[MY_IP_ADDRESS ] removed direct
[MY_IP_ADDRESS ] ory: �deps/postgrex/lib/postgrex/extensions�

I don't get any errors though like I did previously.

Is there a reason why there's an extra space after the IP address in all bootleg output?

[a.b.c.d ]

Create a shortcut rollup command that is the functional equivalent to running

mix bootleg.build
mix bootleg.deploy
mix bootleg.start

Proposed additional task would be called update and would be exposed via a mix task of mix bootleg.update

I'm seeing a rather puzzling and opaque internal error:

$ mix bootleg.build production
Creating remote context at '/tmp/bootleg/build'
[MY_BUILD_HOST] (export REPLACE_OS_VARS="true" && /usr/bin/env mkdir -p /tmp/bootleg/build)
** (SSHError) SSHKit returned an internal error on MY_BUILD_HOST: {:function_clause, [{:public_key, :pem_entry_decode, [nil], [file: 'public_key.erl', line: 141]}, {SSHClientKeyAPI, :user_key, 2, [file: 'lib/ssh_client_key_api.ex', line: 102]}, {:ssh_auth, :get_public_key, 2, [file: 'ssh_auth.erl', line: 145]}, {:ssh_connection_handler, :is_usable_user_pubkey, 2, [file: 'ssh_connection_handler.erl', line: 1769]}, {:ssh_connection_handler, :"-init_ssh_record/4-lc$^0/1-0-", 2, [file: 'ssh_connection_handler.erl', line: 477]}, {:ssh_connection_handler, :init_ssh_record, 4, [file: 'ssh_connection_handler.erl', line: 476]}, {:ssh_connection_handler, :init, 1, [file: 'ssh_connection_handler.erl', line: 412]}, {:ssh_connection_handler, :init_connection_handler, 3, [file: 'ssh_connection_handler.erl', line: 374]}]}
    lib/bootleg/ssh.ex:70: anonymous fn/2 in Bootleg.SSH.run/2
    (elixir) lib/enum.ex:1294: Enum."-map/2-lists^map/1-0-"/2
    lib/bootleg/ssh.ex:163: Bootleg.SSH.run!/2
    lib/bootleg/ssh.ex:87: Bootleg.SSH.validate_workspace/3
    lib/bootleg/ssh.ex:51: Bootleg.SSH.init/2
    lib/strategies/build/distillery.ex:9: Bootleg.Strategies.Build.Distillery.init/0
    lib/strategies/build/distillery.ex:16: Bootleg.Strategies.Build.Distillery.build/0
    lib/bootleg/config.ex:356: Bootleg.Config.invoke/1

I'm using and ed25519 key if that helps. This is a brand new server. Might you be able to offer some insight or means of debugging this error?

I am having some issues getting the assets for phoenix 1.3 working:

this is the default phoenix task in my deploy.exs:

after_task :compile do
  mix_env = Keyword.get(config(), :mix_env, "prod")
  remote :build do
    "[ -f package.json ] && npm install || true"
    "[ -f brunch-config.js ] && [ -d node_modules ] && ./node_modules/brunch/bin/brunch b -p || true"
    "[ -d deps/phoenix ] && MIX_ENV=#{mix_env} mix phoenix.digest || true"
  end
end

I can successfully build, deploy and start the server, however, the priv/static directory will remain empty and no assets are built. I can see the server running but without assets. I assume this is because in Phoenix 1.3 the package.json is in the /assets directory. However adding "cd ./assets" directly after remote :build does fail.

Anyone has a clue? Thanks

By default it would be ["*"], which would clean the entire workspace. A user or package could add additional locations or change the location to something more specific, say just priv/static or something.

A location not being found would not be an error, so things like ["*", "priv/static"] would just work. All relative paths would be relative to the workspace, and absolute paths would work as expected.

The proposed configuration name is clean_locations.

So trying to symlink my prod.secret.exs and getting an error here before invoke :compile, thus not allowing me to symlink/copy with the before_task :compile task. So mix local.rebar --force is the specific command looking for the prod.secret.exs file. I also tried after_task :clean as well with no dice. Eventually I used replace_os_vars in the prod secret file and removed from gitignore. If you think a PR could come out of this, I'd be happy to help. Otherwise, I can close.

** (SSHError) Command exited on 127.0.0.1 with non-zero status (1)
     cmd: MIX_ENV=prod mix local.rebar --force
  stderr: ** (Mix.Config.LoadError) could not load config config/prod.secret.exs
              ** (File.Error) could not read file "/tmp/bootleg/build/config/prod.secret.exs": no such file or directory
              (elixir) lib/file.ex:272: File.read!/1
              (mix) lib/mix/config.ex:180: Mix.Config.read!/2
              (mix) lib/mix/config.ex:217: anonymous fn/3 in Mix.Config.read_wildcard!/2
              (elixir) lib/enum.ex:1811: Enum."-reduce/3-lists^foldl/2-0-"/3
              (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
              (stdlib) erl_eval.erl:878: :erl_eval.expr_list/6
              (stdlib) erl_eval.erl:404: :erl_eval.expr/5

    lib/bootleg/ssh.ex:169: Bootleg.SSH.run_result/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
    lib/strategies/build/distillery.ex:24: Bootleg.Strategies.Build.Distillery.build/0
    lib/bootleg/config.ex:356: Bootleg.Config.invoke/1
    (mix) lib/mix/task.ex:301: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:75: Mix.CLI.run_task/2

If you try to filter by attribute in remote/3, the list of hosts will not be filtered. To reproduce:

use Bootleg.Config

role :app, "validhost.example.com"
[] = remote :app, [foo: :bar], "true"

This should not raise an error (or establish any SSH connections), instead it raises a match error and establishes a connection to validhost.example.com.

Right now bootleg only supports a single config, found in config/deploy.exs. This makes it hard to target different deployment environments (development, staging, UAT, production, etc) without lots of reliance on system ENVs.

To support multiple deployment environments, it makes sense to have another set of optional config files in config/deploy, named for the particular environment (config/deploy/staging.exs, config/deploy/production.exs) that provide additional configuration for that environment. The config for that environment would be loaded after the main config/deploy.exs file is loaded. In addition, the current environment would be accessible from the config agent via the key bootleg_env.

Each bootleg mix command would then take an optional target environment (defaults to production).

For example:
mix bootleg.deploy production
mix bootleg.build staging

This is separate from MIX_ENV, as we need to target deploy environments separately from MIX_ENV. For example, MIX_ENV will likely be prod for both production and staging bootleg environments.

If build server host is the same as app server host, copy release from :build workspace to :app workspace instead of uploading release from local.

For example:

remote :db, primary: true do
  "rm -rf /"
end

mix bootleg.init could create a basic template version of config/deploy.exs

use Bootleg.Config

# Configure the following roles to match your environment.
# `build` defines what remote server your distillery release should be built on.
# `app` defines what remote servers your distillery release should be deployed and managed on.
#
# Some available options are:
#  - `user`: ssh username to use for SSH authentication to the role's hosts
#  - `password`: password to be used for SSH authentication
#  - `identity`: local path to an identity file that will be used for SSH authentication instead of a password
#  - `workspace`: remote file system path to be used for building and deploying this Elixir project

role :build, "build.example.com", workspace: "/tmp/bootleg/build"
role :app, ["app1.example.com", "app2.example.com"], workspace: "/var/app/example"

# Phoenix has some extra build steps which can be defined as task after the compile step runs.
#
# Uncomment the following task definition if this is a Phoenix application. To learn more about 
# hooks and adding additional behavior to your deploy workflow, please refer to the bootleg 
# README which can be found at https://github.com/labzero/bootleg/blob/master/README.md

# after_task :compile do
#   remote :build do
#     "[ -f package.json ] && npm install || true"
#     "[ -f brunch-config.js ] && [ -d node_modules ] && ./node_modules/brunch/bin/brunch b -p || true"
#     "[ -d deps/phoenix ] && mix phoenix.digest || true"
#   end
# end

In addition to the creation of the template config, perhaps we can print out some basic bootleg command usage information

In the case that a user does not need to build on a remote host, bootleg should provide a way to run a build locally without the need to ssh.

When I run mix bootleg.build I pretty regularly (but intermittently) get an error similar to the following during the workspace cleanup step.

** (ArgumentError) argument error
    (stdlib) :io.put_chars(:standard_io, :unicode, [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[...] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[MY_IP_ADDRESS ] "] | "\e[0m"], "removed ‘_build/prod/rel/myapp/lib/db_connection-1.1.2/ebin/db_connection.app’"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[MY_IP_ADDRESS ] "] | "\e[0m"], "removed ‘_build/prod/rel/myapp/lib/db_connection-1.1.2/ebin/Elixir.DBConnection.Sojourn.Broker.beam’"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[MY_IP_ADDRESS ] "] | "\e[0m"], "removed ‘_build/prod/rel/myapp/lib/db_connection-1.1.2/ebin/Elixir.DBConnection.App.beam’"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[MY_IP_ADDRESS ] "] | "\e[0m"], "removed ‘_build/prod/rel/myapp/lib/db_connection-1.1.2/ebin/Elixir.DBConnection.Sojourn.Supervisor.beam’"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[MY_IP_ADDRESS ] "] | "\e[0m"], "removed ‘_build/prod/rel/myapp/lib/db_connection-1.1.2/ebin/Elixir.DBConnection.Poolboy.Worker.beam’"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[MY_IP_ADDRESS ] "] | "\e[0m"], "removed ‘_build/prod/rel/myapp/lib/db_connection-1.1.2/ebin/Elixir.DBConnection.Pool.beam’"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[MY_IP_ADDRESS ] "] | "\e[0m"], "removed ‘_build/prod/rel/myapp/lib/db_connection-1.1.2/ebin/Elixir.DBConnection.Query.beam’"] | "\e[0m"], 10])
    lib/bootleg/ssh.ex:90: Bootleg.SSH.capture/3
    lib/sshkit/ssh/channel.ex:224: SSHKit.SSH.Channel.loop/4
    lib/sshkit/ssh.ex:125: SSHKit.SSH.run/3
    lib/bootleg/ssh.ex:70: anonymous fn/2 in Bootleg.SSH.run/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
    lib/bootleg/ssh.ex:109: Bootleg.SSH.run!/2
    deps/bootleg/lib/bootleg/tasks/build.exs:31: anonymous fn/3 in Bootleg.DynamicTasks.Clean.execute/0

I haven't taken the time to look into this yet, so forgive me if this is a ticket for sshkit or some other dependency.

In #164 we've discussed changing how filters are specified in the remote macro. Given that it may take some time to get there, and it represents a breaking change, we should update the README to remove the filtering example, or at least call out how it may change in the (relatively) near future.

Similar to bootleg.ping and other built-in commands.

Downloading a file from the remote host can be done by using Bootleg.SSH.download/3, but its usage is undocumented and it requires a raw SSHKit.Context. Exposing this functionality via the DSL with proper role/host support is a much better alternative.

use Bootleg.Config

after_task :deploy do
  # dumps to the remote workspace
  download :app, "path/to/a/remote/file"

  # or with an explicit destination
  download :app, "path/to/a/remote/file", "local/path"
end

Uploading a file to the remote host can be done by using Bootleg.SSH.upload/3, but its usage is undocumented and it requires a raw SSHKit.Context. Exposing this functionality via the DSL with proper role/host support is a much better alternative.

use Bootleg.Config

after_task :deploy do
  # dumps to the remote workspace
  upload :app, "path/to/a/local/file"

  # or with an explicit destination
  upload :app, "path/to/a/local/file", "remote/path"
end

build_type: :docker would run a build inside a docker container.

=== stream of conscience ===
Dockerfile name could be defined as config variable (or maybe via :build role definition?)
could default to ./Dockerfile or ./Dockerfile.build
docs should present any required configuration lines

The documentation should link to the appropriate source code.

As @pmeinhardt pointed out in [#213] it may be handy to have BOOTLEG_ENV environment variable always set for remote commands

Currently this is supported by manually setting the environment variable on the :app hosts, but it might be nice to be able to set this via a config setting. perhaps something as obvious as

use Bootleg.Config

config :replace_os_vars, true

alternatively, during role definition

use Bootleg.Config

role :app, ["host1", "host2"], replace_os_vars: true

Hi there 👋,

I was wondering how I could deploy a branch other than master?

I saw a :refspec option mentioned in the README, but adding that to the role :build did not have any effect. Am I doing something wrong?

Thanks a lot ☀️

This is just a leftover from having first supported username and password.

Having our README examples define username and password is probably both a turn-off and encouraging of a bad practice, as deploy configs are likely checked into source control.

Identity file should be the default authentication mechanism.

Right now :generate_release and :compile are the only bundled tasks that use the config agent, but they should use it via the config/0 macro.

# messy
config()[:mix_env] || "prod"
# better
Keyword.get(config(), :mix_env, "prod")
# ideal
# ...[future]

I would like to symlink my prod.secret.exs files from outside the git repo after git clone but before a build. I was hoping there might be a git_clone task or something similar that I can create an after_task hook for. Does that sound like a reasonable feature to add? If there's a better way to handle secrets I'd love to hear.

Thanks, and I'm excited to see where the project leads. :)

When trying to build and deploy an application, I am getting the error below. Since this lib uses sshkit under the hood, I was wondering if anybody has encountered the same problems and got around it? Here is the related issue in sshkit.

Ref Issue: bitcrowd/sshkit.ex#88

** (SSHError) SSHKit returned an internal error on 165.227.58.185: 'User interaction is not allowed'
    lib/bootleg/ssh.ex:70: anonymous fn/2 in Bootleg.SSH.run/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
    lib/bootleg/ssh.ex:163: Bootleg.SSH.run!/2
    lib/bootleg/ssh.ex:87: Bootleg.SSH.validate_workspace/3
    lib/bootleg/ssh.ex:51: Bootleg.SSH.init/2
    deps/bootleg/lib/bootleg/tasks/ping.exs:5: anonymous fn/3 in Bootleg.DynamicTasks.Ping.execute/0
    (elixir) lib/enum.ex:1811: Enum."-reduce/3-lists^foldl/2-0-"/3
    deps/bootleg/lib/bootleg/tasks/ping.exs:5: Bootleg.DynamicTasks.Ping.execute/0

When I implemented local_build in #229 I think I had the short-term blinders on. After giving it a bit more thought I think this should be implemented as a more general build_type config option that we can use into the future. Right now i see at least three use-cases:

build_type: :remote - (DEFAULT) build using remote build host
build_type: :local - run build locally
build_type: :docker - run locally inside a docker container (to be implemented in the future)

I'm trying to write a custom task that has to cd into different directories and run commands. It does something like the following.

remote :build do
  "cd apps/myapp_web/assets && yarn install"
  "cd apps/myapp_web/assets && ./node_modules/brunch/bin/brunch b -p"
  "cd apps/myapp_web && mkdir priv/static && MIX_ENV=#{mix_env} mix phx.digest"
end

The problem I'm running into is that these commands are prefixed with cd /path/to/workspace && /usr/bin/env, so after the cd completes I'm back to my original context.

$ pwd
/tmp
$ env cd dir1 && pwd
/tmp

I can't wrap my entire command in quotes since env will say it can't find the command (thinking the whole string is my command path). The only way I can think of to deal with this with the current code is to explicitly invoke my shell and pass in the commands, like bash -c "cd ... && ...". Alternatively perhaps the remote block could take a custom directory to cd into within the build workspace? I'm not really sure what is ideal but I thought I'd provide the feedback anyway in case it is helpful.

There is currently nothing documented in the README about these macros. We should at minimum call out:

• config :refspec, "branch"
• config :app, "appname"
• config :version, "1.0.0"
• config :env, :production
• config :ex_path, "path/to/elixir/proj/"

According to the doc if the :build role is not set the release should be done locally. If I remove the role :build,... line in config/deploy.exs I get ** (ArgumentError) You must supply a %Host{}, a %Role{} or a defined role_name.

What is the correct way to do it?

Do you plan to add the ability to build a release by pulling code from repository (github..) instead by pushing local code ? (both would be useful)

Would directly invoke a specific task that has been defined by Bootleg or an bootleg task or extension packages

mix bootleg.invoke [task_name]

It seams that rollup updates are not supported. Any intention on supporting them?

Hi, I hope you don’t mind me bothering you 🙂 Thanks a lot for your quick replies!
I was wondering what your strategy for environment variables with bootleg is.

More specifically: I would like to pass a PORT environment variable when starting an app. For now what I did is to override the :start task in the environment – obviously a hack:

# config/deploy/production.exs

use Bootleg.Config

# …

task :start do
  remote :app do
    "PORT=20673 bin/#{Bootleg.Config.app} start"
  end
  Bootleg.UI.info "#{Bootleg.Config.app} started"
  :ok
end

Unfortunately I don't have any control over what environment the system sets by default.

I bet there's a better way, maybe you can help me out?

I have a Phoenix umbrella project with two apps (myapp and myapp_web). At the top level of the repo my rel/config.exs looks like the following.

# Import all plugins from `rel/plugins`
# They can then be used by adding `plugin MyPlugin` to
# either an environment, or release definition, where
# `MyPlugin` is the name of the plugin module.
Path.join(["rel", "plugins", "*.exs"])
|> Path.wildcard()
|> Enum.map(&Code.eval_file(&1))

use Mix.Releases.Config,
    # This sets the default release built by `mix release`
    default_release: :default,
    # This sets the default environment used by `mix release`
    default_environment: Mix.env()

# For a full list of config options for both releases
# and environments, visit https://hexdocs.pm/distillery/configuration.html


# You may define one or more environments in this file,
# an environment's settings will override those of a release
# when building in that environment, this combination of release
# and environment configuration is called a profile

environment :dev do
  set dev_mode: true
  set include_erts: false
  set cookie: :"SECRET_COOKIE"
end

environment :prod do
  set include_erts: true
  set include_src: false
  set cookie: :"SECRET_COOKIE"
end

# You may define one or more releases in this file.
# If you have not set a default release, or selected one
# when running `mix release`, the first release in the file
# will be used by default

release :myapp do
  set version: "0.1.0"
  set applications: [
    :runtime_tools,
    myapp: :permanent,
    myapp_web: :permanent
  ]
end

If I build the release with mix release it builds fine and creates _build/prod/rel/tree/releases/0.1.0/myapp.tar.gz. If I build using mix bootleg.build the same file is created, but I get the following output after the build completes. Is there some special configuration needed to make this work with umbrella projects?

[10.2.0.49 ] ==> Assembling release..
[10.2.0.49 ] ==> Building release myapp:0.1.0 using environment prod
[10.2.0.49 ] ==> One or more direct or transitive dependencies are missing from
[10.2.0.49 ]     :applications or :included_applications, they will not be included
[10.2.0.49 ]     in the release:
[10.2.0.49 ]
[10.2.0.49 ]     :elixir_make
[10.2.0.49 ]
[10.2.0.49 ]     This can cause your application to fail at runtime. If you are sure
[10.2.0.49 ]     that this is not an issue, you may ignore this warning.
[10.2.0.49 ] ==> Including ERTS 9.0 from /usr/lib/erlang/erts-9.0
[10.2.0.49 ] ==> Packaging release..
[10.2.0.49 ] ==> Release successfully built!
[10.2.0.49 ]     You can run it in one of the following ways:
[10.2.0.49 ]       Interactive: _build/prod/rel/myapp/bin/myapp console
[10.2.0.49 ]       Foreground: _build/prod/rel/myapp/bin/myapp foreground
Downloading release archive
[10.2.0.49 ] DOWNLOAD /data/bootleg/build/_build/prod/rel//releases/0.0.1/.tar.gz -> releases/0.0.1.tar.gz
unsupported user key algorithm :"ssh-dss"
CentOS: Access or use of this system is restricted to authorized users only. The following are strictly prohibited: unauthorized disruption of the traceable, correct, and complete functions of this system inclusive of communication, processing and software, destruction, alteration or disclosure of data or cryptographic keys contained. Misuse of this system, software, data, or cryptography is subject to both civil and criminal prosecution.** (RuntimeError) SCP download error: Invalid SCP directive received: scp: /data/bootleg/build/_build/prod/rel/releases/0.0.1/.tar.gz: No such file or directory

 lib/bootleg/ssh.ex:122: Bootleg.SSH.download/3
 lib/strategies/build/distillery.ex:142: Bootleg.Strategies.Build.Distillery.download_release_archive/2
 lib/bootleg/config.ex:311: Bootleg.Config.invoke/1
 deps/bootleg/lib/bootleg/tasks/update.exs:4: Bootleg.DynamicTasks.Update.execute/0
 lib/bootleg/config.ex:311: Bootleg.Config.invoke/1
 (mix) lib/mix/task.ex:301: Mix.Task.run_task/3
 (mix) lib/mix/cli.ex:75: Mix.CLI.run_task/2

I've been trying to get bootleg to work for a while now but seem to be completely stuck on even getting commands to run on the server. Anyone having a clue what is wrong?

I can ssh into the remote server the regular way without issues so that rules out the any problem with the key.

My config:

role :build, "my-domain.com",
  user: "deploy",
  identity: Path.expand("~/.ssh/id_rsa"),
  workspace: "~/tmp/bootleg/build"

Stacktrace from mix bootleg.build production (using elixir 1.5.1 and OTP 20)

Creating remote context at '~/tmp/bootleg/build'
[my-domain.com] /usr/bin/env mkdir -p ~/tmp/bootleg/build
** (SSHError) SSHKit returned an internal error on my-domain.com: {:function_clause, [{:public_key, :pem_entry_decode, [{:RSAPrivateKey, <<191, 8, 24, 153, 216, 76, 147, 2, 19, 72, 42, 182, 142, 174, 157, 112, 161, 94, 6, 160, 222, 224, 247, 247, 52, 200, 16, 19, 244, 44, 128, 219, 71, 131, 153, 55, 179, 128, 124, 232, 217, 140, 10, ...>>, {'AES-128-CBC', <<84, 47, 220, 211, 67, 173, 224, 77, 235, 222, 244, 234, 184, 159, 23, 18>>}}], [file: 'public_key.erl', line: 131]}, {SSHClientKeyAPI, :user_key, 2, [file: 'lib/ssh_client_key_api.ex', line: 102]}, {:ssh_auth, :get_public_key, 2, [file: 'ssh_auth.erl', line: 145]}, {:ssh_connection_handler, :is_usable_user_pubkey, 2, [file: 'ssh_connection_handler.erl', line: 1744]}, {:ssh_connection_handler, :"-init_ssh_record/4-lc$^0/1-0-", 2, [file: 'ssh_connection_handler.erl', line: 467]}, {:ssh_connection_handler, :init_ssh_record, 4, [file: 'ssh_connection_handler.erl', line: 466]}, {:ssh_connection_handler, :init, 1, [file: 'ssh_connection_handler.erl', line: 401]}, {:ssh_connection_handler, :init_connection_handler, 3, [file: 'ssh_connection_handler.erl', line: 364]}]}
    lib/bootleg/ssh.ex:66: anonymous fn/2 in Bootleg.SSH.run/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
    lib/bootleg/ssh.ex:109: Bootleg.SSH.run!/2
    lib/bootleg/ssh.ex:83: Bootleg.SSH.validate_workspace/3
    lib/strategies/build/distillery.ex:9: Bootleg.Strategies.Build.Distillery.init/0
    lib/strategies/build/distillery.ex:16: Bootleg.Strategies.Build.Distillery.build/0
    lib/bootleg/config.ex:311: Bootleg.Config.invoke/1
    (mix) lib/mix/task.ex:301: Mix.Task.run_task/3

Hi, thanks for the great tool!

I had this task in my deploy.exs:

after_task :compile do
  remote :build do
    "[ -f package.json ] && npm install || true"
    "[ -f brunch-config.js ] && [ -d node_modules ] && ./node_modules/brunch/bin/brunch b -p || true"
    "[ -d deps/phoenix ] && mix phoenix.digest || true"
  end
end

Suddenly my builds start failing with this output:

...
[my-build-host] │ │ │ │ │ └── [email protected]
[my-build-host] │ │ │ │ └─┬ [email protected]
[my-build-host] │ │ │ │   └── [email protected]
[my-build-host] │ │ │ ├── [email protected]
[my-build-host] │ │ │ ├─┬ [email protected]
[my-build-host] │ │ │ │ └─┬ [email protected]
[my-build-host] │ │ │ │   ├── [email protected]
[my-build-host] │ │ │ │   ├── [email protected]
[my-build-host] │ │ │ │   └── [email protected]
[my-build-host] │ │ │ ├── [email protected]
[my-build-host] │ │ │ └── [email protected]
[my-build-host] │ │ ├─┬ [email protected]
[my-build-host] │ │ │ └─┬ [email protected]
[my-build-host] │ │ │   ├── [email protected]
[my-build-host] │ │ │   └── [email protected]
[my-build-host] │ │ └── [email protected]
[my-build-host] │ └─┬ [email protected]
[my-build-host] │   ├─┬ [email protected]
[my-build-host] │   │ ├── [email protected]
[my-build-host] │   │ └── [email protected]
[my-build-host] │   └─┬ [email protected]
[my-build-host] │     └─┬ [email protected]
** (ArgumentError) argument error
    (stdlib) :io.put_chars(:standard_io, :unicode, [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[...] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[my-build-host] "] | "\e[0m"], "      ├─┬ [email protected]"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[my-build-host] "] | "\e[0m"], "      │ ├─┬ [email protected]"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[my-build-host] "] | "\e[0m"], "      │ │ ├─┬ [email protected]"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[my-build-host] "] | "\e[0m"], "      │ │ │ └── [email protected]"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[my-build-host] "] | "\e[0m"], "      │ │ └── [email protected]"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[my-build-host] "] | "\e[0m"], "      │ ├── [email protected]"], "\n"] | "\e[0m"] | "\e[1m"] | "\e[34m"], "[my-build-host] "] | "\e[0m"], "      │ └── [email protected]"] | "\e[0m"], 10])
    lib/bootleg/ssh.ex:90: Bootleg.SSH.capture/3
    lib/sshkit/ssh/channel.ex:224: SSHKit.SSH.Channel.loop/4
    lib/sshkit/ssh.ex:125: SSHKit.SSH.run/3
    lib/bootleg/ssh.ex:70: anonymous fn/2 in Bootleg.SSH.run/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2
    lib/bootleg/ssh.ex:109: Bootleg.SSH.run!/2
    (elixir) lib/enum.ex:1255: Enum."-map/2-lists^map/1-0-"/2

I resolved this problem by replacing:

"[ -f package.json ] && npm install || true"

to

"[ -f package.json ] && npm install --silent > /dev/null || true"

Output results returned from remote/3 drop data in some cases where multiple roles are used with differing host counts.

For example:

use Bootleg.Config

role :app, ["app1.example.com", "app2.example.com"]
role :build, ["build.example.com"]

remote [:build, :app], do: "hostname" # should return 3 results, will only return 2

The issue only occurs if the host counts between roles does not match. In the above example, the following would behave correctly:

remote :build, do: "hostname" # returns 1 result
remote :app, do: "hostname" # returns 2 results

In all cases the remote command is executed on all requested hosts, and the the streamed output back to STDOUT will include output for all requested hosts. Only the return value from remote/3 is impacted.

While poking through the build task functional tests I noticed there are two tests that don't actually assert anything. These tests shoulda have at least one assertion

• test "builds the application"
• test "builds the application with an absolute workspace path"

Argument error on parsing env from tasks

mix bootleg.update production

Raise an error

** (ArgumentError) argument error
    :erlang.hd([])
    lib/bootleg/tasks.ex:53: Bootleg.Tasks.parse_env_task/1
    lib/mix/tasks/update.ex:2: Mix.Tasks.Bootleg.Update.run/1
    (mix) lib/mix/task.ex:314: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:80: Mix.CLI.run_task/2

The problem seems to come from https://github.com/labzero/bootleg/blob/master/lib/bootleg/tasks.ex#L50 trying to get head of an empty list.

This was recently observed when phoenix asset digest failed but its error didn't cause the build to be aborted. The assets appeared to still have been bundled into and deployed with the app, but were actually old assets from a previous build.

To repro:

• On build host, echo "foo" > /tmp/bootleg/build/priv/static/bar (using your build workspace)
• Run mix bootleg.update
• On app server, file /opt/app/lib/app-0.0.1/priv/static/bar (using your app name and version)

Expected: "bar: No such file or directory"
Occurred: "bar: ASCII text"

Is it possible to suppress the banner text for ssh connections? It produces a lot of noise in the output. I was hoping perhaps quiet_mode or something can be passed to Erlang's ssh client, but I haven't spent much time researching.