zalando-stups / lizzy-client Goto Github PK

We should remove or change the Kio message in the end of the deploy. Currently it shows:

You can approve this new version using the command:

    $ kio version approve lizzy v1

Which lead to think the new version was registered already in Kio, but that changed in the release2.0

Now lizzy traffic does not wait for the stack to become status UPDATE_COMPLETE, which means next steps in pipeline that assume the traffic is switched will fail or perform actions to the wrong stack. Please make it possible to make lizzy wait for this state before returning (by default would be the best, but an option would be good too).

I had 2 stack already deployed by lizzy.

When I used the command in a third time

$ lizzy create --keep-stacks 1 --traffic 100 my-app.yaml 1.0

I saw in logs:

Wating for new stack.. . . . . . Stack was removed before deployment finished.

Cloud Formation events also shows that stack was immediately deleted by lizzy after deployment

Hi,

I tried to use the ELB V2 in my senza.yaml file, but lizzy couldn't deploy my application. I got the following error:

Generating Cloud Formation template.. EXCEPTION OCCURRED: Component "Senza::WeightedDnsElasticLoadBalancerV2" does not exist

I've executed lizzy-client on Jenkins at 9:59am. Three minutes later, at 10:01am I got following ERROR in logs:

Wating for new stack.. . .FATAL: no longer a configured node for ip-10-144-114-64.z.aws
java.lang.IllegalStateException: no longer a configured node for ip-10-144-114-64.z.aws
at hudson.model.AbstractBuild$AbstractBuildExecution.getCurrentNode(AbstractBuild.java:456)
at hudson.model.AbstractBuild$AbstractBuildExecution.reportBrokenChannel(AbstractBuild.java:807)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
at hudson.model.Build$BuildExecution.build(Build.java:205)
at hudson.model.Build$BuildExecution.doRun(Build.java:162)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
at hudson.model.Run.execute(Run.java:1744)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)

But seems deployment was done, because I see my app "seo-service-staging" successfully deployed on AWS

Would be great to have an option like "-v" or "--verbose" or "--debug" for lizzy-client in order to investigate issues. Currently I'm getting "LIZZY:ERROR EXCEPTION OCCURRED: 1" on server and don't have an ability to investigate a root cause. In "verbose" mode lizzy-client would log parameters, stages, statuses and so on.

Currently we have 3 versions of application deployed by lizzy:

When I try to route a traffic to the v3 of the app using lizzy traffic app v3 100 I'm getting following error:

Requesting traffic change.. EXCEPTION OCCURRED: 400 Client Error: BAD REQUEST
Traceback (most recent call last):
  File "/usr/local/bin/lizzy", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/lizzy_client/cli.py", line 236, in traffic
    lizzy.traffic(stack_id, percentage)
  File "/usr/local/lib/python3.4/dist-packages/lizzy_client/lizzy.py", line 89, in traffic
    request.raise_for_status()
  File "/usr/local/lib/python3.4/dist-packages/requests/models.py", line 851, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST

To reflect senza interface.

Since lizzy-client is a command line tool we should think about vendorizing the dependencies to prevent users having problems with conflicts. Most of the time the users do not have enough knowledge of Python development to understand how to solve those conflicts.

To make lizzy client easier to configure in Jenkins it should receive the configuration from the environment,

Make a traffic subcommand invoked as lizzy traffic to change traffic of lizzy stacks.

Consider making the syntax similar to senza.

Probably this issue zalando-stups/lizzy#135 should apply more to the client than the Lizzy agent.

When using Lizzy we pass the LIZZY_URL to the lizzy-client command line. If we put the wrong url, like https://lizzy-instance.example.com/ instead of https://lizzy-instance.example.com/api we get a "504 Gateway Timeout" error from the load balancer.

There should be a more user-friendly way to handle better that case.

Lizzy-client should support bumping the kio application version. This should be optional.

Relates to release2.0 branch

Output from Jenkins job:

Fetching authentication token.. . OK
Artifact checking is still not supported by lizzy-client.
Requesting new stack.. OK
Stack ID: hello-bus-v44
Waiting for new stack... . . .Deployment failed: Failed to get stack (0 retries left): HTTPError('404 Client Error: NOT FOUND for url: https://[edited]/api/stacks/hello-bus-v44',).
 EXCEPTION OCCURRED: 1
[AGENT] Generating Cloud Formation template..You are deploying an internet-facing ELB that will be publicly accessible! You should have OAUTH2 and HTTPS in place!
[AGENT]  OK
[AGENT] Creating Cloud Formation stack hello-bus-v44.. OK
Build step 'Execute shell' marked build as failure
Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered
Finished: FAILURE

It might be a problem in the agent side. Getting recently created stacks...

Current behaviour:

If for command lizzy create parameter --traffic is not passed, the traffic is switched entirely.

Desired behaviour:

If for command lizzy create parameter --traffic is not passed, the traffic is not switched at all.

Reasoning:
According to the principle of least surprise, a command called create should not do something else by default.

Make a list subcommand invoked as lizzy list to list lizzy stacks.

Consider making the syntax similar to senza.

We have commonly users reporting problems while trying to run pip install --upgrade and other issues (different python versions in the same machine/etc) related to the current way we recommend installing the lizzy-client command line tool. We should provide a self-contained installation for that in a .deb package that can easily be installed using apt-get install.

We have commonly users reporting problems while trying to run pip install --upgrade and other issues (different python versions in the same machine/etc) related to the current way we recommend installing the lizzy-client command line tool. We should provide a self-contained installation for that to be easily be installed using brew install.

Lizzy client should be able to read mint credentials from a mint bucket fetched with (berry)[http://docs.stups.io/en/latest/components/berry.html] to make the process of using it in Zalando's Jenkins instances seamless.

Improve the output by adding a verbose mode and a common function for fatal errors.

Lizzy client should display human readable error messages for common errors.

At the moment, the force flag, while specified in the CLI API, is not passed to the Lizzy Agent and its usage will not work, for instance if the instance already has traffic resulting in the following error message:

ERROR: Error executing command.
> command: senza delete --region eu-central-1 my_application ABCDEFG
> command.output: Error: Stack my_application-ABCDEFG has traffic!                    
Use --force if you really want to delete it

The corresponding TODO is here:

The corresponding API in Lizzy is here:
https://github.com/zalando-stups/lizzy/blob/4c39fce2617cc60be512a1f0587727b041d67e52/lizzy/api.py#L258

If LIZZY_URL has an extra / in the end, we should remove it.

The script should get an authentication token and use it to deploy a Senza yaml with Lizzy and wait until deployment finishes.

Senza supports named parameters, so in YAML file I can define something like

SenzaInfo:
  StackName: my-stack-name
  Parameters:
    - ImageVersion:
        Description: "Docker image version of seo-descriptions-api-staging."
    - ScalyrAccountKey:
        Description: "Scalyr (logs aggregator) account key."

and then use parameters in the YAML definition:

      TaupageConfig:
        application_version: "{{Arguments.ImageVersion}}"
        source: "myserver/teamid/app:{{Arguments.ImageVersion}}"
        scalyr_account_key: {{Arguments.ScalyrAccountKey}}

Seems that currently lizzy create doesn't have a support of named parameters. Would be nice to have it because otherwise we need to hardcode all template parameters into the YAML file (including sensitive data like keys and accounts)

Given:
In my senza definition file, the StackName is defined using an Arguments-variable:

StackName: "wps{{Arguments.StackNamePostfix}}"

Unfortunately, lizzy doesn't resolve the argument. I get the following error:

Requesting new stack.. EXCEPTION OCCURRED: 404 Client Error: NOT FOUND for url: https://lizzy-zprice-test.zprice-test.zalan.do/api/stacks

[AGENT] Stack not found: wps{{Arguments.StackNamePostfix}}-74ee2ddxb35

FAILURE: Build failed with an exception.

I'd like to see that lizzy can handle variables in stack names.

PS
I suppose this can be handle in the client.

For example the "--token_url" option should be changed to "--token-url"

Create a new command line argument to provide a configuration file.

The configuration file should allow setting the user, password, lizzy-url and token_url.

If some parameter is provided both on the configuration file and command line argument the command line argument shall take precedence. If any of the described options is missing in both then the script should exit with an error.

Related to zalando-stups/lizzy#162

Read config file from user config folder if not provided in the command line

In order to report issues or enhancements would be good to have a way to determine a version of dizzy-client installed in the system.

If someone installs lizzy-client on an environment where senza is already installed then lizzy-client would somehow overwrite senza and makes it unusable.

Authentication URLs and scopes should be configurable without forking Lizzy because they can be different for every server.

This depends on Connexion's Issue 1.

Besides the current option of setting the agent url with the LIZZY_URL env variable it should be possible to set the url using a CLI flag, for example, --remote.

Missing option param.

The current command should be a subcommand invoked as lizzy create.

Consider making the syntax similar to senza.

Code should not contain any company-specific settings/URLs:
https://github.com/zalando/lizzy-client/blob/master/lizzy_client/__init__.py#L40

Not necessary anymore.

Make a delete subcommand invoked as lizzy delete to delete lizzy stacks.

Consider making the syntax similar to senza.

Current behaviour:

When lizzy create is called without parameter --keep-stacks, all old stacks are deleted.

Desired behaviour:

When lizzy create is called without parameter --keep-stacks, all stacks are kept as they are.

Lizzy client gets in an infinite loop waiting stack that is not in progress to be ready to change traffic.

Requesting new stack.. OK
Stack ID: benetton-18
Waiting for new stack... . . . . . . . . . . . . . . . . . . . . . OK
Deployment Successful
Requesting traffic change.. OK
Deleting old stacks..[AGENT] Generating Cloud Formation template..You are deploying an internet-facing ELB that will be publicly accessible! You should have OAUTH2 and HTTPS in place!
[AGENT]  OK
[AGENT] Creating Cloud Formation stack benetton-18.. OK

 > benetton-16 current status is ROLLBACK_COMPLETE trying again later
 OK
Deleting old stacks.. OK

 > benetton-16 current status is ROLLBACK_COMPLETE trying again later
Deleting old stacks.. OK

 > benetton-16 current status is ROLLBACK_COMPLETE trying again later
Deleting old stacks.. OK

Solution

Should wait for stack only if the status is *_IN_PROGRESS.

To ensure future changes don't break lizzy client the project should have a high test coverage