Currently the Lmctl project packages include their contents in a "content" directory - this directory is not needed and blocks the packages from becoming Sol compliant.

We should no longer include the content directory in any future packages but maintain support for it in existing packages built with earlier versions of lmctl.

Version: 2.1

The env list command displays the following stack trace when no config file has been set:

Traceback (most recent call last):
  File "/mnt/c/dev/src/github/accanto/lmctl/env/bin/lmctl", line 11, in <module>
    load_entry_point('lmctl', 'console_scripts', 'lmctl')()
  File "/mnt/c/dev/src/github/accanto/lmctl/lmctl/cli/entry.py", line 23, in init_cli
    cli()
  File "/mnt/c/dev/src/github/accanto/lmctl/env/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/c/dev/src/github/accanto/lmctl/env/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/mnt/c/dev/src/github/accanto/lmctl/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/c/dev/src/github/accanto/lmctl/env/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/c/dev/src/github/accanto/lmctl/env/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/c/dev/src/github/accanto/lmctl/env/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/mnt/c/dev/src/github/accanto/lmctl/lmctl/cli/commands/env.py", line 24, in list
    ctl = ctlmgmt.get_ctl(config)
  File "/mnt/c/dev/src/github/accanto/lmctl/lmctl/cli/ctlmgmt.py", line 12, in get_ctl
    return ctl_config.get_ctl(config_path)
  File "/mnt/c/dev/src/github/accanto/lmctl/lmctl/ctl/config.py", line 179, in get_ctl
    config_path = CtlConfigFinder(config_path, 'LMCONFIG').find()
  File "/mnt/c/dev/src/github/accanto/lmctl/lmctl/ctl/config.py", line 155, in find
    raise ConfigError('Config environment variable {0} is not set'.format(self.potential_env_var))
lmctl.ctl.common.ConfigError: Config environment variable LMCONFIG is not set

This is inconsistent with other commands, which show a single line error to improve readability and to avoid leaking stack traces into the console (they should go to the logs instead):

Error: Failed to load configuration - Config environment variable LMCONFIG is not set

Is your feature request related to a problem? Please describe.
Tried to create a project with multiple nested Resources and set the driver param for all of them.

Command:

lmctl project create memcached-operator --contains Resource serviceaccount --contains Resource role --contains Resource role-binding --contains Resource crd-memcached --contains Resource controller --param driver kubernetes

Results in an error:

Error: Unexpected param 'driver' provided to request for 'memcached-operator'. Supported params []

The param is not recognised on the root project, as it's an Assembly, so the only way to do it is repeat the param for each Resource project.

Describe the solution you'd like
Need a simple way to declare global defaults to parameters for nested projects.

Additional context
Lmctl version 2.5.0

Is your feature request related to a problem? Please describe.
Assembly Templates are currently a PoC feature of ALM but we can add support in for PoC users.

Describe the solution you'd like
Assembly Templates to be supported in existing Assembly projects.

A template can be added as an assembly-template.yaml file to the existing Descriptor directory.

The template is optional - so if the project does not include a template then the Assembly project should still be usable.

When creating a new project, there should be a command line option to generate a template file. By default, this option should be disabled.

As templates are pushed to a bolt-on service, it should be possible to configure the port of this service in the existing LM environment of my lmctl config file.

The index.md in the lmctl docs talks about the differences between 2.1 and 2.0. This page needs to be updated to highlight differences in 2.2 and 2.1/2.0.

This page should also be updated to include links to other user guide pages.

It would be useful to include commands to onboard new VIM and Lifecycle drivers (for Brent). This API is public so it's a good candidate for an lmctl command and it would make Brent easier to use.

Version: 2.2.0.dev0

Run a command requiring a secure LM environment, without setting the password. It will attempt to prompt for the password.

Error:

Traceback (most recent call last):
  File "/home/dvs/github/accanto/lmctl/env/bin/lmctl", line 11, in <module>
    load_entry_point('lmctl', 'console_scripts', 'lmctl')()
  File "/home/dvs/github/accanto/lmctl/lmctl/cli/entry.py", line 25, in init_cli
    cli()
  File "/home/dvs/github/accanto/lmctl/env/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/dvs/github/accanto/lmctl/env/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/dvs/github/accanto/lmctl/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dvs/github/accanto/lmctl/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dvs/github/accanto/lmctl/env/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dvs/github/accanto/lmctl/env/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/dvs/github/accanto/lmctl/lmctl/cli/commands/vimdriver.py", line 35, in add
    vim_mgmt_driver = get_vim_driver_mgmt_driver(environment, config, pwd)
  File "/home/dvs/github/accanto/lmctl/lmctl/cli/commands/vimdriver.py", line 14, in get_vim_driver_mgmt_driver
    lm_session = ctlmgmt.create_lm_session(environment_name, pwd, config_path)
  File "/home/dvs/github/accanto/lmctl/lmctl/cli/ctlmgmt.py", line 44, in create_lm_session
    prompt_pwd = click.prompt('Please enter password for LM user {0}'.format(lm_session_config.username), hide_input=secret, default='')
NameError: name 'secret' is not defined

version: 2.3.0

Pushing a Resource project to Brent passes (lmctl shows no errors) however the Resource is not present in LM.

Try to refresh the brent RM from the LM UI and you will see a report displaying the failure to onboard this Resource as it has validation errors.

Potential Fix:
Lmctl should check the result of the RM refresh and display errors relating to the Resource being pushed.

Currently only Resources can be included as subprojects (in the VNFCs directory). This should be expanded to allow for Assemblies to be included.

Attribute error thrown when trying to push an invalid Descriptor, e.g:

description: descriptor for TestDuplicateKey
composition:
  a:
    type: resource::t_simple::1.0
  a:
    type: resource::t_simple::1.0

When creating Brent Resource projects it would be useful to support multiple types of Infrastructure and Lifecycle types, so we can generate the initial files for the intended driver.

This would need to be a configurable option on the create command, allowing us to configure the Inf/Lifecycle type for each Project and subproject to be created.

The value set on the root Project should be inherited by the subprojects.

Initial Support Types:
Infrastructure:
- openstack

Lifecycle:
- sol003
- ansible

In order to support SSL connections to (Brent) VIM and Lifecycle drivers, the onboarding of drivers needs to include an SSL certificate to allow communication with the driver. The lmctl commands should include an option to add a certificate during onboarding.

Version: 2.1

We should make use of click's ANSI color support to display errors in red and (potentially) warnings in a yellow/amber colour, so they stand out in the console.

Version: 2.1
When running project tests the progress of each execution is shown by printing a new line every X seconds. Instead we should look to use click.progressbar to display a progress bar on one line.

Add support for a default LMCONFIG file location at ~/.lmctl/config.yaml
When the LMCONFIG env var is not set (and any --config options have not been used) then lmctl will look for a file at this location, raising an error if not found.

Describe the bug
Lmctl v2.6.0/2.6.1 assume Kami is on the same host as TNCO API gateway, this is not always true in OCP environments.

Proposed Fix
Add configuration option for lm environments to set the kami address

Is your feature request related to a problem? Please describe.
To use lmctl as a python client library it needs additional API client implementations

Describe the solution you'd like
Improve and expand the range of the lmctl client so it can be used in future python projects as the API client for LM

Adding commands to LMCTL for adding/removing deployment locations would help developers through their CICD process.

support the brent APIs to manage shared keys

For CICD pipelines it would be useful if there was a command in the project group to perform common operations on the version of a project, such as:

• increment
• decrement
• append SNAPSHOT
• remove SNAPSHOT
• set to fixed value

This behaviour is often coded into pipelines.

Proposal

Command to set the version to a given value
lmctl project version set <version>

Append "-SNAPSHOT" to the end of the current project version
lmctl project version set-snapshot

Remove "-SNAPSHOT" from the end of the current project version (so it may be released)
lmctl project version set-release

Decrement version (where part is either M (major), m (minor), p (patch) or an index number
lmctl project version inc {part}
lmctl project version dec {part}

As Descriptors can now inherit from each other, there is an API to retrieve an effective version of a Descriptor, which will include all inherited properties. We should ensure this can be retrieved through the Python client.

The "project description" on the lmctl pypi repository here: https://pypi.org/project/lmctl/ is using the same description as the github repo which includes links that aren't valid for the pypi site. A description just for Pypi should be added that doesn't include the github specific links

Is your feature request related to a problem? Please describe.
PoC feature to include Type projects.

Describe the solution you'd like
A Type project is similar to an Assembly project, the descriptor uses the type:: prefix instead.

Version: 2.1.0

Currently environments can be listed with env list. It would be useful if a user could inspect the details of an environment from the command line, instead of having to find and open their config file, before searching through it's contents for the environment.

Proposal

Show the details of an environment:
lmctl env inspect {env_name}
lmctl env show {env_name}

When executing lmctl project build with the --project option set to a directory which is different to the current directory, you end up with the "_lmctl" build directory created in the current directory - rather than the root of the project.

This may have once been by design but we should update this, as keeping the build under the project directory seems more suitable.

Feature
With the completion of #56, we will have a more complete client library for interacting with TNCO environments. It would be good to have a framework in place to easily expand the number of commands available from the command line, to make use of this new client.

Notes:

• review if the current command syntax is best

Rather than specifying the username in the lmconfig file we should support providing it as a command line option AND prompting for it when not set (as we do for passwords).

2.1+ UI access will include a /ui subpath.
The API can also be configured to a subpath, so we should allow LM environments to include the path in the configuration file, otherwise we cannot access them with lmctl.

Pushing any pkg with lmctl pkg push results in this error

Currently all requests to TNCO (command line and/or client) are unverified. We need to add support to configure a certificate/ca bundle to be used to verify requests.

See: https://requests.readthedocs.io/en/master/user/advanced/#ssl-cert-verification

It should be possible to add the path or the contents of the certificates to each environment in the LMCTL configuration file.

For client access, we should allow the certificate to be passed on construction of the client instance.

Currently we can only retrieve a process by ID (https://github.com/IBM/lmctl/blob/master/lmctl/client/api/processes.py).

We have disabled the list API as it's not possible to get all processes, however it should be possible to get all processes associated to a given Assembly (by name or ID).

Project create command to support lifecycle/resource driver param option for the Kubernetes driver.

This will generate a kubernetes directory in the Lifecycle directory and be pre-filled with example files for a Resource.

Would be useful to expand the project command group to include a remove command which removes any artifacts from the given project in a target environment (essentially reversing a push)

lmctl project remove <env>

the command docs show a wrong command for
deployment location list
1 - Usage should read :

lmctl deployment list [OPTIONS] ENVIRONMENT

2- under Arguments: delete "Name"

Create an initial set of tests, and a framework to add further tests, which execute against a live TNCO environment.

• The tests should be runnable from a command line
• The tests should be written in Python (so no extra skills are required)
• The target TNCO environment should be configurable so we can run the tests from a pipeline or a user laptop

Version: 2.1.0

lmctl pkg push only supports pushing packages from the local file system. As packages are often stored in a central repository, this often means a CICD pipeline must be scripted to pull the packages locally first.

It may increase the usability of lmctl if it were able to accept a HTTP(s) path to a package and perform the get-then-push on behalf of the user.

Support packaging projects as a CSAR (Zip) instead of a TGZ.
Also allow a TOSCA-Metadata directory to be included in the package.

Changes have been trialled on the includeArtifacts branch.

• Infrastructure templates moved to Lifecycle directory
• Support Openstack driver directory structure (create, validate, build and push)
• New projects should generate Openstack directory under Lifecycle instead of templates in the Definitions directory
• Infrastructure template details in descriptor replaced with driver details in Create/Delete lifecycle and Queries section of descriptor
• Autocorrect templates by moving them to Lifecycle directory (on validate)
• Autocorrect descriptor refactoring
• Add autocorrect to pkg commands, so old packages can be still be used

lmctl project test executing all the behaviors even when a particular one is specified in the command using '--tests'

Steps to reproduce the behaviour:

use lmctl to execute a specific test :
the ones we tried : " lmctl project test --armname brent --tests "
" lmctl project test --armname brent --tests "
" lmctl project test --tests "

It executes all the tests from the behavior tests folder in the assembly.

using ALM2.2 and LMCTL2.6.0

Version: lmctl 2.1.0

Executing a project pull on a project with a descriptor that references a descriptor not in the same project results in an error:

--> Pull Sources
Pulling descriptor for test
Pulling descriptor assembly::test::1.0 from LM (https://192.168.56.100:8083)
Creating backup of descriptor ./Descriptor/assembly.yml
Saving pulled descriptor to ./Descriptor/assembly.yml
Traceback (most recent call last):
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/mutate/descriptor.py", line 57, in __convert_types
    resolved_project = self.config_references.resolve(project_for_descriptor_reference)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/source/config_references.py", line 20, in resolve
    return super().resolve(reference, self.resolution_map)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/reference.py", line 57, in resolve
    return ReferenceResolver(self.schema, resolution_map).resolve(reference)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/reference.py", line 93, in resolve
    return self.__resolve(reference, reference_parts, self.resolution_map)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/reference.py", line 105, in __resolve
    return self.__resolve(reference, reference_parts, next_value)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/reference.py", line 100, in __resolve
    raise NotResolvableError('Cannot find \'{0}\' in reference: {1}'.format(next_part, reference))
lmctl.reference.NotResolvableError: Cannot find 'resource::hello-world::1.0' in reference: $lmctl:/descriptor_mappings:/resource::hello-world::1.0:/project
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/bin/lmctl", line 10, in <module>
    sys.exit(init_cli())
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/cli/entry.py", line 23, in init_cli
    cli()
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/cli/commands/project.py", line 140, in pull
    exec_pull(controller, project, env_sessions)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/cli/commands/project.py", line 63, in exec_pull
    controller.execute(project.pull, env_sessions, pull_options)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/cli/lifecycle.py", line 252, in execute
    response = exec_func(*args)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/source/core.py", line 184, in pull
    return self.__do_pull(env_sessions, options, journal)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/source/core.py", line 188, in __do_pull
    pull_exec.PullProcess(self, options, journal, env_sessions).execute()
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/processes/pull.py", line 44, in execute
    return PullWorker(self.project, self.options, backup_tree, self.journal, self.env_sessions, self.references).work()
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/processes/pull.py", line 57, in work
    self.__pull_sources()
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/processes/pull.py", line 64, in __pull_sources
    self.project.source_handler.pull_sources(self.journal, backup_tool, self.env_sessions, self.references)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/handlers/assembly/assembly_src.py", line 241, in pull_sources
    self.__pull_descriptor(journal, backup_tool, backup_tree, lm_session, references)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/handlers/assembly/assembly_src.py", line 260, in __pull_descriptor
    descriptor = descriptor_mutations.DescriptorPullMutator(references).apply(descriptor)
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/mutate/descriptor.py", line 44, in apply
    self.__convert_types(descriptor.raw, 'composition')
  File "/mnt/c/Users/hardw/git/lmctl_2/lmctl/env/lib/python3.5/site-packages/lmctl/project/mutate/descriptor.py", line 61, in __convert_types
    self.journal.event('Cannot resolve reference: {0}'.format(element_type))
AttributeError: 'DescriptorPullMutator' object has no attribute 'journal'

Is your feature request related to a problem? Please describe.
Currently lmctl only supports authentication with a username/password

Describe the solution you'd like
Support configuration of an LM environment with client credentials (clientId/clientSecret) either instead of or in combination with username/password. The client credentials should be used to authenticate with ALM for any API requests.

Additional context
Apart from being more consistent with ALM authentication methods, this feature will also allow lmctl to be used more efficiently as a Python client library in applications requiring ALM API access.

Add support for the CreateOrUpgrade and Adopt intents.

Allow LMCTL to build and deploy projects for a Resource and not just Assemblies.

Error thrown

env_sessions = lifecycle_cli.build_sessions_for_project(project.config, environment, pwd, armname, config)
NameError: name 'project' is not defined

Version: 2.1.0/2.1.1

Attempting an lmctl project push on a project with a resource intended for Brent leads to:

--> Push Content
Removing descriptor resource::resource1::1.0 from LM (https://192.168.56.100:8083)
Descriptor resource::resource1::1.0 not found
Removing any existing Resource package named resource::resource1::1.0 (version: 1.0) from Brent: alm (https://192.168.56.100:8083)
 
===========================================================
Push - FAILED
        Request returned unexpected error: status_code=401

LM logs indicate the request is made without any user credentials, even though earlier API requests had them:

principal=anonymousUser, type=AUTHORIZATION_FAILURE, data={requestUri=/api/resource-manager/resource-packages/resource::resource1::1.0, httpMethod=DELETE, message=Access is denied

It would be useful to have a command added to the pkg group which printed details about a package to the console. It would allow the user to inspect the contents of a pkg before pushing it to their environments.

Proposal
lmctl pkg inspect {pkg_path}

Information printed to the console:

• Name and version of the Project
• Type of project
• Names of subprojects
• Names of all descriptors that will be created in an environment if pushed

Currently configuration of a TNCO/LM environment can be bloated, like below:

lm:
  host: somehost
  port: 443
  protocol: https
  secure: True
  username: jack
  auth_host: someotherhost
  auth_port: 443

In many cases we no longer need the auth_ properties (and if we support client credentials #55 we can move people away from using the UI host).

In many cases, needing to split the host, port and protocol is unnecessary work for the user. They were traditionally split to provide defaults to any omitted auth_ values but as stated earlier, we want to move away from using these properties anyway.

We could reduce the need for host, port and protocol to a single address property. Maintain an auth_address for environments that intend to use legacy username/password authentication (pre-client credential support).

LM is replacing the distinction between VIM and LifecycleDrivers with a single type "Resource Driver". A new resourcedriver command should be created to support onboarding a Resource driver.

The current vimdriver and lifecycledriver commands should be deprecated. When used they should display a warning message but then forward to the resourcedriver command on behalf of the user.