Document the usage of the CLI following a step by step case for each branch (e.g. dev, prod).

Currently the update command copies to the venv or the container volume the logo file, with a hardcoded name to a hardcoded path. In order to allow further customization it should read the contents (e.g. list) and update all accordingly.

Change from --dev and --prod by --local and --containers.

Allow regenerating assets (install/collect) without having to install all dependencies.

This could be added by adding a flag and splitting the boostrap script.

When running in development mode and without --bg, the containers are launched blocking the process. The flask application never gets to run.

Use popen piping the logs instead of call here.

Easier to explain by listing the reproducing factors and then detailing what is happening:

Steps to reproduce:

1- install invenio-cli in a virtualenv and activate it
2- (virtualenv)$ invenio-cli init --flavour=RDM --local
3- cd <project>
4- (virtualenv)$ invenio-cli build --pre --local (now have Pipfile and Pipfile.lock for the instance venv)
5- (virtualenv)$ invenio-cli assets --local
Expect:
6- assets to be collected and built
Get:
6- no assets collected or built

Running (virtualenv)$ pipenv run invenio collect and (virtualenv)$ pipenv run invenio webpack buildall does work.

This applies to any of the --local commands.

My theory is it's because the created application (create_cli) is from the invenio-cli virtualenv context rather than the instance virtualenv context so none of the config is setup and there the commands are not operating on the right targets.

Core of the problem is that invenio-cli and the instance's installed dependencies are in different virtualenvs. This happens even if PIPENV_IGNORE_VIRTUALENVS is NOT set, because pipenv associates a Pipfile to a virtualenv: installing invenio-cli with pipenv will create a Pipfile that is different than the Pipfile invenio-cli init creates, so pipenv creates a new virtualenv to match that Pipfile everytime.

WORKAROUND
(extracted from comment below)

Say you have already installed invenio-cli through pipenv:

1- Run the init command to get the project laid out: pipenv run invenio-cli init --flavour=RDM
2- cd <project folder>
3- Install invenio-cli as part of the new Pipfile: pipenv install invenio-cli (check that it is there in the generated Pipfile)
4- Run the rest of the invenio-cli --local commands starting with: pipenv run invenio-cli build --local --pre

Because of this song and dance, we recommend these instructions for developing locally: https://github.com/ppanero/invenio-rdm-extension-demo/wiki

Verbosity in dev mode does not work as expected. Since Popen is being used, logs should be polled somehow when verbose is specified.

The CLI should support the initialization of different types of repositories:

• ILS
• RDM

Currently, it is using webpack buildall from flask-webpackext. It would be nice to split the npm installation from the assets building (e.g. add npm flag?).

Is it possible to avoid courtesy notices from pipenv?

The init command creates a .invenio file in order to store some configuration. If the file already exists, it fails after the cookicutter process, it should check it beforehand.

It should be Copyright (C) 2019 Northwestern University. instead of # Copyright (C) 2019 Northwestern University, Galter Health Sciences Library & Learning Center.

Automate pushing releases to PyPi

Find an appropriate name for the repository. It has been decided IRL to name the module as invenio-cli.

If the folder generated by cookiecutter already exsits, it thows an exception. It is not possible to control the flow (get the name of the folder) before cookicutter finishes. The exception should be controlled and presented to the user in a friendly way.

Create a command to create the starting file/folder structure.

• invenio.cfg: This file would contain all the configuration variables.
• pipenv: This file would contain the python dependencies along with each corresponding versions. Note: take into account some modules might be in pre, and therefore installed from GitHub.
• static/: This folder would contain all the CSS and potential static content (e.g. images).
• templates/: This folder would contain the custom (Jinja?) templates.
• Dockerfile[-base]: Description of the base image (python dependencies) and the application image.
• docker-compose.yml: This file would contain the service components architecture and orchestration. Note: It might need to differ prod from dev.

It should execute cookiecutter-invenio-rdm from Cookiecutter's programmatic API.

Investigate if there is any other option to run the cli commands from invenio from within invenio-cli code. Currently we are using current_app.test_cli_runner() but it is meant for testing.

encapsulate logging semantics in one object:

• get rid of loglevel since is not used
• encapsulate logfile and verbose.

Document the usage of the CLI to deploy and customize an RDM instance as of the specified Roadmap.

When outside the directory with the .invenio file. Most invenio-cli commands output:

ERROR:root:No flavour specified.

which is not as clear as Missing .invenio file in current directory. Use that error message instead.

Flask commands from invenio get shown when using invenio-cli --help.

As a hoster/developer, I want a command to spin up the containers from the Docker images using docker-compose (docker-compose.yml for prod and docker-compose.dev.yml for dev). Both use services from docker-services.yml

Investigate: Only docker-compose file version 2.2 is supported in RHEL 7.6 so it would be nice to have this version used i.e. double-check if version 2.3 is really needed. Note: version 3.x is not used because extends is not supported anymore.

Why? The CLI is crashing in local mode due to a old/ancient dependency on cookiecutter in invenio-base (see inveniosoftware/invenio-base#146). It sets no upper limit, while invenio-cli requires <1.7.0.

Invenio CLI is installed correctly, however after build --local cookiecutter gets installed with a higher version (1.7.0) that collides with the setup.py of invenio-cli and therefore no other command works.

Note: This might only happen if you install both invenio-cli and the local instance in the same venv, which is the case in some cases.

As per RFC#4 the flavour should only be specified in the init command. Therefore, we need a way of storing this choice.

However it is still under discussion if the flavour value would be received in the command, or it will be asked to the user as an interactive step (e.g. cookiecutter).

RIght now invenio-cli --version prints

Python 3.6.8
Flask 1.1.1
Werkzeug 0.16.0

It should print:

invenio-cli <version> 

Currently the volume content is not updated.

related to: inveniosoftware/cookiecutter-invenio-rdm#3

• Starting containers in background and not forwarding output (just print a message)
• Start app in background (just print a message). Needs saving PID for sigint hook.

Text by @fenekku

After some thoughts:

It actually might be that we have been thinking about this incorrectly: there shouldn't be 2 parallel contexts for which each command is implemented. Maybe it's more a sequence of commands and --prod is really just the last one or two. Something like this:

+------------------+     +---------------------+   +-------------------+      +--------------------+         +--------------------------+       +---------------------------------+
|                  |     |                     |   |                   |      |                    |         |                          |       |                                 |
| invenio-cli init +---> | invenio-cli service +-> | invenio-cli setup +----> | invenio-cli server +-------> | invenio-cli containerize +-----> | invenio-cli server --containers |
|                  |     |                     |   |                   |      |                    |         |                          |       |                                 |
+------------------+     +---------------------+   +-------------------+      +--------------------+         +--------------------------+       +---------------------------------+

Then being clear about the workflow will help users understand the 2 main use cases:

1- I want to just see what InvenioRDM gives me out of the box: I go through invenio-cli init, invenio-cli containerize and invenio-cli server --containers so I can finally see the instance running with no configuration on my side.

2- I am ready to start configuring and iterating: I do the first commands only and in particular the first invenio-cli server (precise names are not important in my explanation). This way my development flow is faster and I can see my changes but at the cost of more installation locally (might need to have header files and so on...).

This would make the usage of the cli more clear, remove the need for us to implement weird / necessarily slow commands and provide a better experience overall. What do you think @ppanero ? (Not saying we should switch now either if we decide to go this way)

Possible references:

• cookiecutter/cookiecutter#784
• cookiecutter/cookiecutter#848

New text

Integrate with docker-services-cli, by using the programmatic API not calling commands.

The services command, in invenio-cli/cli.py:services, should call docker-services-cli/services.py:services_up.

Note that is should set the corresponding environmental variables to be able to boot up the corresponding services (e.g. postgresql vs MySQL). The same goes for service version. One idea would be to use the data saved in .invenio when bootstraping.

This would result on the time.wait() being removed.

Open question

• Does it make sense to remove the docker-compose files form the cookiecutter now? We have helm charts for deployment and docker-services-cli/docker-services.yaml for the rest of the cases.

Change --dev by --local and --prod by --containers.

The end result of the initialization:

• user input to cookiecutter form
• list of generated files

Should be part of the .invenio file in order to allow easier updates.

As a hoster/developer, I want a command to clean up the development environment and start from scratch.

• Destroy container images
• Destroy data volumes

The environment variable: export FLASK_ENV=development should be set somehow when running server --local. This allows the debugger and other niceties to be there for developers.

Modify behavior to take into account if a command fails.

Steps to reproduce error:

0- install invenio-cli globally (via sudo pip install invenio-cli or more recommended via pipx)
1- $ invenio-cli init --flavour=RDM
2- $ cd <your project short name>
3- $ invenio-cli build --pre --dev --verbose
Expected
4- assets are collected and built
Got
4- Console output:

$ invenio-cli build --pre --dev --verbose
Building RDM application...
Locking dependencies...
Bootstrapping server...
Checking that dependencies are locked...
Installing dependencies...
Collecting assets......
Usage: cli [OPTIONS] COMMAND [ARGS]...
Try "cli --help" for help.

Error: No such command "collect".

Building assets......
Usage: cli [OPTIONS] COMMAND [ARGS]...
Try "cli --help" for help.

Error: No such command "webpack".

Creating statics folder...
Statics directory already exsits...
Creating development services...

This is because

@with_appcontext
def build_assets(log_config):
    """Build assets."""
    cli = create_cli()
    runner = current_app.test_cli_runner()
    # Collect
    run_command(cli, runner, "collect -v", message="Collecting assets...",
                verbose=log_config.verbose)

    # Build using webpack
    run_command(cli, runner, "webpack buildall",
                message="Building assets...", verbose=log_config.verbose)

calls run_command directly rather than subprocess.call(command, stdout=logpipe, stderr=logpipe) where command = ['pipenv', 'run', 'collect', '--verbose'] much like _bootstrap_dev .

Maybe this should be merged with #24 (and remove "good first ussue" from that one)

When running the commands, the containers are always started and stopped. It would be nice if the commands did not stop the containers if they were already running. Not only does this speed up the experience, it also is more convenient: I can run invenio-cli server in 1 terminal tab, run invenio-cli demo in another and not have to spin the containers back up.

• Enable logging #126
• Customize location: As default behaviour we can ask the user where to dump the logs. Once the folder has been chosen it is created if not exisiting (would find permission errors beforehand). Then the folder path is dumped into the .invenio conf
• Logging level: investigate the possibility of adding level filtering
• Ensure both containerize and development logs are piped correctly. Verbosity in dev mode does not work as expected. Since Popen is being used, logs should be polled somehow when verbose is specified.
• The old implementation had a problem of not closing the pipes, be aware of it #94

As default behaviour we can ask the user where to dump the logs. Once the folder has been chosen it is created if not exisiting (would find permission errors beforehand). Then the folder path is dumped into the .invenio conf

Custom templates must somehow find their way in <instance path>/templates/ .

For --local environment, 2 potential avenues:

• copying
• symlink <---- Chosen

For --containers just copy the folder at image creation. (in view of new workflow)

Part of inveniosoftware/invenio-app-rdm#6

As hoster/developer, I want to have a command to:

• lock the Pipenv dependencies
• generate the Docker image from the dockerfile(s). Image can be base / dev / production.
  Investigate: Are separate Dockerfile and Dockerfile-base files necessary?

Question: Do we want an invenio-app-rdm image in a registry to help?

As a hoster developing or testing out my instance locally before deploying, I want to have development configurations supported while knowing that production configuration will override them in production only, so I can test out configurations easily without having to modify invenio.cfg from dev to prod and back all the time.

When running the --prod setup, getting the instance path for creating the default files location, shouldn't be running pipenv run invenio shell -c "print(...)" as a normal subprocess command in the host system, but instead run invenio shell -c "print(...) with docker exec ... inside the container.

Inside the RDM flavour, support the creation of the repository according to different profiles:

• BioMed Repository
• Generic Repository

If the flag -v is not given, logs should be piped to a log file to avoid the insane amount of logs showed on the screen.

Create a CLI command to flush redis in the appropriate module (invenio-app?).

Currently we are using:

invenio shell --no-term-title -c "import redis; redis.StrictRedis.from_url(app.config['CACHE_REDIS_URL']).flushall(); print('Cache cleared')"

As a single command, however this should be delegate and run something similar to invenio redis flush.