getdkan / dkan-tools Goto Github PK

It would be nice to reproduce our traditional linting workflow in CI locally. IE, to lint only the files we're working on. The CI workflow checks the PR and lints the files that have changed; a local command could accept a base branch as argument, ask git to compare and give a list of modified paths, and lint that list of files.

See guide for upgrade tips.

We now have .circleci working for some projects to use dkan-tools. In the readme, document how to set this up on any project site, and especially how to skip tags or features, or only run a specific tag, in circle behat tests

• Add a Custom Behat Test
• Run Client Tests Locally
• Override an Upstream Behat Suite Configuration
• Getting Content onto a Probo Site

blocked by #20

Need equivalent of ahoy dkan create-QA-users

  create-qa-users:
    usage: Create users for each core role for QA purposes
    cmd: |
      ahoy drush "user-create sitemanager [email protected] --password=sitemanager"
      ahoy drush "user-add-role 'site manager' --name=sitemanager"
      ahoy drush "user-create editor [email protected] --password=editor"
      ahoy drush "user-add-role editor --name=editor"
      ahoy drush "user-create creator [email protected] --password=creator"
      ahoy drush "user-add-role 'content creator' --name=creator"

I want to have a safe development environment that has all the modules I know I need enabled.

When I pull the production DB to my dev environment, I want operations specific to that transition to happen before I start working on my local environment.

When I push new code and the CI server pulls the code to run tests, I want a second set of operations specific to that transition to happen before tests run.

I want a minimum of confusion about where environment switching logic occurs.

Create a second ticket with full implementation details and acceptance criteria.

Child of #16

The original goal for this project was that docker would be the only requirement. There should be a way to download a built version (with autoload and vendor folder).

User Story
The README doc is getting too long and hard to use. Let's upgrade this to a https://www.mkdocs.org/ -like format.

Acceptance Criteria
Documentation is broken into clear sections.

• Add a contributed library
• Add a contributed module
• Add a custom module
• Build a “Contributed” or “DKAN” Module in a Client Project
• Patching or “Overriding” DKAN
• Update a contributed module
• Update to Latest DKAN
• Temporarily disable a module for an environment
• Configure Secure Pages
• Enable or Disable a module
• Change local container settings
• Setting up a project locally

The standard procedure for running tests in DKAN implementation projects ("clients") will be to run only tests that have a @smoketest tag (as opposed to running the full suite on PRs against DKAN repo).

Document how to set this up in a new project in the README using circle config. Also document other use-cases of filtering tests.

Fill out after completing #2

Child of #16

Right now, if a command is not able to continue but has not run a task that produced an error yet, we are throwing an exception. This standard should be replaced with something that still displays an error on the terminal but 1) would exit if calling the command directly, or 2) would return an object that communicates the error state similarly to returning a task result with an exit code of 1.

Child of #19

A first crack at testing suite for DKAN Tools

Child of #19

Currently there is a deployment script built into DKAN starter. The environment switch is part of this, but the deployment script is broader and run in all environments upon deployment, including steps like registry rebuild, updb, etc. Some equivalent of this is still needed and must work with Acquia deployment hooks, but the current architecture is very hard to understand and could be greatly simplified.

Goals:

• Reduce complexity
• No ruby
• Run by Acquia hooks but not acquia-dependent
• Move more things from deployment script to install files
• Allow for custom deployment scripts?

Implementation?

Child of #16

Acceptance Criteria

• Adding a custom .htaccess file to the /src directory would override the docroot/.htaccess file

The DKAN namespace is not really well-defined and we can reorganize the commands to be more intuitive. Also, there is nothing that says this needs to work only for DKAN, let's just use profile when necessary.

This is the current list of dktl commands:

  composer                   Proxy to composer.
  drush                      Run drush command on current site.
  help                       Displays help for a command
  init                       Initialize DKAN project directory.
  list                       Lists commands
  phpcbf                     Proxy to phpcbf.
  phpcs                      Proxy to phpcs.
  test                       Test some things.
 dkan
  dkan:deploy                Performs common tasks when switching databases or code bases.
  dkan:get                   Get DKAN.
  dkan:install               Run DKAN DB installation. Pass --backup (-b) to restore from last backup.
  dkan:make                  Run the DKAN make file and apply any overrides from /config.
  dkan:restore               Restore files and database.
 docker
  docker:compose             Run a docker-compose command. E.g. "dktl docker:compose ps".
  docker:surl                Display the secure (https) web URL for the current project.
  docker:url                 Display the web URL for the current project.
 drupal
  drupal:install-min         Run Drupal minimal installation script. Takes mysql url as optional argument.
  drupal:make                Run make for Drupal core.
  drupal:remove-git-ignores  Remove all gitignores from docroot.
 drush
  drush:uli                  Run "drush uli" command with correct ULI argument.
 init
  init:cicle-ci              Generates basic configuration for a DKAN project to work with CircleCI.
  init:host                  Initialize host settings.
  init:probo-ci              Generates basic configuration for a DKAN project to work with ProboCI.
 test
  test:behat                 Runs DKAN core Behat tests.
  test:init                  Initialize test folders and install dependencies for running tests.
  test:lint                  Preconfigured linting for paths inside of the repo.
  test:lint-fix              Preconfigured lint fixing for paths inside of the repo.
  test:phpunit               Runs DKAN core PhpUnit tests.

Let's change to:

  composer                   Proxy to composer.
  deploy                Performs common tasks when switching databases or code bases.
  drush                      Run drush command on current site.
  help                       Displays help for a command
  init                       Initialize DKAN project directory.
  list                       Lists commands
  make                  Run the Drupal and profile make files and apply any overrides from /config.
  phpcbf                     Proxy to phpcbf.
  phpcs                      Proxy to phpcs.
 profile
  profile:get                   Get the profile.
  profile:install               Run profile DB installation.
 docker
  docker:compose             Run a docker-compose command. E.g. "dktl docker:compose ps".
  docker:surl                Display the secure (https) web URL for the current project.
  docker:url                 Display the web URL for the current project.
 drupal
  drupal:install-min         Run Drupal minimal installation script. Takes mysql url as optional argument.
 init
  init:cicle-ci              Generates basic configuration for a DKAN project to work with CircleCI.
  init:host                  Initialize host settings.
  init:probo-ci              Generates basic configuration for a DKAN project to work with ProboCI.
restore
  restore:db               Restore the db from a remote or local backup
  restore:files              Restore the files dir(s) from a remote backup
test
  test:behat                 Runs DKAN core Behat tests.
  test:init                  Initialize test folders and install dependencies for running tests.
  test:lint                  Preconfigured linting for paths inside of the repo.
  test:lint-fix              Preconfigured lint fixing for paths inside of the repo.
  test:phpunit               Runs DKAN core PhpUnit tests.

This may not require any changes to dktl tool, but rather creating a bash script in dkan's .circle folder that makes the paralelization work. Perhaps a helper PHP function is needed to recreate some of the logic currently in the ruby scripts.

Acceptance criteria

• DKTL test commands are used to run tests on circle
• Circle paralelization logic is either in a bash script under .circle/ or in the PHP app... or take another stab at using circle 2 API?

Work

Work has been happening both in dkan-tools and in the dkan repo:
https://github.com/GetDKAN/dkan/tree/dktl-circle
https://github.com/GetDKAN/dkan-tools/tree/exit

Work Left

1. In dkan-tools, we have some exit calls in our dktl bash script that are too early and are bypassing relevant code. We want to capture the exit code from the relevant commands and exit later with the appropriate code.
2. In dkan, I started to work with circleci's cli-tool to parallelize our test runs, but the work is not finished, we want to be able to run both phpunit and behat tests in parallel.

Robo provides a way to handle temporary directory. In Utils there is also a mechanism. They are both being used. Lets stick with the Robo mechanism for now.

Can't create dir in docroot to symlink to? Needs some experimentation but hopefully an easy fix.

Once our images have been updated, switch to using the new PHP7 container for web server in DKTL.

See GetDKAN/dkan#2319

Acceptance criteria

• If a dktl command fails with the message ERROR: No container found for cli_1 this should be detected by the bash script
• If this occurs, the script it should run docker-compose up -d
• After that command completes, it should try again. If the command fails with the same error message, it should quit

The two main dependencies have BSD-2 and MIT licenses, which means we should be able to release this with a GPL license. Do some research to see if robo's dependencies are all compatable and add a LICENSE.txt.

Child of #19

It would be good to investigate why so many tests are failing with the basic CircleCI configuration. It could be connected to #54, so lets make sure we address that first.

User story

I want patches to be easy to apply as part of the make process
??I want to easily generate patches when I have modified the Drupal or DKAN codebase??

Acceptance Criteria

• I can apply arbitray patches to the docroot

When using the dkan:restore command, either uncompressed, zipped or tarball/gzipped files should work for db dumps or files.

Child of #19

After some discussion, we have settled on the following structure for a dktl project root:

dktl.yml
dkan/
src/
  modules/
  themes/
  make/
  site/
  tests/
docroot/

Update the init command to implement this site structure. Also update drupal:make commands to make symlinks correct.

Child of #16

nice to haves

  vnc:
    usage: Prints the project VNC URL
    cmd: |
      VNC_PORT=`ahoy docker compose port browser 5900|cut -d ':' -f2`
      if [ "$AHOY_CMD_PROXY" = "DOCKER" ] && [ "$(ahoy docker proxy-running)" = "1" ]; then
        echo "$(ahoy docker get-hostname):${VNC_PORT}"
      else
        if [ -z "$DOCKER_MACHINE_NAME"  ]; then
          echo "`ahoy docker web-host`:${VNC_PORT}"
        else
          echo "`docker-machine ip default`:${VNC_PORT}"
        fi
      fi
  cleanup:
    usage: Clean ups docker unused images and volumes. See http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/.
    cmd: |
      EXITED=`docker ps -a -q -f status=exited`
      DANGLING=`docker images -f "dangling=true" -q`
      if [ ! -z "$EXITED" ]
      then
        docker rm -v $EXITED
      fi
      if [ ! -z "$DANGLING" ]
      then
        docker rmi $DANGLING
      fi
      docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

#9 introduces a chown command at the end of every run of the bash script that changes ownership of any root files to the current user. This addresses the common problem in docker that files created by the cli container appear to be owned by "root" to the host machine.

The solution needs to be environment-specific, as one would want to use the chown command in their local environment but not in the CI.

Acceptance criteria

• An environment variable DKTL_CHOWN is checked before running the chown command. If set to FALSE it will not run.
• The environment variable is documented.

We need some wrappers around Drupal coder commands for dktl.

Acceptance criteria

• A dktl test:lint command runs the normal phpcs Drupal linting command
• A dktl test:lint-fix command runs phpcbf to fix fixable issues and outputs the diff.
• Linting commands accept an array of paths

Child of #19

Once GetDKAN/dkan#2643 is complete, remove the DKAN Tools module from the assets directory, and remove all references to it from commands.

Add basic behat and phpUnit commands.

Acceptance Criteria

• Possible to run full behat suite or individual tests
• Possible to run full PHPUnit suite or individual tests

Child of #16

Add two scenarios/tutorials to the README:

1. Starting a DKAN project from scratch
2. Migrating a DKAN Starter project to DKAN Tools

We store DB dumps and file archives in S3, and need to use S3 tooling to get at these private files.

The tool MVP is that we can fully deploy and test a DKAN release using the tool, without a dkan_starter release

Parent of #42
Parent of #10
Parent of #34

Connecting to our memcache container takes a bit of configuration. We should add it to settings.docker.php.

Acceptance criteria

• All modules currently managed through DKAN starter are added to DKTL make files or DKAN
• CircleCI and Probo work for PRs
• Confirm local development environment and CI can load production DB and files and deploy

Most of the commands provided by dkan-tools will work with just the cli container, yet we download all of the images and build all the containers no matter what we are doing.

It would be useful to run, only the necessary containers for a given command, this would make the tool more accessible (less bloated).

branch: dkan2

PHP Fatal error:  Uncaught Error: Call to undefined method DkanTools\Drupal\V7\InitCommands::directoryAndFileCreationCheck() in /usr/local/dkan-tools/src/Drupal/V7/InitCommands.php:43
Stack trace:
#0 /usr/local/dkan-tools/src/Drupal/V7/InitCommands.php(23): DkanTools\Drupal\V7\InitCommands->createDktlYmlFile()
#1 [internal function]: DkanTools\Drupal\V7\InitCommands->init(Array)
#2 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/CommandProcessor.php(235): call_user_func_array(Array, Array)
#3 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/CommandProcessor.php(181): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#4 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/CommandProcessor.php(150): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#5 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/AnnotatedC in /usr/local/dkan-tools/src/Drupal/V7/InitCommands.php on line 43
ERROR: Uncaught Error: Call to undefined method DkanTools\Drupal\V7\InitCommands::directoryAndFileCreationCheck() in /usr/local/dkan-tools/src/Drupal/V7/InitCommands.php:43
Stack trace:
#0 /usr/local/dkan-tools/src/Drupal/V7/InitCommands.php(23): DkanTools\Drupal\V7\InitCommands->createDktlYmlFile()
#1 [internal function]: DkanTools\Drupal\V7\InitCommands->init(Array)
#2 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/CommandProcessor.php(235): call_user_func_array(Array, Array)
#3 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/CommandProcessor.php(181): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#4 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/CommandProcessor.php(150): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#5 /usr/local/dkan-tools/vendor/consolidation/annotated-command/src/AnnotatedC 
in /usr/local/dkan-tools/src/Drupal/V7/InitCommands.php:43

Steps to reproduce

• Switch to V7 on /bin/app.php line 6
• Run dktl init

I want to be able to clone a project site and pull down the DB for development.

Acceptance Criteria

• DB download location defined in dktl.yml file
• Some equivilant to "site up" - download db, run updates, switch environments and run deploy script

Somewhat arbitrary list of features moving toward a 1.0 release.

Acceptance criteria

• A dkan:test-custom command will run custom site tests

Child of #16

I want to pull a site database and run deployment commands, so that I can have a local sandbox that mirrors production.

In ahoy, this is accomplished with the site up command.

Acceptance criteria:

• Project configuration tells command where to find database
• imports database
• download the files

Child of #19

Acceptance Criteria

1. A project can define its own commands that are registered and made available by the tool

Child of #19

The new Datastore module should make the Feeds module unnecessary. Let's include an update hook that will disable the module.

We probably need to keep the module in the codebase for a few more releases so that it doesn't break.

The 'dkan:restorecommand restores the db from a dump at a URL. Thedkan:installcommand restores a db backup from the /backups folder if given a-b` switch. These could be consolidated to make all this more intuitive.

Acceptance Criteria

1. Take the restore command out of the dkan namespace.
2. Make the restoreDb() command public instead of private, so we can do either dktl restore for both or dktl restore:db for just the db.
3. Make the restoreDb() method restore from a URL if given the argument.
4. If the argument is not a URL, assume it's a filename and look for it in the /backups folder.
5. If there is no argument, look in the /backups folder. If there is a single file in there, use that. If there are multiple, present the list as an interactive option.

For instance:

$ dktl restore:db

The following backups are available to restore:

[1] last_backup.sql
[2] some_other_backup.sql.gz

6. .sql, .sql.gz and .sql.zip files should all be supported

It would be good to include an example make file with some commonly-used add-on modules, such as memcache and the Acquia modules. This should go in the documentation.