I've been trying to debug some issues I'm having with running cron on the platform. I have a custom bin-dir to install composer binaries in /app/bin/ instead of /app/vendor/bin because that is what we're used to. However, this seems incompatible with the platform.

Firstly #376 setup a hardcoded alias to /app/vendor/bin/drush but there are a lot more problems:

There's a global drush script installed in /usr/local/bin/drush which is an ancient version (8.4.12)

[canceraus3]master@cli-drupal:/app$ which drush
/usr/local/bin/drush

[canceraus3]master@cli-drupal:/app$ /usr/local/bin/drush --version
Drush Launcher Version: 0.10.2
 Drush Version   :  8.4.12 

This seems to be being used at some points (ignoring the alias).

But the alias is used when running just drush

[canceraus3]master@cli-drupal:/app$ drush --version
bash: /app/vendor/bin/drush: No such file or directory

This seems to be causing issues with the custom cron job wrapper https://www.govcms.support/support/tickets/26269

It should be possible to set a custom bin-dir on this platform.

We should also remove the /usr/local/bin/drush file.

There is an error message in container logs on GovCMS:

/lagoon/entrypoints.sh: /lagoon/entrypoints/20-no-robots.sh: line 7: can't create /app/web/robots.txt: Permission denied

This script seems to be part of the GovCMS base image.

I'd like to propose a few tweaks to the Nginx helpers:

.docker/images/nginx/helpers

1: Can we rename these so they have numerical prefixes inline with the Amazee nginx config. This keeps things consistent and the load order predictable.

2: The Amazee location blocks all have the ability to include additional conf.d files before and after the location definition. This is really nice if you want to do extra stuff in them. Propose that for those helpers declaring a location, that include directives are added for this purpose and to bring them inline with the Amazee locations, for example:

# Only cache binary files for 30 minutes. These are the default allowed file extensions uploads that are not images.
location ~* ^/sites/default/files/.+\.(pdf|doc|...|webp|webm)$ {
    include /etc/nginx/conf.d/drupal/location_expires_prepend*.conf;
    expires 1800s;
    include /etc/nginx/conf.d/drupal/location_expires_append*.conf;
}

In our case, we're using fastcgi_intercept_errors to do some nice things with error handling in sites/default/files and so we need to get a couple of directives into this specific location.

Will put up a PR for these changes shortly.

Proposed resolution

pull:
    usage: Pull latest docker images.
    cmd: docker image ls --format \"{{.Repository}}:{{.Tag}}\" | grep govcms/ | grep -v none | xargs -n1 docker pull | cat

docker pull uselagoon/varnish-drupal:21.4.0

Error response from daemon: manifest for uselagoon/varnish-drupal:21.4.0 not found: manifest unknown: manifest unknown

And 21.4.0 is not available for uselagoon/varnish-drupal:

https://hub.docker.com/r/uselagoon/varnish-drupal/tags?page=1&ordering=last_updated

As a developer I don't want to deploy my code if configuration fails to validate.

This is because content (which I can't control in my deployment) can cause config sync to fail. In the case below I can't delete the content type because there exists "Speech" content.

 [error]  Drupal\Core\Config\ConfigImporterException: There were errors validating the config synchronization.
Entities exist of type <em class="placeholder">Content</em> and <em class="placeholder">Content type</em> <em class="placeholder">Speech</em>. These entities need to be deleted before importing. in Drupal\Core\Config\ConfigImporter->validate() (line 737 of /app/web/core/lib/Drupal/Core/Config/ConfigImporter.php). 

In ConfigImportCommands.php line 261:
                                                                               
  The import failed due to the following reasons:                              
  Entities exist of type <em class="placeholder">Content</em> and <em class="  
  placeholder">Content type</em> <em class="placeholder">Speech</em>. These e  
  ntities need to be deleted before importing.   

The main reason this is a problem is when config is being deleted. Say in my deployment I'm creating a field or content type, and I have code in my theme that is expecting the field or content type to be there. My code fails and potentially causes a white screen for the whole site.

In
https://github.com/govCMS/govcms8lagoon/blob/master/.docker/images/govcms8/scripts/govcms-deploy#L66
we perform an enable/uninstall of the govcms_lagoon module, which is only used to ensure that the dependencies listed in
https://github.com/govCMS/govcms8lagoon/blob/master/modules/lagoon/govcms_lagoon/govcms_lagoon.info.yml#L8
Surely there's a better way (surely?)

The 203-expires.conf nginx helper takes over URLs matching non-image files to set an expiration header:

location ~* ^/sites/default/(files/(?!private/)).+\.(pdf|doc|docx|txt|xls|xlsx|csv|ppt|pptx|pps|ppsx|odt|ods|odp|mp3|mov|mp4|m4a|m4v|mpeg|avi|ogg|oga|ogv|weba|webp|webm)$ {
    ...
}

Since there is no try_files $uri @drupal, Stage File Proxy doesn't work for these URLs. This results in the non-production sites in the GovCMS SaaS returning OpenResty 404 pages for these URLs, making it difficult to test non-image files in staging.

Is this by design, or is there another mechanism to support Stage File Proxy? Would there be any reason not to use the NGINX_DEFAULT_EXPIRES variable instead of this helper?

FYI In my local dev environment I've worked around the issue by adding:

/etc/nginx/conf.d/drupal/location_expires_append_drupal.conf:

try_files $uri @drupal;

We have govCMS/scaffold-tooling being used by the paas scaffold. Can we pull these here for use in the images.

There seems to be inconsistency in documenting GovCMS distribution version changes in release notes. The last release to specifically mention a change to GovCMS distro version was v8.9.1. Given that v8.9.3 contains Drupal core 8.9.3 which is part of GovCMS 1.8.0, then I presume GovCMS distro 1.8.0 was included in release v8.9.3.

It would be good to be consistent in providing this information. The reason for this is that we are creating a starter kit for GovCMS8 and need to be able to keep track of distro updates so that we can make sure relevant config changes are updated and exported for our starter kit. We do this by manually updating the GovCMS image version (e.g. 8.9.3) in the starter kit config and then running and exporting any database updates.

The problem is that when GovCMS updates the distribution, they send a notification specifying the distro version (e.g. 1.8.0) rather than the image version (e.g. 8.9.3). It is then up to us to map the distro version to the corresponding image version. This would be a lot easier if the image version release notes made mention of what distro version it is using.

The upstream image used for mariadb locally is one of the "legacy" unversioned uselagoon images.

The uselagoon/mariadb-10.4-drupal image is the same, but is published as multiarch, avoiding emulation for localdev

Moving database/config onto and off of the lagoon platform is made a little awkward by lagoon_ modules being in the docker image. Can these be moved into into standalone packages?

See .docker/images/nginx/no-robots.sh

This copies a robots.txt file into /app, it should be /app/web.

Using configs directly creates a supply chain poisoning possibility: if the search_api_solr module maintainer's account is compromised, the malicious user can add malicious configs that would be "blindly" added to the image and deployed, which can potentially wipe the Solr index.

The solution is to copy the configurations from the jump-start into a configs/solr directory manually on every new version of search_api_solr module.

We are currently experiencing an issue where the WebP thumbnails do not show up due to an NGINX configuration error.

remove webp from here https://github.com/govCMS/lagoon/blob/3.x-develop/.docker/images/nginx/helpers/203-expires.conf#L2?

related issue: https://www.drupal.org/project/drupal/issues/3120676

The solr settings hard-configured in settings.php have no server machine name specified, so will not apply (thankfully, as the path is also outdated).
https://github.com/govCMS/govcms8lagoon/blob/develop/.docker/images/govcms8/settings/settings.php#L150-L154

The host setting should probably be added to the second code block though (and set to the default lagoon_solr machine name)
https://github.com/govCMS/govcms8lagoon/blob/develop/.docker/images/govcms8/settings/settings.php#L274-L277

We clear the composer cache when we build the CLI container, but this is a lost opportunity. Most of the packages we use are fixed since our versions don't change often, so leaving the composer cache in place means subsequent builds downstream could speed up a lot by not needing to download all the sources - and we have a lot of sources. I propose we leave the composer cache in place here.

Something to be aware of is the main impact is on PaaS currently, but improvements in the way SaaS works may more and more require composer install, in order to follow mainstream best practices.

An alternative approach is to use gitlab CI cache, which might look like this (internal ref GOVCMS-3110).

cache:
  key: "$CI_PROJECT_PATH_SLUG"
  paths:
    - /home/.composer/cache

We added SNAPSHOTS=/app/web/sites/default/files/private/backups to lagoon.yml
govCMS/govcms8-scaffold-paas#37

It ideally lives here and not upstream in amazee images as it's pretty specific to our workflow, and it's a pretty passive piece of configuration.