Ideally, I would like to use the same Docker Compose file to start up a DSpace 5, a DSpace 6 and a DSpace 7 instance of DSpace.

In order to do this, I would like to manage separate volumes for the database and the dspace-install directories and intelligently manage those volumes.

I wounder if an ENV variable could be introduced when setting the volume names in the Docker Compose file to assist with the management of this information.

When I change admin variables by e.g.

$ cat .env
[email protected]
ADMIN_PASS=admin2

the logging still accept the original one not those.

PORT=8081 docker-compose ... up -d

(-e PORT=8081 is not supported)

docker-compose does not support inline/heredoc volume resources

• Consider a variable LOCALCFG that is set as a multiline variable in the yaml.
• Modify the startup script to append these values to the end of the local.cfg

https://jira.duraspace.org/browse/DS-4012

• Checkout PR DSpace/DSpace#2058 to your local DSpace repo
• Set DSPACE_SRC to your DSpace repo directory

export DSPACE_SRC=$(pwd)

• Clone https://github.com/DSpace-Labs/DSpace-Docker-Images.git

cd DSpace-Docker-Images/docker-compose-files/dspace-compose

• Build PR 2058 into a local image dspace/dspace:dspace-local

docker-compose -p d7 -f docker-compose.yml -f d7solr.override.yml -f src.override.yml build

• Start DSpace 7 in Docker with an externalized solr service using the image you built

docker-compose -p d7 -f docker-compose.yml -f d7solr.override.yml -f src.override.yml up -d

• Note: Once PR 2058 is merged, d7solr.override.yml will become d7.override.yml.

Having followed the instructions and set DSPACE_VER=dspace-4_9 and $DPROJ=d4

having run : docker-compose -p $DPROJ up
the database seems to initialise and lots of logging is coming from catalina, but then:

dspace_1    |  INFO [org.dspace.servicemanager.DSpaceKernelImpl] DSpace kernel startup completed in 1594 ms and registered as MBean: org.dspace:name=c778448b-c238-4a75-bf16-ee78694931af,type=DSpaceKernel
dspace_1    |  INFO [org.dspace.core.ConfigurationManager] Loading provided config file: /dspace/config/dspace.cfg
dspace_1    |  INFO [org.dspace.core.ConfigurationManager] Using dspace provided log configuration (log.init.config)
dspace_1    |  INFO [org.dspace.core.ConfigurationManager] Loading: /dspace/config/log4j.properties
dspacedb_1  | 2018-09-18 13:47:14.238 UTC [85] ERROR:  syntax error at or near ")" at character 22
dspacedb_1  | 2018-09-18 13:47:14.238 UTC [85] STATEMENT:  INSERT INTO webapp ( ) VALUES ( ) RETURNING null
dspace_1    | Loading catalog: file:/usr/local/tomcat/work/Catalina/localhost/xmlui/cocoon_xml_resolver_entities/catalog
dspace_1    | 18-Sep-2018 13:47:16.085 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/xmlui] has finished in [8,824] ms
dspace_1    | 18-Sep-2018 13:47:16.085 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/jspui]
dspace_1    | 18-Sep-2018 13:47:22.138 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
dspace_1    | log4j:WARN No appenders could be found for logger (org.dspace.servicemanager.DSpaceKernelInit).
dspace_1    | log4j:WARN Please initialize the log4j system properly.
dspace_1    | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
dspace_1    | INFO: Loading provided config file: /dspace/config/dspace.cfg
dspace_1    | INFO: Using dspace provided log configuration (log.init.config)
dspace_1    | INFO: Loading: /dspace/config/log4j.properties
dspacedb_1  | 2018-09-18 13:47:23.902 UTC [87] ERROR:  syntax error at or near ")" at character 22
dspacedb_1  | 2018-09-18 13:47:23.902 UTC [87] STATEMENT:  INSERT INTO webapp ( ) VALUES ( ) RETURNING null
dspace_1    | 18-Sep-2018 13:47:23.970 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/jspui] has finished in [7,885] ms
dspace_1    | 18-Sep-2018 13:47:23.970 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/rest]
dspace_1    | 18-Sep-2018 13:47:29.545 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
dspace_1    | INFO: Loading provided config file: /dspace/config/dspace.cfg
dspace_1    | INFO: Using dspace provided log configuration (log.init.config)
dspace_1    | INFO: Loading: /dspace/config/log4j.properties
dspace_1    | 18-Sep-2018 13:47:30.811 INFO [localhost-startStop-1] com.sun.jersey.api.core.PackagesResourceConfig.init Scanning for root resource and provider classes in the packages:
dspace_1    |   org.dspace.rest
dspace_1    | 18-Sep-2018 13:47:30.834 INFO [localhost-startStop-1] com.sun.jersey.api.core.ScanningResourceConfig.logClasses Root resource classes found:
dspace_1    |   class org.dspace.rest.HandleResource
dspace_1    |   class org.dspace.rest.BitstreamResource
dspace_1    |   class org.dspace.rest.CommunitiesResource
dspace_1    |   class org.dspace.rest.RestIndex
dspace_1    |   class org.dspace.rest.ItemsResource
dspace_1    |   class org.dspace.rest.CollectionsResource
dspace_1    | 18-Sep-2018 13:47:30.834 INFO [localhost-startStop-1] com.sun.jersey.api.core.ScanningResourceConfig.init No provider classes found.
dspace_1    | 18-Sep-2018 13:47:30.898 INFO [localhost-startStop-1] com.sun.jersey.spi.spring.container.servlet.SpringServlet.getContext Using default applicationContext
dspace_1    | 18-Sep-2018 13:47:30.902 INFO [localhost-startStop-1] com.sun.jersey.server.impl.application.WebApplicationImpl._initiate Initiating Jersey application, version 'Jersey: 1.17.1 02/28/2013 12:47 PM'
dspace_1    | 18-Sep-2018 13:47:31.532 WARNING [localhost-startStop-1] com.sun.jersey.spi.inject.Errors.processErrorMessages The following warnings have been detected with resource and/or provider classes:
dspace_1    |   WARNING: A sub-resource method, public org.dspace.rest.common.Community[] org.dspace.rest.CommunitiesResource.list(java.lang.String), with URI template, "/", is treated as a resource method
dspace_1    |   WARNING: A sub-resource method, public org.dspace.rest.common.Collection[] org.dspace.rest.CollectionsResource.list(java.lang.String,java.lang.Integer,java.lang.Integer), with URI template, "/", is treated as a resource method
dspace_1    | 18-Sep-2018 13:47:31.540 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/rest] has finished in [7,570] ms
dspace_1    | 18-Sep-2018 13:47:31.556 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
dspace_1    | 18-Sep-2018 13:47:31.585 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
dspace_1    | 18-Sep-2018 13:47:31.589 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 49792 ms
dspacedb_1  | 2018-09-18 13:47:49.618 UTC [88] ERROR:  relation "community" does not exist at character 15
dspacedb_1  | 2018-09-18 13:47:49.618 UTC [88] STATEMENT:  SELECT * FROM community WHERE NOT community_id IN (SELECT child_comm_id FROM community2community) ORDER BY name

Something goes wrong in the database. There are three different errors logged from the database (dspacedb_1), the first one repeated twice.

http://localhost:8080/jspui/ - has an Internal error page

Next up: As you may have seen, the team at CNRI has reached out to help us upgrade our Handle server: https://jira.duraspace.org/browse/DS-4205 and DSpace/DSpace#2394

In DSpace 4, the indexer does not run automatically. This should be added to the startup script.

We could use the docker-compose file for DSpace 6.

Since pgcrypto is not needed, we could create a separate compose file that uses the postgres image directly.

If so, I would imagine that we would build and re-publish images each time that there is a major DSpace release.

Once a user has deployed a container, the user would have the option to re-deploy the code within the container.

The primary advantage of this approach is that it would require one fewer volume file to test.

On the other hand. it would make it more complicated to load up a large scale database volume.

@pnbecker has embedded a sample Dockerfile in https://jira.duraspace.org/browse/DS-4160 to illustrate a bug in a specific version of Linux. I recommend that we implement something like this with the following conventions.

• follow the installation directories used in the DSpace/DSpace Dockerfile
• rely on an external postgres instance rather than embedding this locally
• add this Dockerfile to this repo in the Dockerfiles directory

The goal should be to make the image produced from this dockerfile to be swap-able with the dspace/dspace images referenced in the DSpace Docker compose files.

This will depend on https://jira.duraspace.org/browse/DS-4167

To enable registration emails

If so, where should the source for these scripts reside?

How should these scripts be invoked? (From the host computer, or within a bash window in Docker)?

When we initially added Dockerfiles to the DSpace repo, there was a desire to place them in dspace/src/main/docker.

The DockerHub classic build system did not work with this approach.

With the new dockerhub build system, this might be possible

We currently save /dspace/solr to the solr volume.

Because this contains both the conf and data directories for each repo, this could interfere with updates to solr conf files.

Our docker compose files should persist each solr data directory into its own repo.

• solr_search
• solr_statistics
• solr_oai
• solr_authority

We currently maintain a separate docker compose file to support ingest of content into a DSpace instance.

https://github.com/DSpace-Labs/DSpace-Docker-Images/tree/master/docker-compose-files/dspace-ingest-compose

Create optional ENTRYPOINT in DSpace Dockerfile

Rather than using an alternate compose file, I recommend the following changes.

In the DSpace Dockerfile, provide one (or more) optional entrypoint scripts that provide the functionality that currently resides here:

https://github.com/DSpace-Labs/DSpace-Docker-Images/tree/master/add-ons/mount_ingest_tools

Consider installing these to /entrypoint

Provide optional call to alternate ENTRYPOINT in DSpace docker-compose files

# Uncomment the following line to trigger the init of DSpace solr repos
entrypoint: "/entrypoint/createAdmin.sh"

Possible ENTRYPOINT options

• Create Administrator
  • Pass in $ADMIN_EMAIL, $ADMIN_PASS, $ADMIN_FNAME, $ADMIN_LNAME via docker-compose.yml
• Create Administrator and ingest AIP
  • perform create admin as described above
  • download $AIPZIP file containing AIP's and ingest each AIP

Questions

• What other tasks would be useful to run on startup?

o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: no field name specified in query and no default specified via 'df' param

This will require the following changes

• Use a different port number for each instance
• Add the version number to the container names: dspace-dX, dspacedb-dX
• Update tutorial instructions

1. A database must be available to perform this task
2. Database volumes are contained in the database container - cannot be mounted to the tomcat container

How could we re-use the existing published images for DSpace?

The compose file will bring in the proper database image.

We will need to insert the right Oracle properties into local.cfg before tomcat is started.

Could we create a volume to override local.cfg?

Call localhost to hit services

• rest
• solr
• oai
• sword
• rdf

Verify expected status codes from requests

Determine the minimum setting needed to run DSpace 6 and DSpace 7.

Update the windows and macos install pages with these details.

Just noting that the ingetAIP.sh script currently does its own parsing/looping over Community/Collection/Item AIPs (instead of restoring from a Site-level AIP), e.g.
https://github.com/DSpace-Labs/DSpace-Docker-Images/blob/master/add-ons/mount_ingest_tools/ingestAIP.sh

We may want to consider updating to do an import starting at the SITE level AIP (this is the same command we use on demo.dspace.org to do a full site "reset"):

/dspace/bin/dspace packager -u -r -a -f -t AIP -e [email protected] -o skipIfParentMissing=true -o createMetadataFields=true SITE@[handle].zip

The extra benefit here is that this will also create any permissions/users/groups that are stored in the SITE AIP. It also includes flags to auto-create any missing/custom metadata fields, and to skip any "parent not found errors".

The full reset script for demo.dspace.org site is in GitHub here (in case it's useful to see the entire restore process run on demo): https://github.com/DSpace-Labs/demo.dspace.org-site/blob/master/scripts/linux/reset-dspace-content#L234

When the "dspace version" command is run, "SCM revision" and "SCM branch" are set. These values are not getting set in our Docker builds. I wonder if there is a way to pass these values through environment variables. Do you know what mechanism grabs these values?

Kim Shepherd [12:56 PM]
@terrywbrady the file that holds the vars is dspace-api/src/main/resources/scm.properties

branch = ${scmBranch}```
and it seems to be maven that populates it, as the `target` version in dspace-api/target/classes/scm.properties looks like this
```revision = 929312d7e8728d33d3425e2b106c444444ea0468
branch = DS-4136_oai_import_master```
so, not sure how docker is missing out.. the maven build is fairly straight forward isn't it?
(you can see the full class at `org.dspace.app.util.Version`

Mark Wood [1:13 PM]
Yes, Maven creates that file for us.  The resource plugin fills in values from the effective POM while copying the template over to target.  It's not easy to see, but see dspace-api/pom.xml (edited) 
Likewise Ant write its version into 'ant.properties'.

Kim Shepherd [1:54 PM]
hm, interestingly, though  my docker-dev-compose instance works ok (it's an old one) for `dspace version`, it lists each webapp in Applications at least 10 times.. i wonder if there's some caching in the UtilServiceFactory stuff, or some filtering/deduplication needed in the results for that...

Mark Wood [1:56 PM]
That's an old problem that I need to fix someday.  The duplicates for webapp X *should* all be deleted when X starts, but for some reason they persist.
https://jira.duraspace.org/browse/DS-3000

Terry Brady [2:06 PM]
Since the published images are built on Dockerhub, those values are probably not set.  I will add an issue to our Docker project to find a way to set these values.

Based on this report, I would expect a locally built instance to possibly have the branch.

Add the call to rdfize in the script

Example: DSpace/DSpace#2379

This should allow us to collapse the multiple Dockefiles that we have on each branch (jdk7 vs jdk8) or (test vs no test).

See https://docs.docker.com/docker-hub/builds/advanced/

Hello,

I have installed the DSpace7 using this image. Its showing me REST and angular interface on browser.

When i try to sign in/sign up or forgot password, its just come back to home page. There are errors in the console. I am uploading the image as well.
I have installed the docker on window 10 professional. i am using the docker machine ip not the localhost.

https://docs.docker.com/docker-hub/builds/

This would be especially useful if we intend to deploy pre-built images for Docker.

Create a docker compose file that will run 3 containers

• database
• dspace tomcat
• dspace angular

Re-use as much of the DSpace 6 docker file as possible

Images

• dspacedb
  • postgres + pgcrypto
• dspace-tomcat
  • tomcat instance + ant
  • user can mount a DSPACE_SRC directory and deploy from there
  • requires an ant deploy to become usable (otherwise tomcat will start with no DSpace webapps)
• dspace
  • intended for use by DSpace 6 or DSpace 7
  • tomcat instance with a fully built dspace directory
  • includes local.cfg file
  • DSpace code is cloned and built during image build
• dspace-angular
  • dspace angular image

Image Tag Names

• dspace/dspace:dspace-6.3
  • Tagged when the release is generated
• dspace/dspace:dspace-6_x
  • Generated weekly (or on demand)
• dspace/dspace:dspace-6_x-pr1111
  • Generated by travis when a PR is created
  • Or, should this be a separate image in order to ensure a meaningful "latest" image
• dspace/dspace:dspace-6_x-feature
  • Generated manually to illustrate the workings of a combined set of related PRs
• latest
  • how do we manage the "latest" tag if prs may be generated more frequently?

Compose Files

• Question: should compose files be managed in a separate directory, or within a folder containing a Dockerfile? What is the best practice to follow?
• dspace-dev-compose
  • dspace-postgres-pgcrypto
  • dspace-tomcat
• dspace-compose
  • dspace-postgres-pgcrypto
  • dspace
• dspace7-compose
  • dspacedb
  • dspace:master
  • dspace-angular

Directories

• /dspace
  • DSpace install directory
  • The contents of this directory will be written to a docker volume
• /dspace-src (container image)
  • When the image is built, DSpace will be cloned to this directory.
  • A local.cfg file will be written here
  • Mvn will be initiated from this directory
• /dspace-src (user build dir, mounted to DSPACE_SRC)
  • If a user wants to build/deploy changes into a docker image, this could serve that purpose
  • When docker is used in this manner, should an entirely different image be used?
  • What name should be assigned to such an image?
• /dspace-aip (user provided directory for providing aip files to ingest)
  • The contents of this folder are only needed once.
  • Should this be a volume mount, or should the content be copied into this folder when needed?
• /dspace-tools
  • Would it be a good idea to provide tools such as this?
  • Scripts copied into the docker image to simplify the use of the container
  • loadAip.sh - create administrator and load AIP files found in /dspace-aip
  • deployDSpace.sh - run ant deploy, restart tomcat (if that does not kill the container)
  • buildDSpace.sh - should we create an image with mvn + ant + tomcat?

Eliminate the test_database check

Also update tutorials

This is a paid feature, we would need to present a case to the steering group for approval.

The cost should be modest.

https://docs.docker.com/docker-hub/builds/

I was closely watching my Docker logs on startup (docker logs -f dspace) and noticed that it threw a syntax error on this line:
https://github.com/DSpace-Labs/DSpace-Docker-Images/blob/master/add-ons/dspace-docker-tools/ingestAIP.sh#L76

I believe the variable either needs to be surrounded by quotes, or we might need to use -eq. I haven't had a chance to dig into the correct syntax yet though.

If so, there volumes would already contain sample users, communities, collections, items, and bitstreams.

These files would provide a quick restart point for someone deploying a new container.

These volumes would need to be tagged to specific DSpace releases.

Would it be possible to make the instructions so simple that any repository manager with a local Docker instance could test a specific version of the code?

When travis performs the automated build of code for a specific PR, could the built assets be retrievable for quick testing? Or, would the download of built assets be so slow that it would be advisable just to build locally?

If a user must build locally, what can we do to make it as easy as possible?

The current dockerfile has made the entire dspace-install directory into a volume.

This can be convenient because it allows you to stop and start a container and the state of the solr directories is retained.

Because the installed code is in a volume, if you swap versions (dspace-5_x to dspace-5.9), the code is not updated.

Consider creating separate volumes for the assetstore and for solr. Let the code remain within the image.

The config framework introduced in DSpace 6 allows environment variables to override config values set in dspace.cfg or local.cfg.

I did some testing of this with one of our Docker compose files.

The environment variable did get passed into tomcat and worked as expected.

When running a command line task using docker exec dspace bash, the environment variables were not carried into the shell environment.

Terry Brady [7:52 AM]
Since I have never seen Sword in action, it would be interesting to create a docker compose file that runs a sword client alongside DSpace. Perhaps both a v1 and v2.

Does such a client exist?

Tim Donohue [7:53 AM]
DSpace XMLUI has a built in SWORDv1 client. But, yes, I agree
I'm not aware of many SWORDv2 clients -- there surely are some, but I think that's part of the reason why SWORDv1 is still in wide use

James Creel [7:54 AM]
The use case for the XMLUI SWORDv1 client isn't super strong, as I recall. You could also do export/import, harvesting, other things.
But Vireo uses SWORDv1

Tim Donohue [7:54 AM]
SWORDv2 implementations: http://swordapp.org/sword-v2/sword-v2-implementations/

James Creel [7:54 AM]
Just a curl request and a METS structure sitting next door could suffice for demonstration

Tim Donohue [7:55 AM]
Yes, technically you can send content to either SWORDv1 or SWORDv2 via curl. We have examples of both in our DSpace Docs

Mark Wood [7:55 AM]
I'm beginning to imagine a bunch of projects standing around, all thinking "SWORDv1 should be deprecated," but none of them wanting to be the first to do so.

Tim Donohue [7:56 AM]
Example curl commands for SWORDv1 are in our docs here: https://wiki.duraspace.org/display/DSDOC5x/SWORDv1+Server#SWORDv1Server-DeposittoSWORDServer

Mark Wood [7:56 AM]
Meanwhile ResourceSync is hovering in the background, looking for an opening.

Terry Brady [7:56 AM]
I started a github issue. I will copy this notes there. #73

Tim Donohue [7:56 AM]
Example curl commands for SWORDv2 are in our docs here: https://wiki.duraspace.org/display/DSDOC6x/SWORDv2+Server#SWORDv2Server-DeposittoSWORDv2Server

I discovered/fixed a bug in the base image.

terrywbrady@1bfd9fa

1. Create build env var to specify the pr number
2. Clone/checkout base image
3. Merge PR
4. Build a basic image

Make this invokable from dockerhub dspace/dspace

Related links

• DSpace/DSpace#2385
• https://stackoverflow.com/questions/55347752/dockerhub-autobuild-for-a-github-pull-request-using-the-docker-tag-variable
• https://forums.docker.com/t/passing-variables-to-an-automated-build-docker-tag/71875