artefactual / deploy-pub Goto Github PK

This is an example: https://github.com/artefactual/deploy-pub/blob/master/playbooks/archivematica/vars-singlenode-qa.yml#L15-L16

As described in artefactual-labs/ansible-elasticsearch#5 and artefactual-labs/ansible-elasticsearch#3 the oracle java installer referenced in the lines above no longer exists.

These lines could be removed entirely from the vars-singlenode files, relying on the defaults defined in the ansible-elasticsearch role instead.

I've tried to provision a few times now using vars-singlenode-qa.yml instead of vars-singlenode-1.6.yml, and I keep getting the same error.

TASK [artefactual.archivematica-src : Checkout archivematica-storage-service code] ***
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "/usr/bin/git checkout --track -b qa/0.x origin/qa/0.x", "failed": true, "msg": "Failed to checkout qa/0.x", "rc": 1, "stderr": "error: Your local changes to the following files would be overwritten by checkout:\n\t
storage_service/static/js/vendor/base64.js\nPlease, commit your changes or stash them before you can switch branches.\nAborting\n", "stderr_lines": ["error: Your local changes to the following files would be overwritten by checkout:", "\tstorage_service/static/js/vendor/
base64.js", "Please, commit your changes or stash them before you can switch branches.", "Aborting"], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/vagrant/singlenode.retry

When I look at src/archivematica-storage-service, I see that branch stable/0.10.x is checked out even though archivematica_src_ss_version is set to "qa/0.x".

roles/artefactual.archivematica-src/tasks/ss-main.yml looks good. The version corresponds to the yml file and force is set to "yes".

Investigating...

I was looking at installing Archivematica via Vagrant today, but I have a Windows workstation and Windows isn't supported as a control machine with Ansible. I suppose I could try using the Windows Subsystem for Linux, but that's not really supported by anyone either.

In any case, I think this could be the solution:
https://www.vagrantup.com/docs/provisioning/ansible_local.html

This has been used with Koha to allow Windows-based developers to use Vagrant as well.

I'm going to look at trying this out tomorrow, and then I'll send a pull request if I get it up and running.

I installed Archivematica 1.7 using digital-ocean-install-example.rst

I used Ubuntu 16.04. The instructions still work great, but I noticed a few details that would be helpful to add / correct:

• using Ubuntu 16.04, digital ocean only installs Python 3 by default. I had to add python 2.7 manually first.
• the step about using user.yml wasn't totally clear to me, if nothing else because it was missing the actual ansible command $ ansible-playbook user.yml

I also had real troubles executing this deployment from either my Mac Air latop or iMac desktop. In both cases I had repeated problems with a particular dependency because of TLS errors, due to my Macs using an old version of OpenSSL.

- downloading role 'nodejs', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-nodejs/archive/4.1.1.tar.gz
 [ERROR]: failed to download the file: Failed to validate the SSL certificate for github.com:443. Make sure your managed systems have a valid CA certificate installed. You can use
validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this platform: /etc/ssl/certs, /etc/ansible,
/usr/local/etc/openssl. The exception msg was: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590).

 [WARNING]: - geerlingguy.nodejs was NOT installed successfully.

I tried to resolve this by updating OpenSSL but couldn't get it to work. In the end, I found it faster to simply set up another digital ocean droplet from which to run the ansible scripts. (I'm not sure whether any of this last issue is worth noting in the installation instructions, but I note it here just in case anyone else comes across this problem).

Since we have two branches in Binder now, stable/0.8.x (using ES 1.x) and qa/0.9.x (using ES 5.x), which will require to use the official Ansible role from ES, and considering that Binder can only run in PHP 5.6 in both cases; I think it's better to create a new playbook for Binder using Ubuntu Trusty and the ES Ansible role or the ES Artefactual role based on the version.

Since we remove the database creation from the ansible-archivematica-src role, the playbook is failing.

As the percona role was updated to run in redhat/centos deployment here, we should add it to the roles and may be, use the same playbook in Ubuntu and RH.

Using master at 86d70f4 without any changes, vagrant up fails in the following task:

TASK [artefactual.archivematica-src : Generate special sampledata transfers] ***
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "/home/vagrant/archivematica-sampledata/createtransfers.py create-variously-encoded-files", "msg": "[Errno 2] No such file or directory", "rc": 2}

When restarting the vagrant VM (vagrant halt -> vagrant up), archivematica services (archivematica-dashboard, archivematica-mcp-server, archivematica-mcp-client) require manual start up:
$ sudo start archivematica-storage-service
$ sudo start archivematica-mcp-server
$ sudo start archivematica-mcp-client
$ sudo start archivematica-dashboard

Seeing the following errors in the upstart log files:

Invalid value for chdir: /usr/lib/archivematica/storage-service
Error: can't chdir to '/usr/lib/archivematica/storage-service'

Invalid value for chdir: /usr/share/archivematica/dashboard
Error: can't chdir to '/usr/share/archivematica/dashboard'

/proc/self/fd/9: 4: /proc/self/fd/9: /usr/lib/archivematica/MCPServer/archivematicaMCP.py: not found

/proc/self/fd/9: 7: /proc/self/fd/9: /usr/lib/archivematica/MCPClient/archivematicaClient.py: not found

Currently, we sync the "current" folder (the one we use to run "vagrant up") using "/vagrant" as path in the virtual machine.

Changing this path to /home/local-sync (or any other name), would allow users to put their own transfers on the same folder than the playbooks, and process them, beacause /home is configured as the default transfer source in the storage service

Needed to be able to use in a non-Vagrant environment.

We have changed the Ubuntu version check to install the Percona server in artefactual-labs/ansible-percona@16a44e5

After those changes it installs Percona 5.6 by default in Ubuntu Trusty if it's not changed in the playbook using the new mysql_version_major and mysql_version_minor variables.

We have a working packer configuration to build atom boxes, we should move it here and build also archivematica vagrant boxes.

On the other side, the elasticsearch replication role available here, doesn't seem to fit with the repo, we should create a new git repo for it.

To ease the use of tools like https://github.com/appbaseio/dejavu

Trying to connect to the shared folder (when using "atom_devbox" set to true) as explained in here was giving me a lot of permission issues from a Linux OS host. Using Vagrant synced folders solved the issues for me.

Compare the following two files

• https://github.com/artefactual/deploy-pub/blob/master/playbooks/archivematica/vars-singlenode-1.7.yml
• https://github.com/artefactual/deploy-pub/blob/master/playbooks/archivematica/vars-singlenode-qa.yml

The first contains the following but the second does not:

mysql_databases:
  - name: "{{ archivematica_src_am_db_name }}"
    collation: "utf8_general_ci"
    encoding: "utf8"

mysql_users:
  - name: "{{ archivematica_src_am_db_user }}"
    pass: "{{ archivematica_src_am_db_password }}"
    priv: "{{ archivematica_src_am_db_name }}.*:ALL,GRANT"
    host: "{{ archivematica_src_am_db_host }}"

We removed the database creation from the archivematica role at artefactual-labs/ansible-archivematica-src@fb87517 so now it needs to be created using the percona role , as we do in atom.

A new variable has been added to the AtoM role to determine the ES config used. See artefactual-labs/ansible-atom@72ddedb

AtoM Xenial should be installing 2.5 instead of 2.4 (the default)

Some acceptance tests need to be able to scp files from VM to local and therefore need to know the password for the ubuntu user created by the ubuntu/xenial64 box. However, this user has no password (that I can ascertain)—see https://bugs.launchpad.net/cloud-images/+bug/1569237. Solution would seem to be to specify the ubuntu user's password in the archivematica-xenial/Vagrantfile.

create a requirements-160.yml, vars-singlenode-160.yml and maybe a singlenode-160.yml, or else change singlenode.yml to run 160 stuff by default.

Columbia University Library would like a way to install Archivematica without Elasticsearch
As described here: artefactual-labs/archivematica-acceptance-tests#23.

This is not meant to be the only way to deploy Archivematica, so the solution needs to allow a person deploying Archivematica to pick whether or not Elasticsearch is installed.

This will be one step towards being able to deploy a 'headless' Archivematica instance, where the dashboard is used only as a REST api, driven by requests from outside applications (such as the automation tools).

We install gearmand as part of the singlenode.yml playbook for centos, but we need also to add the line

OPTIONS="--listen=127.0.0.1 --port=4730 --queue-type=builtin"

to /etc/sysconfig/gearmand.

We can also start using the gearmand role available at https://github.com/artefactual-labs/ansible-gearman

I installed Archivematica version 1.7.0 onto a digital ocean VM (using the digital-ocean-install-example.rst)

Running the singlenode.yml playbook failed attempting to install dependencies for ElasticSearch:

TASK [artefactual.elasticsearch : Install dependencies] *******************************************************************************************************************
fatal: [am-local]: FAILED! => {"changed": false, "msg": "No package matching 'openjdk-7-jre-headless' is available"}

I updated the playbook to replace '7' with '8' in the following 2 lines:
elasticsearch_apt_java_package: "openjdk-7-jre-headless"
elasticsearch_java_home: "/usr/lib/jvm/java-7-openjdk-amd64"

I was then able to re-run the playbook and complete the installation successfully.

For using these in a non-Vagrant VM, need to be able to configure a firewall.

As f80bf57, if someone tries to deploy Archiveamtica using the deploy-pub/playbooks/archivematica it's unclear for a newcomer if stable or qa branches are being deployed. This just beat @mcantelon which is a sign that we're doing a poor job explaining it.

I suggest to have two environments:

• deploy-pub/playbooks/archivematica
• deploy-pub/playbooks/archivematica-qa

Or we could do a better job in the README file explaning how to download the right branch of the AM role (e.g. qa/1.7.x branch targets AM 1.7 and so on) and the right vars file.

It's being executed as 'ubuntu' when the AtoM dir is owned by 'vagrant'. We should change the default 'atom_user' and 'atom_group' in the Vagrantfile.

Inspired by @jhsimpson in artefactual-labs/ansible-archivematica-src#191:

Now that there is an official v1.7.0 tag in Archivematica, we need a way to make sure that this repo has an easy way to install that version.

This could be done with a stable/1.7.x branch in this repo, and a v1.7.0 tag?

Using master at 86d70f4 without any changes, vagrant up fails in the following task:

TASK [artefactual.archivematica-src : Install front-end dependencies] **********
changed: [am-local] => (item=/vagrant/src/archivematica/src/dashboard/frontend/appraisal-tab)
failed: [am-local] (item=/vagrant/src/archivematica/src/dashboard/frontend/transfer-browser) => {"changed": true, "cmd": ["npm", "install"], "delta": "0:00:16.859546", "end": "2018-03-20 15:43:55.774615", "item": "/vagrant/src/archivematica/src/dashboard/frontend/transfer-browser", "msg": "non-zero return code", "rc": 1, "start": "2018-03-20 15:43:38.915069", "stderr": "npm WARN prepublish-on-install As of npm@5, `prepublish` scripts are deprecated.\nnpm WARN prepublish-on-install Use `prepare` for build steps and `prepublishOnly` for upload-only.\nnpm WARN prepublish-on-install See the deprecation note in `npm help scripts` for more information.\nsh: 1: webpack: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! [email protected] prepublish: `webpack --progress --colors --entry ./app.js`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the [email protected] prepublish script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /var/lib/archivematica/.npm/_logs/2018-03-20T15_43_55_756Z-debug.log", "stderr_lines": ["npm WARN prepublish-on-install As of npm@5, `prepublish` scripts are deprecated.", "npm WARN prepublish-on-install Use `prepare` for build steps and `prepublishOnly` for upload-only.", "npm WARN prepublish-on-install See the deprecation note in `npm help scripts` for more information.", "sh: 1: webpack: not found", "npm ERR! file sh", "npm ERR! code ELIFECYCLE", "npm ERR! errno ENOENT", "npm ERR! syscall spawn", "npm ERR! [email protected] prepublish: `webpack --progress --colors --entry ./app.js`", "npm ERR! spawn ENOENT", "npm ERR! ", "npm ERR! Failed at the [email protected] prepublish script.", "npm ERR! This is probably not a problem with npm. There is likely additional logging output above.", "", "npm ERR! A complete log of this run can be found in:", "npm ERR!     /var/lib/archivematica/.npm/_logs/2018-03-20T15_43_55_756Z-debug.log"], "stdout": "\n> [email protected] prepublish /vagrant/src/archivematica/src/dashboard/frontend/transfer-browser\n> webpack --progress --colors --entry ./app.js", "stdout_lines": ["", "> [email protected] prepublish /vagrant/src/archivematica/src/dashboard/frontend/transfer-browser", "> webpack --progress --colors --entry ./app.js"]}

Based on instructions at https://github.com/artefactual-labs/archivematica-acceptance-tests, create ansible tasks/roles that install the AM acceptance tests and their dependencies:

• acceptance tests and Python deps
• Xvfb
• firefox v. 47 (or a chrome version that gets the tests to pass reliably)

There is an error trying to install bower, which is not a dependency of the 1.5 branch so it shouldn't be trying.

You need to manually restart it currently to get it working:

sudo service nginx reload

Also I think this repo could use some version tags as a lot of things are changed in it and it's quite hard to install and older version. You need to explicitly lookup a certain commit where it was changed.

Ansible deployment -- any changes in https://github.com/artefactual/deploy-pub merged to master or changes required in https://github.com/artefactual-labs/ansible-archivematica-src merged to qa/1.7.x

This is perhaps more of a question, or a suggestion. When using deploy-pub locally with Vagrant I see that deployments are significantly faster when I enable SSH pipelining with:

[ssh_connection]
pipelining = True

Would it be a good idea to include that directive in the ansible.cfg documents in this repository? It seems like a good default, officially supported by Ansible. Or would that be a conflict with other tools you use like mitogen?