personium / ansible Goto Github PK

setenv.sh

target :
1-server_unit/README.md,3-server_unit/README.md
Modify point:
Ansible configuration white_check_mark
1: Install Ansible(Client server : Bastion server)

• NG:yum localinstall http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
• OK:yum localinstall http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm

Add a procedure to confirm the setting that depends on the user.

• SSL Certificate
• Unit Certificate

should use [all:vars].

Change the version of nginx installed by default when executing Ansible from 1.13.3 to 1.14.0.

1.13 series deviates from Stable, Mainline version,
Because it becomes an unsupported version

Review of unit-config.properties

• Consider introduction
• Confirm action
• Measure performance
• Modify the document

There is an inconsistency in the definition method of "base_url" defined in "$ ansbile / static_inventory / hosts".
In "$ ansible / resource / ap / personium / personium - core / conf / 18888 / personium - unit - config.properties.j2" you need to specify it in the form of "https: // FQDN".
But "init_service_elasticsearch.yml" does not work if "https: //" is specified.
Review the overall policy.

Large scale modification of Ansible.
Vagrant also needs to respond together.

• Correct logback.xml(reference:personium/personium-core#126)
• Unnecessary directory delet
• Review of unit-config.properties(#74)

Before executing ansible

# hostname
example.com

# cat /etc/hosts
...
127.0.0.1   localhost localhost.localdomain test-ap
127.0.0.1 unitadmin.example.com
127.0.0.1 example.com

After executing ansible

# hostname
test-ap

# cat /etc/hosts
...
127.0.0.1   localhost localhost.localdomain test-ap
127.0.0.1 unitadmin.example.com

Then, UNIT ADMIN Cell creation failed.

OS：CentOS 7.6
Memory : 8GB

TASK [Execute /tmp/personium-init-svcmgr.sh] *********************************************************************
fatal: [192.168.56.102]: FAILED! => {"changed": true, "cmd": "/bin/sh /tmp/personium-init-svcmgr.sh > /tmp/personium-init-svcmgr.log", "delta": "0:00:00.815381", "end": "2019-07-21 13:37:12.603507", "failed_when_result": true, "msg": "non-zero return code", "rc": 2, "start": "2019-07-21 13:37:11.788126", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

PLAY RECAP *******************************************************************************************************
192.168.56.102             : ok=268  changed=43   unreachable=0    failed=1    skipped=2    rescued=0    ignored=0

2019年  7月 21日 日曜日 13:37:12 JST

# cat /tmp/personium-init-svcmgr.log
######  Create UNIT ADMIN Cell  ######
HTTP/1.1 404
Date: Sun, 21 Jul 2019 04:37:12 GMT
Content-Type: application/json
Content-Length: 74
Connection: keep-alive
X-Personium-Version: 1.7.15
Server: Personium

{"code":"PR404-DV-0003","message":{"lang":"en","value":"Cell not found."}}
status:404
-- Cell check
HTTP/1.1 404
Date: Sun, 21 Jul 2019 04:37:12 GMT
Content-Type: application/json
Content-Length: 74
Connection: keep-alive
X-Personium-Version: 1.7.15
Server: Personium

{"code":"PR404-DV-0003","message":{"lang":"en","value":"Cell not found."}}
status:404
UNIT ADMIN Cell createdfaild.

See:
https://github.com/personium/ansible/blob/2fb4ca4ad446c3fb2b98adbcccfc2bf7bf6ef1f3/1-server_unit/tasks/common/init_hostname.yml

Overview

Currently, while executing Ansible, middleware is downloaded and used,
Basically, the version is specified as a variable, and the URL is created and downloaded using that variable.
However, this method can not cope with the case where the URL is changed unexpectedly.

It is necessary to take measures such as changing Ansible executor to set the URL itself.

Target URLs

The target will be the following URL listed in yuml of tasks.

• http://archive.apache.org/dist/tomcat/tomcat-8/v{{ tomcat_version }}/bin/apache-tomcat-{{ tomcat_version }}.tar.gz
• https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-{{ version }}.tar.gz
• http://www.memcached.org/files/memcached-{{ memcached_version }}.tar.gz
• http://nginx.org/download/nginx-{{ nginx_version }}.tar.gz
• https://github.com/agentzh/headers-more-nginx-module/archive/v{{ nginx_hm_version }}.tar.gz

(Note that the version is described directly)

• http://ftp.yz.yamagata-u.ac.jp/pub/network/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.zip

In addition, the following middleware is downloaded with wget of the command module.
Since there is no need to implement it with wget, it should be arranged in the get_url module.

• http://archive.apache.org/dist/activemq/5.15.2/apache-activemq-5.15.2-bin.tar.gz
• http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
• http://logback.qos.ch/dist/logback-1.0.3.tar.gz
• http://www.slf4j.org/dist/slf4j-1.6.4.tar.gz

Correction accompanying personium/legacy-docs#160 .

tagets:
README.md
1-server_unit/README.md
3-server_unit/README.md

• NG:CRLF
• OK:LF

git clone core repo in git_clone.yml is not idempotent because some files have CRLF line endings while decleared text eol=lf in .gitattributes, so that those files are regarded as “modified” just after cloned, and git in Ansible refuse to override modifications unless force: yes.

[/tmp/personium-core]$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   src/main/resources/ajax.js
	modified:   src/main/resources/html/authform.html
	modified:   src/main/resources/html/error.html

We should fix those line endings, or have force: yes in git_clone.yml.

We will test that Personium operates normally with Apache Tomcat 9.0.10.

Since Common Name is optional, it is mandatory to correct it.

Current ansible files are confirmed whether work on only Fujitsu K5 environment. We check them on other environment.

Overview

Currently "X-Personium-RequestKey" request header value is only written in the following log files.

• personium-core.log
• personium-engine.log
• Cell Level event log

and not written in the nginx logs, which is making log analysis difficult.

By including "X-Personium-RequestKey" request header value in the nginx log settings in our ansible file distribution, such difficulties will be removed.

How

Update nginx.conf file in the ansible playbook and change its log_format item.

see
https://github.com/personium/ansible/blob/master/1-server_unit/resource/web/opt/nginx/conf/nginx.conf#L40
https://siguniang.wordpress.com/2013/10/08/logging-request-response-headers-with-nginx/

Current self-signed certificate generation has 2 improvements.

• Example of CommonName value like '*.personium.example' is not documented.
• Subject alternative names are not set, so chrome (later 58) makes the error.

To set subject alternative names, following procedures are necessary.

# cat << EOS > san.txt
> subjectAltName = DNS:personium.example, DNS:*.personium.example
> EOS
# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt -extfile san.txt

#55 で｛base_url｝の指定方法が変更となった。
変更前：https://personium.example.com
変更後：personium.example.com

変更前と同じように「https://」を付けて指定されるとAnsibleの最後の方の処理でエラーとなる。
それを避けるため、Ansible実行時の初期段階でチェック処理を追加する。

チェック内容：先頭に「https://」が付いているか
付いている場合「https://」を省く

Add note of certificate

Refer to the description on the form of the following certificate and secret key
En
https://doc.cloud.global.fujitsu.com/lib/personiumservice/en/manual_en/00_manual.html#sect8

Ja
https://doc.cloud.global.fujitsu.com/lib/personiumservice/jp/manual/00_manual.html#sect8

Currently 1.6.14 is used, but change to use the latest.

Changed the copyright of the file corrected with version upgrade from 2017 to 2018
Target file:

• /1-server_unit/resource/ap/personium/personium-core/conf/18888/personium-unit-config.properties.j2
• /1-server_unit/resource/es/personium/check_cluster_health.sh
• /1-server_unit/tasks/bastion/git_clone.yml
• /1-server_unit/tasks/bastion/init_servicemanager.yml
• /1-server_unit/tasks/common/config_firewalld.yml
• /1-server_unit/tasks/es/init_service_elasticsearch.yml
• /3-server_unit/resource/ap/personium/personium-core/conf/18888/personium-unit-config.properties.j2
• /3-server_unit/resource/es/personium/check_cluster_health.sh
• /3-server_unit/tasks/bastion/git_clone.yml
• /3-server_unit/tasks/bastion/init_servicemanager.yml
• /3-server_unit/tasks/common/config_firewalld.yml
• /3-server_unit/tasks/es/init_service_elasticsearch.yml

Correction is necessary because README.md does not mention Ansible folder.
In doing so, we need the following description.

1. Create ansible directory in the local environment.
2. Copy the 1-server_unit (3-server_unit) obtained from git to the created ansible directory.

There are some warnings in the current 2.0.7 playbook. So it needs to handle them.

The warnings are:

• sudo

[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default). This feature will be removed in version 2.9.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

• restart_iptables.yml not found

[DEPRECATION WARNING]: Included file '/root/ansible/handlers/common/restart_iptables.yml' not found, however since this include is not explicitly marked as 'static:
yes', we will try and include it dynamically later. In the future, this will be an error unless 'static: no' is used on the include task. If you do not want missing
includes to be considered dynamic, use 'static: yes' on the include or set the global ansible.cfg options to make all includes static for tasks and/or handlers. This
feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

• old git version

 [WARNING]: Your git version is too old to fully support the depth argument. Falling back to full checkouts.

• Invoking "yum" only once while using a loop via squash_actions

[DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying
`pkg: "{{ item }}"`, please use `pkg: ['nfs-utils']` and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting
 deprecation_warnings=False in ansible.cfg.

• jinja2 templating delimiters

 [WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: "{{ disklabel }}" not in disk_label.stdout

• wget

 [WARNING]: Consider using the get_url or uri module rather than running 'wget'.  If you need to use command because get_url or uri is insufficient you can add 'warn:
false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.

• /usr/bin/python

 [WARNING]: Platform linux on host 172.18.10.6 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter
could change this. See https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html for more information.

• tar

 [WARNING]: Consider using the unarchive module rather than running 'tar'.  If you need to use command because unarchive is insufficient you can add 'warn: false' to
this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.

Setting default zone in config_firewalld.yml may fail if the firewalld was already started when Start firewalld is executed.

If the firewalld was already started, Start firewalld is skipped and /etc/firewalld/zones/personium-zone.xml, which is deployed by Deploy /etc/firewalld/zones/personium-zone.xml, is not loaded. The firewalld may be enabled by CentOS installer.

ansible.log:

2018-04-18 22:20:23,801 p=25468 u=root |  TASK [Setting default zone] ****************************************************
2018-04-18 22:20:25,719 p=25468 u=root |  fatal: [192.168.77.4]: FAILED! => {"changed": true, "cmd": ["firewall-cmd", "--set-default-zone=personium-zone"], "delta": "0:00:01.585800", "end": "2018-04-18 22:20:25.703559", "msg": "non-zero return code", "rc": 112, "start": "2018-04-18 22:20:24.117759", "stderr": "\u001b[91mError: INVALID_ZONE: personium-zone\u001b[00m", "stderr_lines": ["\u001b[91mError: INVALID_ZONE: personium-zone\u001b[00m"], "stdout": "", "stdout_lines": []}

Start firewalld should have state: restarted instead of state: started.

With the introduction of Personium 1.6.0, introduce ActiveMQ
Please note that ActiveMQ must be activated before tomcat startup

I got an error for token authentication (i.e. grant_type=urn:ietf:params:oauth:grant-type:saml2-bearer) with default installation because /opt/x509/cacert.crt is missing.

Sample invocation:

$ curl -k -s https://sample-personium/sample-user-01/__token -d "grant_type=urn:ietf:params:oauth:grant-type:saml2-bearer&assertion=$APP_USER_ACCESS_TOKEN"
{
  "error_description": "[PR400-AN-0009] - Token parse error.",
  "error": "invalid_grant"
}

/opt/tomcat/logs/catalina.log:

2018-05-09 14:26:02.949 [catalina-exec-5] [INFO ] PersoniumCoreLog [PL-AU-0001] - [io.personium.core.rs.cell.TokenEndPointResource#receiveSaml2:408] - Token parse error. Reason=/opt/x509/cacert.crt (no such file or directory)

personium-unit-config.properties.j2 have the following line:

io.personium.core.x509.root=/opt/x509/unit-self-sign.crt /opt/x509/cacert.crt

But with default installation, we don't have /opt/x509/cacert.crt while we have /opt/x509/unit-self-sign.crt. Changing the line to the following works, but I don't sure it is a correct way.

io.personium.core.x509.root=/opt/x509/unit-self-sign.crt

Maybe we should update the installation instructions to create cacert.crt, or delete /opt/x509/cacert.crt from personium-unit-config.properties.j2.

Calling restart_iptables.yml with ansible init_personium.yml for 3server configuration,
But restart_iptables.yml does not exist.
The change is required to use restart_firewalld.yml instead of restart_iptables.yml.

ansible/3-server_unit/init_personium.yml
ansible/3-server_unit/handlers/common/restart_firewalld.yml

Similar issue in setup-vagrant

wget or git supports environment value 'http_proxy', but maven does not support. So this ansible playbook failed in execution maven under http proxy even if you set 'http_proxy'.

This playbook needs maven proxy settings in settings.xml as follows .

<proxies>
  <proxy>
      <id>proxy00</id>
      <active>true</active>
      <protocol>http</protocol>
      <username>{username}</username>
      <password>{pass}</password>
      <host>{host}</host>
      <port>{port}</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
  </proxy>
  <proxy>
      <id>proxy01</id>
      <active>true</active>
      <protocol>https</protocol>
      <username>{username}</username>
      <password>{pass}</password>
      <host>{host}</host>
      <port>{port}</port>
      <nonProxyHosts>local.net|some.host.com</nonProxyHosts>
  </proxy>
</proxies>

Correct to use the version currently checking the operation.

• JDK 11 tests whether Personium operates normally.
• Upgrade

Overview

Do not use personium-rt account in OSS Personium.
So delete the personium-rt account creation process.

Correction target

• 1-server_unit\resource\bastion\tmp\personium-init-svcmgr.sh.j2
• 3-server_unit\resource\bastion\tmp\personium-init-svcmgr.sh.j2

Overview

Grant unitadmin role to unitadmin account of unitadmin cell

Precondition

It is necessary to correspond to Issue #31 before correspondence of this Issue

Correction target

• 1-server_unit\resource\bastion\tmp\personium-init-svcmgr.sh.j2
• 3-server_unit\resource\bastion\tmp\personium-init-svcmgr.sh.j2

Correspondence contents

• Create UnitAdmin role
• Create CellContentsAdmin role
• Link unitadmin - CellContentsAdmin

/1-server_unit/README.md
/3-server_unit/README.md

the link of "Create SSL certificate" (http://qiita.com/nenokido2000/items/1d1c79a0a443ed923e92x) has expired.

This "Create SSL certificate" link is in the column of confirmation after creation of SSL certificate / private key, it is substantially unnecessary and it deletes it.

Also, check other links for broken links.

From ansible 2.5 it became an error to list files that do not exist in the fetch module in src.
Therefore, correspondence is necessary.

init_servicemanager.yml

The file servicemanager_account does not currently exist. Remains of the past.
Therefore, delete the processing of servicemanager_account.

Remove Vagrant's [Sorry, this pattern does not work now. Please use 3 server version ansible.].
Also delete [Sorry, this pattern does not work now. Please use 3 server version ansible.] of 1-server.

Depending on the version of Personium, it becomes an error in the current personium_regression.sh. (Event API)
We need to fix personium_regression.sh to latest.
Also, since there is no processing to delete Box and Cell at the beginning of the shell,
it will fail with execution of the shell and will fail with Cell creation when re-executing.

Ansible 中の /opt/apache-maven-3.3.9/bin/mvn package -Dmaven.test.skip=true 実行中に、
pom.xml が見つからなく、以下のようなエラーとなる。

TASK [build module] *****************************************************************************************************************************************
fatal: [10.0.14.4]: FAILED! => {"changed": true, "cmd": ["/opt/apache-maven-3.3.9/bin/mvn", "package", "-Dmaven.test.skip=true"], "delta": "0:00:12.180705", "end": "2019-07-22 05:00:00.287837", "msg": "non-zero return code", "rc": 1, "start": "2019-07-22 04:59:48.107132", "stderr": "", "stderr_lines": [], "stdout": "[INFO] Scanning for projects...\n

※ 以下のファイルのコマンドに、-f オプションで指定すると回避はできる。
vi ./tasks/bastion/mvn_package.yml
command: /opt/apache-maven-3.3.9/bin/mvn package -Dmaven.test.skip=true
↓
command: /opt/apache-maven-3.3.9/bin/mvn package -Dmaven.test.skip=true -f /tmp/personium-{{ build_mod }}/pom.xml

The playbooks yaml is separated of bastion and web. But executing playbooks under bastion server and other web server, it failed. To solve it, followings should be fixed.

• move the yaml to bastion.
  

  ansible/3-server_unit/tasks/web/init_personium_regression.yml

  Line 4 in 2fb4ca4

  template: src=./resource/web/personium_regression.sh.j2 dest=/root/ansible/personium_regression.sh mode=744

• fix the source address to 'web_network_sep'
  

  ansible/3-server_unit/resource/ap/etc/firewalld/zones/personium-zone.xml.j2

  Line 14 in 2fb4ca4

  <source address="{{ bastion_network_sep }}"/>

• fix 127.0.0.1 to web private ip address.
  

  ansible/3-server_unit/resource/web/personium_regression.sh.j2

  Line 25 in 2fb4ca4

  echo "127.0.0.1" ${CELL_NAME}.${FQDN} >> /etc/hosts

Review of all materials related to ansible

ex.
1-server_unit/README.md

• NG:construct Personium unit on 4 servers using Ansible
• OK:construct Personium unit on 1 servers using Ansible

1-server_unit/README.md,3-server_unit/README.md

• NG:/home/demo/.ssh/id_rsa.pub
• OK:/root/.ssh/id_rsa.pub

1-server_unit/Ansible_Settings_Instruction.md,3-server_unit/Ansible_Settings_Instruction.md

• NG:root/.ssh/id_rsa
• OK:/root/.ssh/id_rsa

1-server_unit/README.md,3-server_unit/README.md

• NG:sudo su -,su root
• OK:su -

shoud replace with {{ ansible_ssh_user }}.

Because Tomcat 8 had EOL at the end of June.
It will not be available for download at the end of September.

https://tomcat.apache.org/tomcat-80-eol.html

When building with one server configuration,
Tomcat may start up before Elasticsearch, which may result in an error.

It is necessary to start Tomcat waiting for Elasticsearch to start up.

Target: How_to_generate_Self-signed_Unit_Certificate.md

An example of specifying the domain name of the unit at CSR creation is example.com,
If you copy it mistakenly and execute Ansible, it will fail with personium-init-svcmgr.sh execution.

• Modify the following notes written in How_to_generate_Self - Signed_Unit_Certificate.md in a noticeable way.
  　Common Name value should be the unit domain name (required)

• For the explanation of the example, it is stated that Personium unit domain name is "personium.example.com".

The key length of the server certificate is short. Currently it is 1024 bits, but it changes to 2048 bits.