infra cloud init fails, hanging stack-create about openshift-on-openstack HOT 8 CLOSED

Hi Judd,
cloud-init runs all scripts even if some of them fail which is not optimal because then it might happen that the last script signals false-positive successful return code (https://github.com/redhat-openstack/openshift-on-openstack/blob/master/fragments/infra-boot.sh).

What I'm not sure about is why check didn't catch the missing os-collect-config:
https://github.com/redhat-openstack/openshift-on-openstack/blob/master/fragments/infra-boot.sh#L229

This check was added exactly for making sure that os-collect-config is present to avoid the situation you hit.

from openshift-on-openstack.

If you hit it again, could you please attach /var/log/cloud-init(-output).log?

from openshift-on-openstack.

Failed again.

Here are the params and cloud-init.log attached. THANK YOU! I'm totally blocked by this, and I'm actively trying to work it out so I can progress.

| links | http://192.168.190.125:8004/v1/c1b740d8571f4cb5aa66f8ddcfdec015/stacks/openshift/3e7ce435-7c5a-4905-a5d4-86bbf6a448f5 (self) | [22/1163]
| notification_topics | [] |
| parameters | { |
| | "OS::project_id": "c1b740d8571f4cb5aa66f8ddcfdec015", |
| | "node_docker_volume_size_gb": "25", |
| | "openshift_nodes_min": "1", |
| | "rhn_pool": "_", |
| | "os_password": "__", |
| | "flavor": "m1.medium", |
| | "master_server_group_policies": "[u'affinity']", |
| | "deploy_registry": "True", |
| | "rhn_password": "_**", |
| | "os_username": "", |
| | "domain_name": "example.com", |
| | "loadbalancer_type": "neutron", |
| | "openshift_nodes_max": "5", |
| | "ssh_user": "cloud-user", |
| | "master_docker_volume_size_gb": "25", |
| | "os_tenant_name": "", |
| | "node_count": "4", |
| | "master_hostname": "openshift-master", |
| | "deployment_type": "openshift-enterprise", |
| | "OS::stack_name": "openshift", |
| | "autoscaling": "False", |
| | "master_count": "3", |
| | "os_region_name": "", |
| | "deploy_router": "True", |
| | "rhn_username": "dellcloudsol", |
| | "node_hostname": "openshift-node", |
| | "internal_subnet": "192.168.0.0/24", |
| | "skip_ansible": "False", |
| | "openshift_ansible_git_url": "", |
| | "infra_hostname": "infra", |
| | "skip_dns": "False", |
| | "OS::stack_id": "3e7ce435-7c5a-4905-a5d4-86bbf6a448f5", |
| | "extra_repository_urls": "[]", |
| | "os_auth_url": "", |
| | "external_network": "nova", |
| | "loadbalancer_image": "", |
| | "ssh_key_name": "key_name", |
| | "lb_hostname": "lb", |
| | "openshift_ansible_git_rev": "", |
| | "openshift_sdn": "openshift-sdn", |
| | "dns_nameserver": "[u'8.8.4.4', u'8.8.8.8']", |
| | "server_image": "rhel72" |
| | } |
| parent | None |
| stack_name | openshift |
| stack_owner | None |
| stack_status | CREATE_IN_PROGRESS |
| stack_status_reason | |
| stack_user_project_id | 9c985405ca194bc398eda5b9ff8c2241 |
| tags | null |
| template_description | Deploy Atomic/OpenShift 3 on OpenStack. |
| timeout_mins | 120 |
| updated_time | None |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------+
2016-05-16 16:09:05 [openshift]: CREATE_IN_PROGRESS Stack CREATE started
2016-05-16 16:09:05 [ansible_keys]: CREATE_IN_PROGRESS state changed

....

Gist of the cloud-init.log: https://gist.github.com/juddmaltin-dell/cc84f450e3778062abd8d257affcdcf3

from openshift-on-openstack.

The test here fails with a false positive:
https://github.com/redhat-openstack/openshift-on-openstack/blob/master/fragments/infra-boot.sh#L162

because it's called with --no-block

It fails because the os-collect-config service was installed without the os-collect-config package being installed.

os-collect-config lives in the rhel-7-server-openstack-8-director-rpms repo on another box of mine. But it does not live in any repos available to my OpenShift NFR pool-id.

The repos that exist from the OpenShift NFR pool-id are:

[root@openshift-infra ~]# yum repolist disabled|grep openstack
rhel-7-server-openstack-7.0-tools-debug-rpms/7Server/x86_64 Red Hat OpenSta
rhel-7-server-openstack-7.0-tools-rpms/7Server/x86_64 Red Hat OpenSta
rhel-7-server-openstack-7.0-tools-source-rpms/7Server/x86_64 Red Hat OpenSta
rhel-7-server-openstack-8-tools-debug-rpms/7Server/x86_64 Red Hat OpenSta
rhel-7-server-openstack-8-tools-rpms/7Server/x86_64 Red Hat OpenSta
rhel-7-server-openstack-8-tools-source-rpms/7Server/x86_64 Red Hat OpenSta

Can we attach multiple pool-ids so the os-collect-config is available?

from openshift-on-openstack.

Re-Opened because it's still maybe a bug.

from openshift-on-openstack.

Thanks for deep investigation. Yes, this is a bug and the "systemctl is-enabled os-collect-config" is insufficient - the reason is that the service file is not part of the package but is always created here:
https://github.com/redhat-openstack/openshift-on-openstack/blob/master/collect-config-setup/fragments/start_config_agent.sh#L13

Yum returns success event if some packages fail to install (https://github.com/redhat-openstack/openshift-on-openstack/blob/master/collect-config-setup/fragments/install_config_agent_yum.sh#L19) - this is not the first time I hit this "feature" which I really don't like :/. As a first step we should install os-collect-config in a separate "yum install".

from openshift-on-openstack.

Is it possible to include more than one pool? The OpenShift NFR pool is not reliably responding with -director- repos which include the os-collector-config package.

from openshift-on-openstack.

Thank you, fixed.

from openshift-on-openstack.