pndaproject / pnda-heat-templates Goto Github PK

View Code? Open in Web Editor NEW

2.0 14.0 26.0 902 KB

Resources for launching PNDA on Openstack

License: Other

Python 27.79% Shell 72.21%

pnda-heat-templates's Introduction

PNDA Heat Templates

Requirements

Python

To use the PNDA command line interface, you will need to install the python, heat and nova clients. To install them on Ubuntu, run:

$ sudo apt-get -y update
$ sudo apt-get -y install python python-pip python-dev
$ cd cli
$ sudo pip install -r requirements.txt

Git credentials

The stack will make use of git repositories. In case git authentication is required, a private key is required. It is then necessary to have an ssl keypair, whose public key is registered in the git server. To authenticate against the server the private key value must be passed. By default a deploy file must be present in the templates directory. If you already own a keypair, it probably stands in the .ssh subdirectory of your home directory. If this keypair is registered against the git server, it is only necessary to copy its content into the deploy file.

$ cd pnda-heat-templates
$ cp ~/.ssh/id_rsa deploy

If you do not own a keypair or if your public key value is not stored on the git server, please follow this guide to generate a public key.

Also uncomment #deploy and specifiy url of your git repo in pnda_env.yaml file

Deploy PNDA using the CLI

Pre-requisite

As a pre-requisite to use the CLI, it is necessary to setup some openstack required environment variables. The easy way to do so is to retrieve an rc file from the horizon dashboard of your OpenStack platform, in the context of the targeted OpenStack project. Once retrieved just source it.

$ . <project>-openrc.sh

Environment configuration

The Heat PNDA stack default parameters can be set in the pnda_env.yaml file. Please refer to the file ENVIRONMENTS.md in the pnda-heat-templates repository for a full description of every configuration point.

As this file is specific to the target environment it needs to be created by the PNDA integrator. You can start by renaming the example pnda_env_example.yaml to pnda_env.yaml or create your own following this guide and the instructions in ENVIRONMENTS.md.

A brief summary of the main points follows.

You MUST update the KEYSTONE CREDENTIALS section in the pnda_env.yaml file with your openstack credentials in order for PNDA to use swift. For example:

keystone_user: 'pnda'
keystone_password: '32165468321654'
keystone_tenant: 'pnda'
keystone_auth_url: 'https://openstack.example.com:5000/v2.0'
keystone_auth_version: '2'    
keystone_region_name: 'europe'

You MUST edit the pnda_apps_container, pnda_apps_folder parameters to change which location to use for the application repository in Swift. For example:

pnda_apps_container: 'pnda.applications'
pnda_apps_folder: 'releases'

You MUST edit the pnda_archive_container to point to the Swift container where data will be archived. For example:

pnda_archive_container: 'pnda_archives'

You can optionally add the JavaMirror, ClouderaParcelsMirror and AnacondaParcelsMirror parameters to point to your private mirrors of the Sun Oracle Java tarball and Cloudera Parcels mirror. It can make deployment faster. For example:

JavaMirror: 'http://pnda-mirror.example.com/java/jdk/8u74-b02/jdk-8u74-linux-x64.tar.gz'
ClouderaParcelsMirror: 'http://pnda-mirror.example.com/mirror/archive.cloudera.com/cdh5/parcels/5.9.0/'

Package repository

The default backend storage for the package repository is Swift and so you should have the containers created as defined with the parameters 'pnda_apps_container' and 'pnda_apps_folder' as describe just above. So then, for the package repository backend storage type, it is the 'fstype' argument of the heat cli, it could be either:

'swift': in case you will store your packages within Swift so the configuration will be:

  package_repository_fs_type: 'swift'

's3': in case you will want to use AWS S3 so the configuration will be:

  # AWS configuration
  S3_ACCESS_KEY_ID=xxxx
  S3_SECRET_ACCESS_KEY=xxxx
  AWS_REGION=xxxx

'sshfs': if you want to have the packages on another host which will be connected from the package repository host. For more information about sshfs, go to SSHFS libfuse. The configuration will be:

  package_repository_fs_location_path: '/opt/pnda/packages'
  package_repository_sshfs_user: 'cloud-user'
  package_repository_sshfs_host: '127.0.0.1'
  package_repository_sshfs_path: '/mnt/packages'
  package_repository_sshfs_key: pr_key

Do not forget also to copy the key.pem file used by sshfs to connect to the remote host:

$ cd pnda-heat-templates
$ cp PATH/key.pem pr_key

'volume': in case of the standard flavor, we have attached a volume in order to be able to store lot of packages and this is mount by default on the edge node. The configuration will be:

  package_repository_fs_location_path: '/mnt/packages'

'local': if you want the packages to be store localy on the package repository node. The configuration will be:

  package_repository_fs_location_path: '/opt/pnda/packages'

The CLI

The heat_cli.py scripts allows to launch a PNDA deployment. It sits in the cli subdirectory.

$ cd cli
$ ./heat_cli.py
usage: heat_cli.py [-h] [-y] [-e PNDA_CLUSTER] [-n DATANODES]
                   [-o OPENTSDB_NODES] [-k KAFKA_NODES] [-z ZK_NODES]
                   [-f {standard}] [-b BRANCH] [-s KEYPAIR]
                   {create,destroy}

"the -s KEYPAIR will be used to connect to the bastion instance through ssh. KEYPAIR refers to the name given to the public key of the pair in the target OpenStack tenant. Therefore, a pre-requisite is that you should create a new OpenStack keypair, or import an existing one to the OpenStack platform.

Make sure you have access to the private key of KEYPAIR otherwise you will not be able to connect to the bastion node and access the cluster."

There are various PNDA flavors: this allows changing role distribution across the resources.

Creating a PNDA cluster

We create a standard deployment named cation. We specify 3 data nodes, 1 opentsdb node, 2 kafka nodes, 3 zookeeper nodes:

$ ./heat_cli.py -e cation -n 3 -o 1 -k 2 -z 3 -f standard -s <existing_neutron_keypair> create
+--------------------------------------+------------------+--------------------+----------------------+--------------+
| id                                   | stack_name       | stack_status       | creation_time        | updated_time |
+--------------------------------------+------------------+--------------------+----------------------+--------------+
| 80d510fe-8fb5-4fbc-b4ba-a65cafa12103 | cation           | CREATE_IN_PROGRESS | 2016-04-11T12:47:22Z | None         |
+--------------------------------------+------------------+--------------------+----------------------+--------------+
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| Property              | Value                                                                                                                  |
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| capabilities          | []                                                                                                                     |
| creation_time         | 2016-04-11T12:47:22Z                                                                                                   |
| description           | No description                                                                                                         |
| disable_rollback      | True                                                                                                                   |
| id                    | 80d510fe-8fb5-4fbc-b4ba-a65cafa12103                                                                                   |
| links                 | http://XXX.XXX.XXX.XXX:8004/v1/7875ed368a8043449c9dc58aac73ab7e/stacks/cation/80d510fe-8fb5-4fbc-b4ba-a65cafa12103 (self) |
| notification_topics   | []                                                                                                                     |
| parameters            | {                                                                                                                      |
|                       |   "OS::project_id": "7875ed368a8043449c9dc58aac73ab7e",                                                                |
|                       |   "private_net_cidr": "10.10.10.0/24",                                                                                 |
|                       |   "EdgeFlavor": "m1.xlarge",                                                                                           |
|                       |   "private_net_pool_end": "10.10.10.200",                                                                              |

[...]

| stack_user_project_id | d4e029af62ee4b9bbc7c038e2b5ef350                                                                                       |
| template_description  | No description                                                                                                         |
| timeout_mins          | None                                                                                                                   |
| updated_time          | None                                                                                                                   |
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
2016-04-11 12:47:23Z [cation]: CREATE_IN_PROGRESS  Stack CREATE started
2016-04-11 12:47:23Z [saltmaster_sec_group]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:24Z [highstate_config]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:24Z [swift_config]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:24Z [PndaSecGroup]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:25Z [private_net]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:25Z [orchestrate_config]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:25Z [router]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:26Z [install_config]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:26Z [Key]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:26Z [orchestrate_config]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [saltmaster_sec_group]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [router]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [install_config]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [private_net]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [swift_config]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [highstate_config]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [Key]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [PndaSecGroup]: CREATE_COMPLETE  state changed
2016-04-11 12:47:27Z [private_subnet]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:29Z [private_subnet]: CREATE_COMPLETE  state changed
2016-04-11 12:47:29Z [saltmaster_port]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:30Z [router_interface]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:47:31Z [saltmaster_port]: CREATE_COMPLETE  state changed
2016-04-11 12:47:31Z [router_interface]: CREATE_COMPLETE  state changed
2016-04-11 12:47:31Z [saltmaster_server]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:48:04Z [saltmaster_server]: CREATE_COMPLETE  state changed
2016-04-11 12:48:04Z [deploy_install]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:50:40Z [deploy_install]: CREATE_COMPLETE  state changed
2016-04-11 12:50:40Z [pnda_cluster]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:53:31Z [pnda_cluster]: CREATE_COMPLETE  state changed
2016-04-11 12:53:31Z [deploy_swift]: CREATE_IN_PROGRESS  state changed
2016-04-11 12:54:36Z [deploy_swift]: CREATE_COMPLETE  state changed
2016-04-11 12:54:36Z [deploy_highstate]: CREATE_IN_PROGRESS  state changed
2016-04-11 13:01:34Z [deploy_highstate]: CREATE_COMPLETE  state changed
2016-04-11 13:01:34Z [deploy_orchestrate]: CREATE_IN_PROGRESS  state changed
2016-04-11 13:33:31Z [deploy_orchestrate]: CREATE_COMPLETE  state changed
2016-04-11 13:33:31Z [cation]: CREATE_COMPLETE  Stack CREATE completed successfully

[...]

 Stack cation CREATE_COMPLETE

Anatomy of a PNDA stack

If we examine the previously created stack :

$ heat stack-list -n | grep cation
| 80d510fe-8fb5-4fbc-b4ba-a65cafa12103 | cation                                                                  | CREATE_COMPLETE | 2016-04-11T12:47:22Z | None         | None                                 |
| d8c4e4aa-b4de-4742-a430-d85553e66987 | cation-pnda_cluster-obz7kxxsnatx                                       | CREATE_COMPLETE | 2016-04-11T12:50:40Z | None         | 80d510fe-8fb5-4fbc-b4ba-a65cafa12103 |
| 3ed6689c-45c1-4201-aab2-96beb2a82974 | cation-pnda_cluster-obz7kxxsnatx-bastion-ccdyikzg2gar                  | CREATE_COMPLETE | 2016-04-11T12:50:44Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| 7e911922-a4d5-416b-a69f-5ec3871f2677 | cation-pnda_cluster-obz7kxxsnatx-edge-pribufdbiz3s                     | CREATE_COMPLETE | 2016-04-11T12:50:45Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| cc35476a-e5ab-4620-9d38-0f28ce2dc84d | cation-pnda_cluster-obz7kxxsnatx-cm-anaz62owf25q                       | CREATE_COMPLETE | 2016-04-11T12:50:46Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| fa48786c-84c1-42ed-adac-167e850b396b | cation-pnda_cluster-obz7kxxsnatx-manager1-pkyrsg6kywg5                 | CREATE_COMPLETE | 2016-04-11T12:50:49Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| d5c7baf7-0115-4b97-ac6c-2a9ae2a9f8e4 | cation-pnda_cluster-obz7kxxsnatx-opentsdb-pdzsym7ukyvz                 | CREATE_COMPLETE | 2016-04-11T12:50:52Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| f4bd32b0-9df8-40eb-a13f-b8ee3f7c5260 | cation-pnda_cluster-obz7kxxsnatx-zookeeper-2pqkyjnmt4b4                | CREATE_COMPLETE | 2016-04-11T12:50:52Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| 689e88ca-0ea0-4747-8753-dfbcd25c0bcd | cation-pnda_cluster-obz7kxxsnatx-opentsdb-pdzsym7ukyvz-0-koz6d4y52euh  | CREATE_COMPLETE | 2016-04-11T12:50:54Z | None         | d5c7baf7-0115-4b97-ac6c-2a9ae2a9f8e4 |
| 436f9371-8430-4937-9580-bd923d9e8f78 | cation-pnda_cluster-obz7kxxsnatx-tools-rlnmhbimn7le                    | CREATE_COMPLETE | 2016-04-11T12:50:55Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| 482f1e2a-ba44-48f3-92d5-3aa7f4e15515 | cation-pnda_cluster-obz7kxxsnatx-zookeeper-2pqkyjnmt4b4-0-ekw7zjofmcr6 | CREATE_COMPLETE | 2016-04-11T12:50:56Z | None         | f4bd32b0-9df8-40eb-a13f-b8ee3f7c5260 |
| d7e4d502-4057-4afa-be87-fd27c93323c0 | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx                       | CREATE_COMPLETE | 2016-04-11T12:50:57Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| 557f23e6-205b-4ad4-8ef3-d489c692ca46 | cation-pnda_cluster-obz7kxxsnatx-tools-rlnmhbimn7le-0-ww3hsajtgf6v     | CREATE_COMPLETE | 2016-04-11T12:50:58Z | None         | 436f9371-8430-4937-9580-bd923d9e8f78 |
| ca0ed237-3026-44b9-8bb9-7813913f561a | cation-pnda_cluster-obz7kxxsnatx-kafka-cz7budcvxmvk                    | CREATE_COMPLETE | 2016-04-11T12:50:58Z | None         | d8c4e4aa-b4de-4742-a430-d85553e66987 |
| 52e1a7ed-c7f4-419d-b7d5-6749dcc498a4 | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx-0-yykdys4kctd6        | CREATE_COMPLETE | 2016-04-11T12:50:59Z | None         | d7e4d502-4057-4afa-be87-fd27c93323c0 |
| 8e0318ed-5b1b-46f7-ae0f-766fb8077a82 | cation-pnda_cluster-obz7kxxsnatx-kafka-cz7budcvxmvk-0-r2e7tx2lmlkb     | CREATE_COMPLETE | 2016-04-11T12:51:00Z | None         | ca0ed237-3026-44b9-8bb9-7813913f561a |

A PNDA deployment is a heat stack, and nested stacks deploying a cluster made of again another nested stack per role/cluster. Each line in the previous output describes a heat stack which can be examinded again.

We can have a look at the resources of the stack :

$ heat resource-list cation
+----------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+
| resource_name        | physical_resource_id                                                                | resource_type                | resource_status | updated_time         |
+----------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+
| deploy_highstate     | 2bc4d5ba-441d-439e-ba41-f2ebb3009b76                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:22Z |
| deploy_orchestrate   | 04c82c5a-470c-4bbf-8105-6b602b0e3994                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:22Z |
| deploy_swift         | 92ca062d-ed22-4167-a02f-31ee4eb902c9                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:22Z |
| pnda_cluster        | d8c4e4aa-b4de-4742-a430-d85553e66987                                                | OS::Pnda::cluster           | CREATE_COMPLETE | 2016-04-11T12:47:22Z |
| Key                  | cation                                                                              | OS::Nova::KeyPair            | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| PndaSecGroup        | 5852df8a-2c10-4d27-b11e-acdfcc2b40f2                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| deploy_install       | e625e78c-15b1-430f-ae69-6bb41f05efd4                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| highstate_config     | 1cd5fbee-c6b9-4596-858f-deac94dc1060                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| install_config       | b664a413-0a2e-46c8-b6ba-3303045a582c                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| orchestrate_config   | f70201c2-5814-41b2-bdb1-9158903a8a4e                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| private_net          | c232c59f-9cb7-457f-aa8b-50bc285ba14f                                                | OS::Neutron::Net             | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| private_subnet       | 54156224-3749-496c-b39c-bd5c6121e9d6                                                | OS::Neutron::Subnet          | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| router               | 31a1a963-d686-4488-aa66-5d655f416308                                                | OS::Neutron::Router          | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| router_interface     | 31a1a963-d686-4488-aa66-5d655f416308:subnet_id=54156224-3749-496c-b39c-bd5c6121e9d6 | OS::Neutron::RouterInterface | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| saltmaster_port      | 94177921-9829-4630-93aa-ba50117980c0                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| saltmaster_sec_group | b06b1142-3c7e-4115-8b04-5424071fd134                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| saltmaster_server    | d5f4f0d7-70e0-4d54-bb94-11766e25c471                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
| swift_config         | b49e4621-995e-4779-b360-1067f9082928                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z |
+----------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+

And ultimately the resources from the nested stacks as well :

$ heat resource-list -n 5 cation                                                                                                                                                        [113/489]
+----------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+---------------------------------------
----------------------------------+
| resource_name        | physical_resource_id                                                                | resource_type                | resource_status | updated_time         | stack_name
                                  |
+----------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+---------------------------------------
----------------------------------+
| deploy_highstate     | 2bc4d5ba-441d-439e-ba41-f2ebb3009b76                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:22Z | cation
                                  |
| deploy_orchestrate   | 04c82c5a-470c-4bbf-8105-6b602b0e3994                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:22Z | cation
                                  |
| deploy_swift         | 92ca062d-ed22-4167-a02f-31ee4eb902c9                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:22Z | cation
                                  |
| pnda_cluster        | d8c4e4aa-b4de-4742-a430-d85553e66987                                                | OS::Pnda::cluster           | CREATE_COMPLETE | 2016-04-11T12:47:22Z | cation
                                  |
| Key                  | cation                                                                              | OS::Nova::KeyPair            | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| PndaSecGroup        | 5852df8a-2c10-4d27-b11e-acdfcc2b40f2                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| deploy_install       | e625e78c-15b1-430f-ae69-6bb41f05efd4                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| highstate_config     | 1cd5fbee-c6b9-4596-858f-deac94dc1060                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| install_config       | b664a413-0a2e-46c8-b6ba-3303045a582c                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| orchestrate_config   | f70201c2-5814-41b2-bdb1-9158903a8a4e                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| private_net          | c232c59f-9cb7-457f-aa8b-50bc285ba14f                                                | OS::Neutron::Net             | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| private_subnet       | 54156224-3749-496c-b39c-bd5c6121e9d6                                                | OS::Neutron::Subnet          | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| router               | 31a1a963-d686-4488-aa66-5d655f416308                                                | OS::Neutron::Router          | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| router_interface     | 31a1a963-d686-4488-aa66-5d655f416308:subnet_id=54156224-3749-496c-b39c-bd5c6121e9d6 | OS::Neutron::RouterInterface | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| saltmaster_port      | 94177921-9829-4630-93aa-ba50117980c0                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| saltmaster_sec_group | b06b1142-3c7e-4115-8b04-5424071fd134                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| saltmaster_server    | d5f4f0d7-70e0-4d54-bb94-11766e25c471                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| swift_config         | b49e4621-995e-4779-b360-1067f9082928                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:47:23Z | cation
                                  |
| bastion              | 3ed6689c-45c1-4201-aab2-96beb2a82974                                                | OS::Pnda::bastion           | CREATE_COMPLETE | 2016-04-11T12:50:41Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| cm                   | cc35476a-e5ab-4620-9d38-0f28ce2dc84d                                                | OS::Pnda::cm                | CREATE_COMPLETE | 2016-04-11T12:50:41Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| edge                 | 7e911922-a4d5-416b-a69f-5ec3871f2677                                                | OS::Pnda::edge              | CREATE_COMPLETE | 2016-04-11T12:50:41Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| opentsdb             | d5c7baf7-0115-4b97-ac6c-2a9ae2a9f8e4                                                | OS::Heat::ResourceGroup      | CREATE_COMPLETE | 2016-04-11T12:50:41Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| DNSecGroup           | fb6e6b38-40d1-4b80-85a1-626969c8dbe7                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| KafkaSecGroup        | 958ada28-4e65-47c6-929c-e2b4445313ad                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| OpentsdbSecGroup     | 191320d3-3ecf-47e8-8e6b-bb259a282e05                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| ToolsSecGroup        | 4141c2fb-ee52-4828-a125-433234f1b14b                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| ZKSecGroup           | 332fe76d-4c66-486a-8b8c-6f4550ea8588                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| dn                   | d7e4d502-4057-4afa-be87-fd27c93323c0                                                | OS::Heat::ResourceGroup      | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| kafka                | ca0ed237-3026-44b9-8bb9-7813913f561a                                                | OS::Heat::ResourceGroup      | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| manager1             | fa48786c-84c1-42ed-adac-167e850b396b                                                | OS::Pnda::manager1          | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| tools                | 436f9371-8430-4937-9580-bd923d9e8f78                                                | OS::Heat::ResourceGroup      | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| zookeeper            | f4bd32b0-9df8-40eb-a13f-b8ee3f7c5260                                                | OS::Heat::ResourceGroup      | CREATE_COMPLETE | 2016-04-11T12:50:42Z | cation-pnda_cluster-obz7kxxsnatx
                                  |
| floatingIP           | 0c45ee3f-0817-4cd7-852f-c2331cb367e9                                                | OS::Neutron::FloatingIP      | CREATE_COMPLETE | 2016-04-11T12:50:44Z | cation-pnda_cluster-obz7kxxsnatx-bast
ion-ccdyikzg2gar                  |
| install_config       | 03f94be1-2673-43eb-ae22-1f90ef71e2f6                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:50:44Z | cation-pnda_cluster-obz7kxxsnatx-bast
ion-ccdyikzg2gar                  |
| install_deployment   | 8a83fe6a-e494-4460-b9a0-1bd2860be5e0                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:50:44Z | cation-pnda_cluster-obz7kxxsnatx-bast
ion-ccdyikzg2gar                  |
| port                 | ecbb0a84-9da8-4ed4-af52-4d58c97fe029                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:44Z | cation-pnda_cluster-obz7kxxsnatx-bast
ion-ccdyikzg2gar                  |
| sec_group            | 6447961b-b12c-4719-a068-4a8cb4161b3a                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:44Z | cation-pnda_cluster-obz7kxxsnatx-bast
ion-ccdyikzg2gar                  |
| server               | 62b63880-a80a-4142-adc7-6a0e258546c1                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:50:44Z | cation-pnda_cluster-obz7kxxsnatx-bast
ion-ccdyikzg2gar                  |
| install_config       | 3c54dbfc-5fcc-4b8f-bef4-b443806363f5                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:50:45Z | cation-pnda_cluster-obz7kxxsnatx-edge
-pribufdbiz3s                     |
| install_deployment   | 3c7e3df2-1520-4eec-84e7-4ef764c95614                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:50:45Z | cation-pnda_cluster-obz7kxxsnatx-edge
-pribufdbiz3s                     |
| port                 | 1628aace-0f3f-4edb-8152-31fb6c098a03                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:45Z | cation-pnda_cluster-obz7kxxsnatx-edge
-pribufdbiz3s                     |
| sec_group            | 510b83e9-bf7c-424e-8dc1-a8dc60bc701a                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:45Z | cation-pnda_cluster-obz7kxxsnatx-edge
-pribufdbiz3s                     |
| server               | feeff5db-e3c6-4035-8654-38bf70f09677                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:50:45Z | cation-pnda_cluster-obz7kxxsnatx-edge
-pribufdbiz3s                     |
| install_config       | 219ea1b4-730d-4e0a-8928-70de06c0cf20                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:50:46Z | cation-pnda_cluster-obz7kxxsnatx-cm-a
naz62owf25q                       |
| install_deployment   | 24d36818-2447-4161-8b22-40e5c9373e2f                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:50:46Z | cation-pnda_cluster-obz7kxxsnatx-cm-a
naz62owf25q                       |
| port                 | 9992d132-918b-4121-a8d9-d17078bdbbdc                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:46Z | cation-pnda_cluster-obz7kxxsnatx-cm-a
naz62owf25q                       |
| sec_group            | 01ce3f03-1740-4769-924c-5698206c7714                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:46Z | cation-pnda_cluster-obz7kxxsnatx-cm-a
naz62owf25q                       |
| server               | 3a71802d-c5f7-40f7-aae2-56ea64b25071                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:50:46Z | cation-pnda_cluster-obz7kxxsnatx-cm-a
naz62owf25q                       |
| install_config       | b6e42e21-d4eb-4c71-b5be-29ac4bd514d8                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:50:49Z | cation-pnda_cluster-obz7kxxsnatx-mana
ger1-pkyrsg6kywg5                 |
| install_deployment   | 0388f83f-dab8-4398-a812-1fae1d3648c4                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:50:49Z | cation-pnda_cluster-obz7kxxsnatx-mana
ger1-pkyrsg6kywg5                 |
| port                 | 51fb1400-7067-4434-9dce-3b714efb0eaa                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:49Z | cation-pnda_cluster-obz7kxxsnatx-mana
ger1-pkyrsg6kywg5                 |
| sec_group            | ae76a14a-3f4d-4309-a0b9-3f54ea24072b                                                | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-04-11T12:50:49Z | cation-pnda_cluster-obz7kxxsnatx-mana
ger1-pkyrsg6kywg5                 |
| server               | 8cd4101f-efa3-4fe3-9a2e-8ba55fcd1885                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:50:49Z | cation-pnda_cluster-obz7kxxsnatx-mana
ger1-pkyrsg6kywg5                 |
| 0                    | 482f1e2a-ba44-48f3-92d5-3aa7f4e15515                                                | OS::Pnda::zookeeper         | CREATE_COMPLETE | 2016-04-11T12:50:52Z | cation-pnda_cluster-obz7kxxsnatx-zookeeper-2pqkyjnmt4b4                |
| 0                    | 689e88ca-0ea0-4747-8753-dfbcd25c0bcd                                                | OS::Pnda::opentsdb          | CREATE_COMPLETE | 2016-04-11T12:50:52Z | cation-pnda_cluster-obz7kxxsnatx-opentsdb-pdzsym7ukyvz                 |
| install_deployment   | d8016fcb-ee82-43c8-ae63-dbd87bd607cc                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:50:54Z | cation-pnda_cluster-obz7kxxsnatx-opentsdb-pdzsym7ukyvz-0-koz6d4y52euh  |
| port                 | 3741d558-040e-4899-aa09-0acd55fa5ed1                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:55Z | cation-pnda_cluster-obz7kxxsnatx-opentsdb-pdzsym7ukyvz-0-koz6d4y52euh  |
| 0                    | 557f23e6-205b-4ad4-8ef3-d489c692ca46                                                | OS::Pnda::tools             | CREATE_COMPLETE | 2016-04-11T12:50:56Z | cation-pnda_cluster-obz7kxxsnatx-tools-rlnmhbimn7le                    |
| install_config       | 95cca8b6-e463-4383-803e-8c5729714834                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:50:56Z | cation-pnda_cluster-obz7kxxsnatx-opentsdb-pdzsym7ukyvz-0-koz6d4y52euh  |
| server               | e8f63fb4-36cd-4e2d-be3b-c4e2e58d2da8                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:50:56Z | cation-pnda_cluster-obz7kxxsnatx-opentsdb-pdzsym7ukyvz-0-koz6d4y52euh  |
| 0                    | 52e1a7ed-c7f4-419d-b7d5-6749dcc498a4                                                | OS::Pnda::dn                | CREATE_COMPLETE | 2016-04-11T12:50:57Z | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx                       |
| install_config       | dd671a45-fb27-4b76-a10d-807bf87fd7b0                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:50:57Z | cation-pnda_cluster-obz7kxxsnatx-zookeeper-2pqkyjnmt4b4-0-ekw7zjofmcr6 |
| install_deployment   | 87147bbb-7a4d-46e9-8d27-0a45a0678f84                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:50:57Z | cation-pnda_cluster-obz7kxxsnatx-zookeeper-2pqkyjnmt4b4-0-ekw7zjofmcr6 |
| port                 | 6b55e5d5-6309-4a38-ad58-8fe588ab7f57                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:57Z | cation-pnda_cluster-obz7kxxsnatx-zookeeper-2pqkyjnmt4b4-0-ekw7zjofmcr6 |
| server               | 834e3201-316b-4b77-81c8-6e48af9f6476                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:50:57Z | cation-pnda_cluster-obz7kxxsnatx-zookeeper-2pqkyjnmt4b4-0-ekw7zjofmcr6 |
| install_config       | d1d4046a-6537-411d-9bb1-846243215801                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:50:58Z | cation-pnda_cluster-obz7kxxsnatx-tools-rlnmhbimn7le-0-ww3hsajtgf6v     |
| install_deployment   | e2acf060-ad02-4785-9900-1cf71c365097                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:50:58Z | cation-pnda_cluster-obz7kxxsnatx-tools-rlnmhbimn7le-0-ww3hsajtgf6v     |
| 0                    | 8e0318ed-5b1b-46f7-ae0f-766fb8077a82                                                | OS::Pnda::kafka             | CREATE_COMPLETE | 2016-04-11T12:50:59Z | cation-pnda_cluster-obz7kxxsnatx-kafka-cz7budcvxmvk                    |
| port                 | 7061585f-3b60-41b4-a458-3ad443f59adf                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:59Z | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx-0-yykdys4kctd6        |
| port                 | c2ad7911-67ed-4632-8a9f-2e2238ce2836                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:50:59Z | cation-pnda_cluster-obz7kxxsnatx-tools-rlnmhbimn7le-0-ww3hsajtgf6v     |
| server               | 38fd7a6b-e846-410d-90ba-dd222e5aabd9                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:50:59Z | cation-pnda_cluster-obz7kxxsnatx-tools-rlnmhbimn7le-0-ww3hsajtgf6v     |
| install_config       | b678c181-2e12-4d0b-9a8c-1cd85cb78348                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:51:00Z | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx-0-yykdys4kctd6        |
| install_deployment   | 6d46cb0b-0066-4b24-bb53-fc3a1b92c2dd                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:51:00Z | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx-0-yykdys4kctd6        |
| server               | d4013a0c-8ae5-4bda-a101-2b618471cfdd                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:51:00Z | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx-0-yykdys4kctd6        |
| volume               | ccd54e94-ed5d-40de-9e44-7ac0ea68c599                                                | OS::Cinder::Volume           | CREATE_COMPLETE | 2016-04-11T12:51:00Z | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx-0-yykdys4kctd6        |
| volume_attachment    | ccd54e94-ed5d-40de-9e44-7ac0ea68c599                                                | OS::Cinder::VolumeAttachment | CREATE_COMPLETE | 2016-04-11T12:51:00Z | cation-pnda_cluster-obz7kxxsnatx-dn-c47jwj3v4idx-0-yykdys4kctd6        |
| install_config       | f35da354-b6af-47c2-b242-10e0cda7f697                                                | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-04-11T12:51:01Z | cation-pnda_cluster-obz7kxxsnatx-kafka-cz7budcvxmvk-0-r2e7tx2lmlkb     |
| install_deployment   | 2f62cb64-19c7-4cb5-ac9e-57d1f57f520e                                                | OS::Heat::SoftwareDeployment | CREATE_COMPLETE | 2016-04-11T12:51:01Z | cation-pnda_cluster-obz7kxxsnatx-kafka-cz7budcvxmvk-0-r2e7tx2lmlkb     |
| port                 | 11b75015-7178-4830-a1a4-dbab9a360562                                                | OS::Neutron::Port            | CREATE_COMPLETE | 2016-04-11T12:51:01Z | cation-pnda_cluster-obz7kxxsnatx-kafka-cz7budcvxmvk-0-r2e7tx2lmlkb     |
| server               | c420c207-ae8e-47de-b81d-c3551ce3254e                                                | OS::Nova::Server             | CREATE_COMPLETE | 2016-04-11T12:51:01Z | cation-pnda_cluster-obz7kxxsnatx-kafka-cz7budcvxmvk-0-r2e7tx2lmlkb     |
+----------------------+-------------------------------------------------------------------------------------+------------------------------+-----------------+----------------------+-------------------------------------------------------------------------+

Examining the provisioned instances :

$ nova list | grep cation
| 61f96b51-264a-4e76-8b2b-d0d287f24be8 | cation-lab-0-bastion     | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.18, 10.60.18.34 |
| e8091ceb-222f-4d91-b5bf-04943da01558 | cation-lab-0-hadoop-cm      | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.19              |
| 953c4e1e-6671-49ee-aced-838547f71514 | cation-lab-0-hadoop-dn-0    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.31              |
| af64183c-e674-4824-8fb0-bcaf193e7ea7 | cation-lab-0-hadoop-dn-1    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.35              |
| a802878e-32a9-49fe-9d6a-3b431d6f888c | cation-lab-0-hadoop-dn-2    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.36              |
| a6d86587-04ca-4385-8871-faf3fd67061b | cation-lab-0-hadoop-edge    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.24              |
| 22c6fcd0-ebfe-4bd9-ae48-401f6a83dcbb | cation-lab-0-hadoop-jupyter | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.20              |
| 35987268-b218-4f46-9fa1-6a81ea7a982a | cation-lab-0-hadoop-mgr1    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.17              |
| 7c9f25d8-ee58-4d43-9177-d0a40e19b6c6 | cation-lab-0-hadoop-mgr2    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.16              |
| a7411e57-34b0-4d87-ad62-1d1b471cfe5e | cation-lab-0-hadoop-mgr3    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.14              |
| eebf2ac7-d25e-49db-8b70-78c3b7c49774 | cation-lab-0-hadoop-mgr4    | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.15              |
| fcb087ab-789f-4a13-891a-71ac70fc2b3f | cation-lab-0-collector   | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.38, 10.60.18.57 |
| e7941b24-dec3-405c-b61d-cbedd315b0c8 | cation-lab-0-kafka-0     | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.25              |
| b3ba4c44-939d-4fc4-bc88-c2657ffafc52 | cation-lab-0-kafka-1     | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.23              |
| 9854cc56-780f-49d8-bd11-1e2086ab886a | cation-lab-0-logserver   | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.22              |
| 4aca9943-b5f1-4599-8fdb-0f9f4b4df3c4 | cation-lab-0-opentsdb-0  | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.37              |
| 95a21ded-71ab-4576-9850-3ab0ac82938d | cation-lab-0-saltmaster  | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.13              |
| aa4c214d-1d9a-423e-b8e4-43bcaec2fa5c | cation-lab-0-tools-0     | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.28              |
| 49090d85-3d2b-4135-b1e7-f28c00b982fd | cation-lab-0-zookeeper-0 | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.32              |
| bf1cbadb-5910-45ec-8aaf-918f2b7f4231 | cation-lab-0-zookeeper-1 | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.29              |
| cae2fd4d-6d81-4c29-b75f-b15510329a7c | cation-lab-0-zookeeper-2 | ACTIVE | -          | Running     | cation-lab-0-net=192.168.10.26              |

The only instance reachable from outside the deployment is the bastion instance. It can be sshed:

$ ssh -i <your_private_key file> [email protected]

From the bastion instance you can then ssh to the other instances using the dynamically created private key whose file sits in the cloud-user home directory.

cloud-user@cation-bastion:~$ ls -l
total 4
-rw------- 1 cloud-user cloud-user 1675 Apr 11 12:51 cation-pnda_cluster-obz7kxxsnatx-bastion-ccdyikzg2gar.pem

ssh to the saltmaster instance:

cloud-user@cation-bastion:~$ ssh -i cation-pnda_cluster-obz7kxxsnatx-bastion-ccdyikzg2gar.pem salt
The authenticity of host 'salt (10.10.10.13)' can't be established.
ECDSA key fingerprint is 67:f4:3d:79:7e:a1:7f:94:59:6e:83:ba:2d:bd:23:1c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'salt,10.10.10.13' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-83-generic x86_64)

[...]

cloud-user@cation-saltmaster:~$

References

Some interesting links:

pnda-heat-templates's People

Contributors

Stargazers

Watchers

pnda-heat-templates's Issues

Kafka Vlan grains missing

Recent changes in platform-salt to fetch the Kafka ingest and pnda interface name is changed, removed the default value, and added the value in bootstrap script,
So we need to update this in Heat template, for AWS Kafka node uses the 2 ports (ingest and pnda) , but in open stack only one is defined.

Repo change :
pndaproject/platform-salt@55d13d9#diff-a049e53b5544d0b9c1ee40d66378f2e0L13
-{%- set internal_ip = salt'network.interface_ip' -%}
-{%- set ingest_ip = salt'network.interface_ip' -%}
+{%- set internal_ip = salt'network.interface_ip' -%}
+{%- set ingest_ip = salt'network.interface_ip' -%}

New Grains in AWS , is not available in heat
https://github.com/pndaproject/pnda-cli/blob/develop/bootstrap-scripts/kafka.sh#L24
cat >> /etc/salt/grains <<EOF
vlans:
pnda: $PNDA_INTERNAL_NETWORK
ingest: $PNDA_INGEST_NETWORK
EOF

Correction of "hdp_core_stack_repo" path

Old path:
hdp_core_stack_repo: '$pnda_mirror$/mirror_hdp/HDP/$HDP_OS/'

New path:
hdp_core_stack_repo: '$pnda_mirror$/mirror_hdp/HDP/$HDP_OS/2.6.3.0-235/'

Modification required in salt-master bootstarp-script file "saltmaster_install.sh"

Error validating value 'trusty-pnda-cloud-user-software-config-mirror': No images matching

Hi,
I comment the dns_nameserversin pnda_env_standard.yaml, but another error I get:
JSON response : {"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": **{"message": "Property error: : resources.saltmaster_server.properties.image: : Error validating value 'trusty-pnda-cloud-user-software-config-mirror': No images matching {'name': u'trusty-pnda-cloud-user-software-config-mirror'}. (HTTP 404)"**, "traceback": "Traceback (most recent call last):\n\n File \"/opt/stack/heat/heat/common/context.py\", line 424, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/opt/stack/heat/heat/engine/service.py\", line 823, in create_stack\n template_id)\n\n File \"/opt/stack/heat/heat/engine/service.py\", line 698, in _parse_template_and_validate_stack\n stack.validate()\n\n File \"/usr/lib/python2.7/site-packages/osprofiler/profiler.py\", line 147, in wrapper\n return f(*args, **kwargs)\n\n File \"/opt/stack/heat/heat/engine/stack.py\", line 802, in validate\n result = res.validate()\n\n File \"/opt/stack/heat/heat/engine/resources/openstack/nova/server.py\", line 1391, in validate\n super(Server, self).validate()\n\n File \"/opt/stack/heat/heat/engine/resource.py\", line 1449, in validate\n return self.validate_template()\n\n File \"/opt/stack/heat/heat/engine/resource.py\", line 1480, in validate_template\n message=ex.error_message)\n\nStackValidationFailed: Property error: resources.saltmaster_server.properties.image: Error validating value 'trusty-pnda-cloud-user-software-config-mirror': No images matching {'name': u'trusty-pnda-cloud-user-software-config-mirror'}. (HTTP 404)\n", "type": "StackValidationFailed"}, "title": "Bad Request"} from (pid=4927) to_json /opt/stack/heat/heat/common/serializers.py:40
What should I config or where can I get the image?
Thank you!

Hadoop installation failed as execution failed due to "java illegal argument hostname".

Mgr node template in pnda-heat-templates provide hostname in /etc/hosts as "%stackname%-hadoop-mgr1", whereas in platform-salt templates we see the hadoop name node as "%(cluster_name)s-hadoop-mgr-1". This causes hostname name resolution failure.

"hadoop_distro" param missing in templates/pico/mgr1.yaml.j2

Heat template of pico flavor for "mgr node" calls the bootstrap script "base_install.sh".
The script expects parameter "hadoop_distro" to set the value of the grain.
This parameter is not passed to this script in the mgr1.yaml.j2 file.

SALTSTACK-GPG-KEY.pub fails through http from http://repo.saltstack.com

Getting salt stack key from
http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2015.8.11/SALTSTACK-GPG-KEY.pub
and
http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2015.8.11/ trusty main

is failing, if the the company infrastructure not allowed http, but https is working.

Issue :
https://github.com/pndaproject/pnda-heat-templates/blob/develop/scripts/package_install.sh#L13

Fix :
Change http to https.

Creating PNDA on OpenStack.It always stop with [deploy_package]: CREATE_FAILED CREATE aborted.

I encountered a problem when I built the PNDA platform on openstack with heat.Your help is greatly appreciated.
1.The command:
root@test:/home/pnda/git3/pnda-heat-templates/cli# ./heat_cli.py -e cation -f pico -s mykey create
The log is as follows：

2018-03-20 01:07:43 [cation]: CREATE_IN_PROGRESS Stack CREATE started
2018-03-20 01:07:43 [pre_config]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:43 [expand_config]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:43 [PndaSecGroup]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:44 [install_config]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:44 [Key]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:44 [private_net]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:44 [saltmaster_sec_group]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:45 [orchestrate_config]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:45 [package_config]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:45 [router]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:45 [highstate_config]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:45 [expand_config]: CREATE_COMPLETE state changed
2018-03-20 01:07:45 [saltmaster_sec_group]: CREATE_COMPLETE state changed
2018-03-20 01:07:45 [orchestrate_config]: CREATE_COMPLETE state changed
2018-03-20 01:07:45 [install_config]: CREATE_COMPLETE state changed
2018-03-20 01:07:45 [pre_config]: CREATE_COMPLETE state changed
2018-03-20 01:07:45 [PndaSecGroup]: CREATE_COMPLETE state changed
2018-03-20 01:07:45 [router]: CREATE_COMPLETE state changed
2018-03-20 01:07:46 [package_config]: CREATE_COMPLETE state changed
2018-03-20 01:07:46 [highstate_config]: CREATE_COMPLETE state changed
2018-03-20 01:07:46 [Key]: CREATE_COMPLETE state changed
2018-03-20 01:07:46 [private_net]: CREATE_COMPLETE state changed
2018-03-20 01:07:46 [private_subnet]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:47 [private_subnet]: CREATE_COMPLETE state changed
2018-03-20 01:07:47 [saltmaster_port]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:48 [router_interface]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:49 [router_interface]: CREATE_COMPLETE state changed
2018-03-20 01:07:49 [saltmaster_port]: CREATE_COMPLETE state changed
2018-03-20 01:07:49 [saltmaster_server]: CREATE_IN_PROGRESS state changed
2018-03-20 01:07:59 [saltmaster_server]: CREATE_COMPLETE state changed
2018-03-20 01:07:59 [deploy_package]: CREATE_IN_PROGRESS state changed
2018-03-20 03:07:43 [deploy_package]: CREATE_FAILED CREATE aborted
2018-03-20 03:07:43 [cation]: CREATE_FAILED Create timed out

Stack cation CREATE_FAILED

It always stop at [deploy_package] for a long time.It eventually ended because it reached timeout(120 minutes).Did I misconfigured something?
Sometime I run the command with other parameters.For example:
./heat_cli.py -e cation -n 3 -o 1 -k 2 -z 3 -f standard -s mykey create
But it have the same problem.
2.I configure the parameter 'keystone_tenant' in pnda_env.yaml.
keystone_tenant: 'pnda'
Should I created the project 'pnda' before creating PNDA?
3.The keypair 'mykey' is created by the user 'demo' in openstack.
It is exist before create PNDA.
I run the command with the parameter -s mykey.
Why there will create a keypair 'cation' during the process of creating PNDA?

The swift list output:

The content of a file named 592a5fa5-94a4-4748-973b-4e0bfd91f725 which in container cation-saltmaster_server-iljkq4mpppti.

{   
        "os-collect-config": 
            {
                "request": {
                    "metadata_url": "http://controller:8081/v1/AUTH_d5d19fdbe22e4f679ea38fde968b9308/cation-saltmaster_server-iljkq4mpppti/592a5fa5-94a4-4748-973b-4e0bfd91f725?temp_url_sig=caf167275fc365562174612ce32332729d573c30&temp_url_expires=2147483587"
                }, 
                "collectors": ["ec2", "request", "local"]
             }, 
         "deployments": []
    }

Some parameters in pnda_env.yaml:

os_user: 'root'
keystone_user: 'demo'
keystone_tenant: 'pnda'
keystone_auth_url: 'http://10.10.4.13:5000/v3'
keystone_auth_version: '3'
PndaMirror: 'http://10.10.4.13/pnda-root/'
git_private_key_file: deploy
package_repository_fs_type:  'swift'
pnda_archive_container:  'pnda_archive'
pnda_apps_container:  'pnda_apps'
pnda_apps_folder:  'releases'

My environment:

operating system:Ubuntu 14.04 server amd64 LTS
OpenStack:Mitaka
swift bind_port:8081

Thank you very much.

How can I launch the console ?

Hi,
I create a pnda cluster with the command ./heat_cli.py create -e squirrel-land -f standard -n 5 -o 1 -k 2 -z 3 -s pnda -y.
And now,

[root@devstack162 pnda-heat-templates]# openstack stack list
+--------------------------------------+---------------+--------------------+---------------------+--------------+
| ID                                   | Stack Name    | Stack Status       | Creation Time       | Updated Time |
+--------------------------------------+---------------+--------------------+---------------------+--------------+
| 4aa9d000-7033-4552-890a-0d545bb23978 | squirrel-land | CREATE_IN_PROGRESS | 2016-08-23T07:43:16 | None         |
+--------------------------------------+---------------+--------------------+---------------------+--------------+
[root@devstack162 pnda-heat-templates]# heat resource-list squirrel-land
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+----------------------+--------------------------------------+------------------------------+-----------------+---------------------+
| resource_name        | physical_resource_id                 | resource_type                | resource_status | updated_time        |
+----------------------+--------------------------------------+------------------------------+-----------------+---------------------+
| Key                  | squirrel-land                        | OS::Nova::KeyPair            | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| PndaSecGroup         | 91d371dc-9329-4289-a589-9864f6147046 | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| deploy_expand        |                                      | OS::Heat::SoftwareDeployment | INIT_COMPLETE   | 2016-08-23T07:43:17 |
| deploy_highstate     |                                      | OS::Heat::SoftwareDeployment | INIT_COMPLETE   | 2016-08-23T07:43:17 |
| deploy_install       |                                      | OS::Heat::SoftwareDeployment | INIT_COMPLETE   | 2016-08-23T07:43:17 |
| deploy_orchestrate   |                                      | OS::Heat::SoftwareDeployment | INIT_COMPLETE   | 2016-08-23T07:43:17 |
| expand_config        | 2f237e89-e01d-416f-a74e-2a56f447a819 | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| highstate_config     | 3793dc6c-b3f2-472c-bc65-8df28182c254 | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| install_config       | 43432319-a4fc-4d83-bd23-7572122729e2 | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| orchestrate_config   | 137837c7-0775-494d-ac54-736c630e70d2 | OS::Heat::SoftwareConfig     | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| pnda_cluster         |                                      | OS::Pnda::cluster            | INIT_COMPLETE   | 2016-08-23T07:43:17 |
| private_net          | 1887e09d-4b08-49ef-af01-d163fbcdf9d1 | OS::Neutron::Net             | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| private_subnet       | 5b69c8bd-2bf6-4c6f-bbaa-39410c5cc383 | OS::Neutron::Subnet          | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| router               |                                      | OS::Neutron::Router          | INIT_COMPLETE   | 2016-08-23T07:43:17 |
| router_interface     |                                      | OS::Neutron::RouterInterface | INIT_COMPLETE   | 2016-08-23T07:43:17 |
| saltmaster_port      | 2aeee4f1-7865-4840-b159-23a47ec1983b | OS::Neutron::Port            | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| saltmaster_sec_group | b5f33925-2cb8-4580-b621-840d5ec303a7 | OS::Neutron::SecurityGroup   | CREATE_COMPLETE | 2016-08-23T07:43:17 |
| saltmaster_server    |                                      | OS::Nova::Server             | INIT_COMPLETE   | 2016-08-23T07:43:17 |
+----------------------+--------------------------------------+------------------------------+-----------------+---------------------+

Then, I connect to http://squirrel-land-cdh-edge/ to launch the console , and I get the text Name Error: The domain name does not exist..
Is it because the dns_name_server not configed?
Thanks you.

Cleanup of unicode characters erroneously present due to the previous commit

Cleanup of Unicode characters erroneously present due to the previous commit

QA tests were successful with those same files. Unicode characters could have been introduced while committing but changes weren't seen git CLI

Pnda stack completed sucessfully, but errors in the logs

Hi,

I created pnda stack successfully using OpenStack. When finished the command "./heat_cli.py -e cation -f pico -s pnda_key create -o 1 -n 1 -k 1 -z1", the terminal showed the message "Stack CREATE completed successfully". I can access the nodes via SSH and can connect to the PNDA console (via http). But I checked the logs and found some errors. How can I solve them? Or are these errors not important?

Logs: https://pastebin.com/4jqgY0mQ

Pnda: Release 3.5
OpenStack: Mitaka (Devstack) installed on Ubuntu 16.04 Hardware: 72 GB RAM, 32 cpu, 500 GB.
Installation client: Ubuntu Server 14.04

ambari-server installation is failing

ambari-installation is failing during HDP deployment, as following public key is not imported:

rpm --import $PNDA_MIRROR/mirror_rpm/RPM-GPG-KEY-Jenkins

The above code needs to be added in package-install.sh under pnda-heat-templates

Property error: : resources.private_subnet.properties.dns_nameservers is not a list

Hi,
I get the next error, when I run the command to create a pnda cluster.
What's wrong?
Thanks!

[root@devstack162 cli]# ./heat_cli.py create -e squirrel-land -f standard -n 3 -o 1 -k 1 -z 1 -s pnda -y
+---------+
| P N D A |
+---------+

Please wait while your PNDA cluster is being created.

This process can last for 1 or 2 hours.

openstack stack create --timeout 120 --wait --template pnda_standard.yaml --environment pnda_env_standard.yaml --parameter ZookeeperNodes=1 --parameter KafkaNodes=1 --parameter DataNodes=3 --parameter OpentsdbNodes=1 --parameter PndaFlavor=standard --parameter KeyName=pnda squirrel-land
**ERROR: Property error: : resources.private_subnet.properties.dns_nameservers: : "u"**[u'173.38.200.100']"" is not a list

Need help option for check-stack.sh tool

in check-stack.sh tool to be update for help option for better understanding about the command line options.

pnda-heat-template for Pico flavor

I am trying to setup pnda pico flavor on openstack.

Please share me the heat_template_version: 2014-10-16 for this pnda pico setup.

cluster_deployment issue

I use release/3.4 to create image,
seems all nodes use ec2-user now, but script for salt-master in the heat deployment still use cloud-user,
I checked /etc/password ,no cloud-user now,
Is this a bug? or any update?

pnda 3.5 heat stack deploys successfully but PNDA console inaccessible

Hi,

I am using devstack Openstack newton release and trying to deploy pico flavor. Below are the logs. Your help would be really appreciated.

https://pastebin.com/r6fpQhAF

Does it necessary to have mysql on mgr1 VM?

Thanks

Document : Keystone and swift timer configs

When deploying PNDA, some times, exit with 401 unauthorized error due to time out, The swift and keystone configuration to be updated in the Openstack setup.

Swift Auth Error

I am trying to setup pnda standard on openstack 3.0.0 version. Pnda_cluster deploy fails with below error. Your help is greatly appreciated. I am stuck at this point and can’t proceed further.

I have created containers in admin tenant. And below are my pnda_env.yaml settings.

keystone_user: 'admin'

keystone_user: Password for the openstack clients to use

keystone_password: 'admin1'

keystone_tenant: Name of the tenant / project in the openstack environment. The

PNDA stack will be created in this project.

keystone_tenant: 'service'

keystone_auth_url: Keystone authentication URL. The Openstack console provides this

under the Access & Security section.

I am trying to setup pnda standard on openstack 3.0.0 version. Pnda_cluster deploy fails with below error. Your help is greatly appreciated. I am stuck at this point and can’t proceed further.

I have created containers in admin tenant. And below are my pnda_env.yaml settings.

keystone_user: 'admin'

keystone_user: Password for the openstack clients to use

keystone_password: 'admin1'

keystone_tenant: Name of the tenant / project in the openstack environment. The

PNDA stack will be created in this project.

keystone_tenant: 'admin'

keystone_auth_url: Keystone authentication URL. The Openstack console provides this

under the Access & Security section.

Resource Create Failed: Remoteerror: Resources.Deploy Install: Remote Error: Clientexception Object Head Failed: Http://147.75.99.215:8080/V1/Auth 88798ee06ebd4409be26e437cd07ec35/A4913856-C4e6-4a3c-9b5d-647e3767ccdd/Pnda-Cluster-Deploy Install-4hzazk5l5obh 401 Unauthorized [U'Traceback (Most Recent Call Last):\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Oslo Messaging/Rpc/Server.Py", Line 133, In Process Incoming\N Res = Self.Dispatcher.Dispatch(Message)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Oslo Messaging/Rpc/Dispatcher.Py", Line 150, In Dispatch\N Return Self. Do Dispatch(Endpoint, Method, Ctxt, Args)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Oslo Messaging/Rpc/Dispatcher.Py", Line 121, In Do Dispatch\N Result = Func(Ctxt, **New Args)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Osprofiler/Profiler.Py", Line 154, In Wrapper\N Return F(*Args, **Kwargs)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Heat/Common/Context.Py", Line 424, In Wrapped\N Return Func(Self, Ctx, *Args, **Kwargs)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Heat/Engine/Service.Py", Line 2215, In Show Software Deployment\N Cnxt, Deployment Id)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Heat/Engine/Service Software Config.Py", Line 239, In Show Software Deployment\N Sd = Self. Show Software Deployment(Cnxt, Deployment Id)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Heat/Engine/Service Software Config.Py", Line 232, In Show Software Deployment\N Cnxt, Sd, Input Values.Get('Deploy Signal Id'))\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Heat/Engine/Service Software Config.Py", Line 136, In Refresh Swift Software Deployment\N Headers = Swift.Head Object(Container, Object Name)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Swiftclient/Client.Py", Line 1744, In Head Object\N Return Self. Retry(None, Head Object, Container, Obj, Headers=Headers)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Swiftclient/Client.Py", Line 1647, In Retry\N Service Token=Self.Service Token, **Kwargs)\N', U' File "/Var/Lib/Kolla/Venv/Local/Lib/Python2.7/Site-Packages/Swiftclient/Client.Py", Line 1186, In Head Object\N Raise Clientexception.From Response(Resp, 'Object Head Failed', Body)\N', U'Clientexception: Object Head Failed: Http://147.75.99.215:8080/V1/Auth 88798ee06ebd4409be26e437cd07ec35/A4913856-C4e6-4a3c-9b5d-647e3767ccdd/Pnda-Cluster-Deploy Install-4hzazk5l5obh 401 Unauthorized\N'].

Pillar data fails to render on salt-master node

Pillar fails to render with following message:
"Specified SLS 'hadoop.$' in environment 'base' is not available on the salt master"

In AWS environment edge node also takes role of salt-master, "hadoop.distro" grain is added from bootstrap-script "base.sh".
Whereas in OpenStack environment, a dedicated node serves as salt-master server. While rendering pillars, it expects "hadoop.distro" grain.

This grain "hadoop.distro" should be added on salt-master bootstrap-script.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

pndaproject / pnda-heat-templates Goto Github PK

pnda-heat-templates's Introduction

PNDA Heat Templates

Requirements

Python

Git credentials

Deploy PNDA using the CLI

Pre-requisite

Environment configuration

Package repository

The CLI

Creating a PNDA cluster

Anatomy of a PNDA stack

References

pnda-heat-templates's People

Contributors

Stargazers

Watchers

Forkers

pnda-heat-templates's Issues

keystone_user: Password for the openstack clients to use

keystone_tenant: Name of the tenant / project in the openstack environment. The

PNDA stack will be created in this project.

keystone_auth_url: Keystone authentication URL. The Openstack console provides this

under the Access & Security section.

keystone_user: Password for the openstack clients to use

keystone_tenant: Name of the tenant / project in the openstack environment. The

PNDA stack will be created in this project.

keystone_auth_url: Keystone authentication URL. The Openstack console provides this

under the Access & Security section.

Recommend Projects

Recommend Topics

Recommend Org