fiunchinho / ansible-aws-elasticsearch-module Goto Github PK

View Code? Open in Web Editor NEW

24.0 24.0 20.0 17 KB

Ansible module to create a new AWS ElasticSearch cluster

Python 100.00%

ansible-aws-elasticsearch-module's People

Contributors

Stargazers

Watchers

ansible-aws-elasticsearch-module's Issues

Error: IncompleteSignatureException

I'm using echo -e "{{ec2_access_key}}\n{{ec2_secret_key}}\n{{aws_region}}\njson\n" | aws --profile aws.sandbox configure to create the profile. Should I be using a different method?

ansible 2.2.1.0
aws-cli/1.11.49 Python/2.7.13 Darwin/16.4.0 botocore/1.5.12

main.yml

---

- name: "Create ElasticSearch cluster"
  ec2_elasticsearch:
    name: "elastic-cloud"
    elasticsearch_version: "5.1"
    region: "{{aws_region}}"
    instance_type: "m3.medium.elasticsearch"
    instance_count: 1
    dedicated_master: False
    #dedicated_master_instance_type: "t2.micro.elasticsearch"
    #dedicated_master_instance_count: 2
    zone_awareness: False
    ebs: False
    volume_type: "standard"
    volume_size: 10
    snapshot_hour: 0
    access_policies: "{{ lookup('file', 'access_policy.json') | from_json }}"
    profile: "aws.sandbox"
  register: response

AWS IAM Profile Policy:
AmazonRDSFullAccess
AmazonEC2FullAccess
IAMFullAccess
AmazonElastiCacheFullAccess
AmazonS3FullAccess
AWSElasticBeanstalkFullAccess
AmazonVPCFullAccess
AmazonRoute53FullAccess

Config not working

I am using this custom Elasticsearch Service with Ansible. The following Config produces ValidationException.

- hosts: 127.0.0.1
  connection: local

  tasks:
    - name: "Create ElasticSearch cluster"
      ec2_elasticsearch:
        name: "Testing"
        elasticsearch_version: "2.3"
        region: "us-east-1"
        instance_type: "t2.medium.elasticsearch"
        instance_count: 1
        dedicated_master: False
        zone_awareness: False
        ebs: True
        volume_type: "standard"
        volume_size: 10
        snapshot_hour: 1
        access_policies: "{{ lookup('file', 'es_policies.json') | from_json }}"
      register: es_info

Output with -vvv

No config file found; using defaults

PLAYBOOK: setup_elasticsearch.yaml *********************************************
1 plays in data_pipeline/ansible/setup_elasticsearch.yaml

PLAY [127.0.0.1] ***************************************************************

TASK [setup] *******************************************************************
Using module file /Users/nehiljian/.pyenv/versions/2.7.11/envs/recommender_env/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: nehiljain
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922 `" && echo ansible-tmp-1486674667.03-193453987758922="` echo $HOME/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922 `" ) && sleep 0'
<127.0.0.1> PUT /var/folders/gn/zdn4js5n7m96lpdtvx7h7d440000gn/T/tmpfDSuhx TO /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/ /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/nehiljian/.pyenv/versions/2.7.11/envs/recommender_env/bin/python2.7 /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/setup.py; rm -rf "/Users/nehiljian/.ansible/tmp/ansible-tmp-1486674667.03-193453987758922/" > /dev/null 2>&1 && sleep 0'
ok: [127.0.0.1]

TASK [Create ElasticSearch cluster] ********************************************
task path: ansible/setup_elasticsearch.yaml:6
Using module file ansible/library/ec2_elasticsearch.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: nehiljain
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884 `" && echo ansible-tmp-1486674668.31-219225481684884="` echo $HOME/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884 `" ) && sleep 0'
<127.0.0.1> PUT /var/folders/gn/zdn4js5n7m96lpdtvx7h7d440000gn/T/tmpfajB91 TO /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ec2_elasticsearch.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ec2_elasticsearch.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/nehiljian/.pyenv/versions/2.7.11/envs/recommender_env/bin/python2.7 /Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/ec2_elasticsearch.py; rm -rf "/Users/nehiljian/.ansible/tmp/ansible-tmp-1486674668.31-219225481684884/" > /dev/null 2>&1 && sleep 0'
fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "access_policies": {
                "Statement": [
                    {
                        "Action": "es:*",
                        "Effect": "Allow",
                        "Principal": {
                            "AWS": "arn:aws:iam::XXXXXXXXXX:user/APIAccess"
                        },
                        "Resource": "arn:aws:es:us-east-1:XXXXXXXXXX:domain/xxxxx/*"
                    },
                    {
                        "Action": "es:*",
                        "Condition": {
                            "IpAddress": {
                                "aws:SourceIp": [
                                    "xx.xx.xx.xx"
                                ]
                            }
                        },
                        "Effect": "Allow",
                        "Principal": {
                            "AWS": "*"
                        },
                        "Resource": "arn:aws:es:us-east-1:XXXXXXXXXX:domain/xxxxx/*",
                        "Sid": ""
                    }
                ],
                "Version": "2012-10-17"
            },
            "aws_access_key": null,
            "aws_secret_key": null,
            "dedicated_master": false,
            "dedicated_master_instance_count": null,
            "dedicated_master_instance_type": null,
            "ebs": true,
            "ec2_url": null,
            "elasticsearch_version": "2.3",
            "instance_count": 1,
            "instance_type": "t2.medium.elasticsearch",
            "name": "Testing",
            "profile": "default",
            "region": "us-east-1",
            "security_token": null,
            "snapshot_hour": 1,
            "validate_certs": true,
            "volume_size": 10,
            "volume_type": "standard",
            "zone_awareness": false
        },
        "module_name": "ec2_elasticsearch"
    },
    "msg": "Error: ValidationException"
}
	to retry, use: --limit @ansible/setup_elasticsearch.retry

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=1    changed=0    unreachable=0    failed=1

Any clue what is wrong? Thanks

Unknown parameter in input: "ElasticsearchVersion"

Perhaps something with the aws sdk changed but I wasn't able to get this module to work for me.

Traceback (most recent call last):
  File "/tmp/ansible_xGoIah/ansible_module_ec2_elasticsearch.py", line 239, in <module>
    main()
  File "/tmp/ansible_xGoIah/ansible_module_ec2_elasticsearch.py", line 225, in main
    AccessPolicies=pdoc,
  File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/client.py", line 228, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/client.py", line 468, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/client.py", line 521, in _convert_to_request_dict
    api_params, operation_model)
  File "/home/bsmith01/.local/lib/python2.7/site-packages/botocore/validate.py", line 269, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "ElasticsearchVersion", must be one of: DomainName, ElasticsearchClusterConfig, EBSOptions, AccessPolicies, SnapshotOptions, AdvancedOptions

Here is my config:

- name: "Create ElasticSearch cluster"
  ec2_elasticsearch:
    name: "my-cluster-test"
    elasticsearch_version: "2.3"
    region: "us-east-1"
    instance_type: "t2.small.elasticsearch"
    instance_count: "3"
    zone_awareness: False
    dedicated_master: False
    ebs: True
    volume_type: "gp2"
    volume_size: 50
    snapshot_hour: 13
    access_policies: "{{ lookup('template', playbook_dir + '/roles/retraced/templates/elasticsearch_iam.yml.j2') }}"
    profile: "default"
  register: response

thanks!

Module should be published on ansible galaxy

My fork is currently the only published version at this stage, be good to have the original up.

License mismatch

Hello,

I was looking at using this earlier but noticed a license mismatch: the meta/main.yml file says MIT, but the ec2_elasticsearch.py says GPL3. Which is it?

Support EbsVolume and DedicatedMaster = false

I ran into errors with this module when setting ebs and dedicated_master to false.

I suggest something like the following-

    esconfig = {
           'InstanceType': module.params.get('instance_type'),
           'InstanceCount': int(module.params.get('instance_count')),
           'DedicatedMasterEnabled': module.params.get('dedicated_master'),
           'ZoneAwarenessEnabled': module.params.get('zone_awareness')
    }

    ebsoptions = {
           'EBSEnabled': module.params.get('ebs')
    }

    if esconfig['DedicatedMasterEnabled']:
        esconfig['DedicatedMasterType'] = module.params.get('dedicated_master_instance_type')
        esconfig['DedicatedMasterCount'] = int(module.params.get('dedicated_master_instance_count'))

    if ebsoptions['EBSEnabled']:
        ebsoptions['VolumeType'] = module.params.get('volume_type')
        ebsoptions['VolumeSize'] = int(module.params.get('volume_size'))

    response = client.create_elasticsearch_domain(
        DomainName = module.params.get('name'),
        ElasticsearchClusterConfig = esconfig,
        EBSOptions = ebsoptions,
        SnapshotOptions = {
            'AutomatedSnapshotStartHour': module.params.get('snapshot_hour')
        }
    )

Module is not idempotent

If you run this module twice, it will put the cluster into a processing state, even if there are no changes. Since the processing state takes 10-20 minutes to finish, this is a big time sink.

There should probably be a request to describe the elasticsearch domain and only perform updates if there are differences in configuration?

fiunchinho / ansible-aws-elasticsearch-module Goto Github PK

ansible-aws-elasticsearch-module's People

Contributors

Stargazers

Watchers

Forkers

ansible-aws-elasticsearch-module's Issues

Error: IncompleteSignatureException

Config not working

Unknown parameter in input: "ElasticsearchVersion"

Module should be published on ansible galaxy

License mismatch

Support EbsVolume and DedicatedMaster = false

Module is not idempotent

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent