joaquincasares / cassandralauncher Goto Github PK

It would be great to have the ability to balance the ec2 nodes instead of just specifying the ec2 placement. The script would select different zones for each node that way having a better availability across different zones.

Traceback (most recent call last):
File "/usr/local/bin/imagelauncher", line 4, in
cassandralauncher.ClusterLauncher()
File "/Library/Python/2.7/site-packages/cassandralauncher/init.py", line 11, in init
clusterlauncher.run()
File "/Library/Python/2.7/site-packages/cassandralauncher/clusterlauncher.py", line 259, in run
main()
File "/Library/Python/2.7/site-packages/cassandralauncher/clusterlauncher.py", line 196, in main
ec2.terminate_cluster(config.get('EC2', 'aws_access_key_id'), config.get('EC2', 'aws_secret_access_key'), config.get('Shared', 'handle'))
TypeError: terminate_cluster() takes at least 4 arguments (3 given)

I'm getting the following. Was wondering if this is a key permissions issue or an AMI permissions issue?

Thanks!

Using configuration file: ~/.clusterlauncher.conf

Welcome to DataStax' Cassandra Cluster Launcher!
The easiest way to get Apache Cassandra up and running in Amazon's EC2
in under 5 minutes!

ShortName (Handle): tron-east
AWS Access Key ID: ***
AWS Secret Access Key: *****
No existing clusters currently running!

Cluster Name: tron-east
Total Nodes: 1
Version:
[0] Community
[1] Enterprise
0

Starting an EC2 cluster of type m1.large with image ami-05caca6c...
Ensuring DataStaxLauncher pem key exists on AWS...
Ensuring DataStaxLauncher pem key exists on filesystem...
Ensuring DataStaxLauncher pem key's permissions are acceptable...
Configuring ports...
Launching cluster...

AWS Error: EC2ResponseError: 400 Bad Request
AuthFailure: Not authorized for images: [ami-05caca6c]

When working with the ami provided, I'd like to be able to change the way some versions initialize in our system. I can do this by launching the default AMI, edititng the python, then creating my own ami from this image. However, this source file is not present in the ami.

ds2_configure.pyc

Only the compiled code is. Could you please update this so that I can configure the ami to meet our configuration?

OR

As an alternative solution, we want to be able to configure vnodes as well as the amount of heap used. I can't seem to find any documentation on which configuration options can be passed in the "user_data" value. Can these be set from the client when starting the instance?

Thanks,
Todd

cassandralauncher checks if .ssh/DataStaxLauncher.pem exists and has the correct permissions. If it doesn't, a new one is created. However, if one is copied from another use of cassandralauncher under a different account it continues to launch the cluster with no errors but the nodes are not correctly configured (again with no errors, opsc just hangs waiting for agents).

cassandralauncher should check that the fingerprint for the keypair on the file system matches the fingerprint for the keypair on the EC2 account being used and error out before a cluster launch is attempted (since otherwise it will launch the nodes but they won't be usable)

Starting an EC2 cluster of type m1.large with image ami-6139e708...
Ensuring DataStax pem key exists on AWS...
Ensuring DataStax pem key exists on filesystem...
Ensuring DataStax pem key's permissions are acceptable...
Configuring ports...
Traceback (most recent call last):
  File "/usr/local/bin/cassandralauncher", line 4, in <module>
    cassandralauncher.CassandraLauncher()
  File "/Library/Python/2.7/site-packages/cassandralauncher/__init__.py", line 6, in __init__
    cassandralauncher.run()
  File "/Library/Python/2.7/site-packages/cassandralauncher/cassandralauncher.py", line 701, in run
    main()
  File "/Library/Python/2.7/site-packages/cassandralauncher/cassandralauncher.py", line 630, in main
    user_data, cli_options['CLI_noprompts'])
  File "/Library/Python/2.7/site-packages/cassandralauncher/ec2.py", line 148, in create_cluster
    deauthorize(ds_security_group, 9160)
  File "/Library/Python/2.7/site-packages/cassandralauncher/ec2.py", line 52, in deauthorize
    security_group.revoke('tcp', port, port_end_range, '0.0.0.0/0')
  File "/Library/Python/2.7/site-packages/boto/ec2/securitygroup.py", line 195, in revoke
    src_group_owner_id, cidr_ip)
  File "/Library/Python/2.7/site-packages/boto/ec2/securitygroup.py", line 122, in remove_rule
    if len(rule.grants) == 0:
UnboundLocalError: local variable 'rule' referenced before assignment

This appears to be caused by an error in boto referencing rule outside of the for rule in self.rules loop.
https://github.com/boto/boto/blob/develop/boto/ec2/securitygroup.py#L122

I'm assuming this is a Python change between versions where rule only exists in scope of the for loop and not in the method scope now? (or indentation error?)

https://github.com/joaquincasares/cassandralauncher/blob/master/cassandralauncher/cassandralauncher.py#L677

My company EC2 account does not have a default VPC and look like these script expect a default VPC to be present as I am getting "No default VPC for this user" error while running the script.

Is there any version in progress to allow specifying the VPC in the configuration?

Error Message -

Starting an EC2 cluster of type m1.large with image ami-05caca6c...
Ensuring vsangwan-keypair20150203 pem key exists on AWS...
Ensuring vsangwan-keypair20150203 pem key exists on filesystem...
Ensuring vsangwan-keypair20150203 pem key's permissions are acceptable...
Traceback (most recent call last):
File "/Users/vsangwan/anaconda/bin/cassandralauncher", line 4, in
cassandralauncher.CassandraLauncher()
File "/Users/vsangwan/anaconda/lib/python2.7/site-packages/cassandralauncher/init.py", line 6, in init
cassandralauncher.run()
File "/Users/vsangwan/anaconda/lib/python2.7/site-packages/cassandralauncher/cassandralauncher.py", line 751, in run
main()
File "/Users/vsangwan/anaconda/lib/python2.7/site-packages/cassandralauncher/cassandralauncher.py", line 675, in main
check_cascading_options('security_public_inbound_source', optional=True))
File "/Users/vsangwan/anaconda/lib/python2.7/site-packages/cassandralauncher/ec2.py", line 150, in create_cluster
ds_security_group = conn.create_security_group('DataStax', 'Security group for running DataStax products')
File "/Users/vsangwan/anaconda/lib/python2.7/site-packages/boto/ec2/connection.py", line 3003, in create_security_group
SecurityGroup, verb='POST')
File "/Users/vsangwan/anaconda/lib/python2.7/site-packages/boto/connection.py", line 1207, in get_object
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request

VPCIdNotSpecifiedNo default VPC for this userc725df67-ec48-48e5-9ff5-41f8e83c71f4
vsangwan-mbp15:~ vsangwan$ vi .clusterlauncher.conf

Hi guys,
I'm attempting to launch a cluster on ec2, and it's failing due to the dsc version. I may be configuring it incorrectly. Here are my relevant properties.

clustername = UGCluster
totalnodes = 3
version = Community
installopscenter = False
release = 1.1.12

When I view the ami log, the installer appears to be attempting to install an incorrect version of dsc. Below is the apt-get line from the node.

Fetched 23.0 kB in 1s (15.5 kB/s)
Reading package lists...

[INFO] Performing deployment install...
[EXEC] 10/09/13-19:04:35 sudo apt-get install -y python-cql cassandra=1.1.12 dsc1.1=1.1.12:
Reading package lists...
Building dependency tree...
Reading state information...

[ERROR] 10/09/13-19:04:35 sudo service cassandra stop:
cassandra: unrecognized service

This appears to be due to an incorrect version on dsc. Below is the output of versions from apt-cache madison.

dsc1.1 |   1.1.12-1 | http://debian.datastax.com/community/ stable/main amd64 Packages
    dsc1.1 |   1.1.11-1 | http://debian.datastax.com/community/ stable/main amd64 Packages
    dsc1.1 |   1.1.10-1 | http://debian.datastax.com/community/ stable/main amd64 Packages
    dsc1.1 |    1.1.9-1 | http://debian.datastax.com/community/ stable/main amd64 Packages
    dsc1.1 |    1.1.7-1 | http://debian.datastax.com/community/ stable/main amd64 Packages
    dsc1.1 |    1.1.6-1 | http://debian.datastax.com/community/ stable/main amd

When starting a new instance with current master code, the security group "Datastax" appears to be re-set every time. This means that I lose the security permission of port 9160 from my elastic-beanstalk security group. Can this be changed to one of the following?

1. Each rule is evaluated independently, if it is missing, it is added. This will ensure that all the firewall rules are correct for the default setup, and doesn't remove any existing permissions

2. Disable the firewall rule creation

Hi Joaquin el al.,
I just pip installed cassandralauncher. The script launches the ec2 instances but gets stuck at at "Priming connections..." Would you look into this? If so, please let me know if you need anything.

Thanks!
-Jeff

see below error log. It appears that agent installation was attempted before the AMI finished installing other packages

2012-07-03 01:25:15 +0000 Removing old opscenter-agent files.
2012-07-03 01:25:15 +0000 opscenter-agent: unrecognized service
2012-07-03 01:25:15 +0000 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
2012-07-03 01:25:15 +0000 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
2012-07-03 01:25:15 +0000 Starting agent installation process for version 2.9-1
2012-07-03 01:25:15 +0000 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
2012-07-03 01:25:15 +0000 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
2012-07-03 01:25:15 +0000 FAILURE: The package "sysstat" does not appear to be available or installable

I tried to use this image: ami-373e2b07 at us-west-2 , but I got this error:

Traceback (most recent call last):
File "/usr/local/bin/cassandralauncher", line 4, in
cassandralauncher.CassandraLauncher()
File "/usr/local/lib/python2.7/dist-packages/cassandralauncher/init.py", line 6, in init
cassandralauncher.run()
File "/usr/local/lib/python2.7/dist-packages/cassandralauncher/cassandralauncher.py", line 751, in run
main()
File "/usr/local/lib/python2.7/dist-packages/cassandralauncher/cassandralauncher.py", line 561, in main
ec2.terminate_cluster(check_cascading_options('aws_access_key_id'), check_cascading_options('aws_secret_access_key'), config.get('EC2', 'placement'), check_cascading_options('handle'), prompt_continuation=True)
File "/usr/local/lib/python2.7/dist-packages/cassandralauncher/ec2.py", line 248, in terminate_cluster
reservations = conn.get_all_instances()
AttributeError: 'NoneType' object has no attribute 'get_all_instances'

Any idea?