riptano / comboami Goto Github PK

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html

Write ssd logic and include these commands when ssds are detected:

echo noop > /sys/block/sda/queue/scheduler
          
echo 0 > /sys/block/sda/queue/read_ahead_kb
          
echo 0 > /sys/block/sda/queue/rotational

Hi,

I have an automatic script that installs a bunch of stuff on a new Cassandra AMI, and one of this things is sysdig, but it seems like the SSL certificates are not working:

$ curl https://s3.amazonaws.com/download.draios.com/stable/install-agent
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

The same command works well on any other host.

Thanks.

I launched 3 node cassandra ec2 cluster using ami-6139e708
with user data:
--clustername TokCluster --totalnodes 3 --version community --release 1.0

When launched had the error:
Exception seen in ds1_launcher.py. Please check ~/datastax_ami/ami.log for more info.

This is the content of ami.log

[EXEC] 11/07/12-11:16:00 git pull:
Updating 38b1733..0b637b9
Fast-forward
ds1_launcher.py | 9 ++-
ds2_configure.py | 218 +++++++++++++++++++++++++++++++++++++++-------------
ds4_motd.py | 11 +--
presetup/pre_1.sh | 4 +-
presetup/pre_2.py | 4 +-
tokentoolv2.py | 4 +-
6 files changed, 180 insertions(+), 70 deletions(-)

[EXEC] 11/07/12-11:16:01 sudo rm ds2_configure.py
[INFO] Deleting ds2_configure.py now. This AMI will never change any configs after this first run.
[EXEC] 11/07/12-11:16:01 gpg --keyserver pgp.mit.edu --recv-keys 40976EAF437D05B5:
gpgkeys: key 40976EAF437D05B5 not found on keyserver

[PIPE] 11/07/12-11:16:01 gpg --export --armor 40976EAF437D05B5 | sudo apt-key add -
[EXEC] 11/07/12-11:16:01 sudo rm -rf /etc/motd
[EXEC] 11/07/12-11:16:01 sudo touch /etc/motd
[INFO] Started with user data set to:
[INFO] --clustername TokCluster --totalnodes 3 --version community --release 1.0
[INFO] Using cluster name: TokCluster
[INFO] Using cluster size: 3
[INFO] Using seed indexes: [0, 3, 3]
[PIPE] 11/07/12-11:16:01 echo "deb http://debian.datastax.com/community stable main" | sudo tee -a /etc/apt/sources.list.d/datastax.sources.list:
deb http://debian.datastax.com/community stable main

[PIPE] 11/07/12-11:16:01 echo "deb http://debian.datastax.com/community stable main" | sudo tee -a /etc/apt/sources.list.d/datastax.sources.list:
deb http://debian.datastax.com/community stable main

[PIPE] 11/07/12-11:16:02 curl -s http://installer.datastax.com/downloads/ubuntuarchive.repo_key | sudo apt-key add -:
OK

[PIPE] 11/07/12-11:16:02 curl -s http://installer.datastax.com/downloads/ubuntuarchive.repo_key | sudo apt-key add -:
OK

[PIPE] 11/07/12-11:16:02 curl -s http://opscenter.datastax.com/debian/repo_key | sudo apt-key add -:
OK

[PIPE] 11/07/12-11:16:02 curl -s http://opscenter.datastax.com/debian/repo_key | sudo apt-key add -:
OK

[PIPE] 11/07/12-11:16:02 curl -s http://debian.datastax.com/debian/repo_key | sudo apt-key add -:
OK

[PIPE] 11/07/12-11:16:02 curl -s http://debian.datastax.com/debian/repo_key | sudo apt-key add -:
OK

[EXEC] 11/07/12-11:16:12 sudo apt-get update:
Get:1 http://debian.datastax.com stable Release.gpg [490B]
Get:2 http://us-east-1.ec2.archive.ubuntu.com maverick Release.gpg [198B]
Ign http://debian.datastax.com/community/ stable/main Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/main Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/universe Translation-en
Get:3 http://debian.datastax.com stable Release [2859B]
Get:4 http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release.gpg [198B]
Get:5 http://security.ubuntu.com maverick-security Release.gpg [198B]
Ign http://security.ubuntu.com/ubuntu/ maverick-security/main Translation-en
Ign http://security.ubuntu.com/ubuntu/ maverick-security/universe Translation-en
Get:6 http://archive.canonical.com lucid Release.gpg [198B]
Ign http://archive.canonical.com/ lucid/partner Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/main Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/universe Translation-en
Get:7 http://debian.datastax.com stable/main amd64 Packages [5522B]
Get:8 http://us-east-1.ec2.archive.ubuntu.com maverick Release [39.8kB]
Get:9 http://security.ubuntu.com maverick-security Release [39.8kB]
Get:10 http://archive.canonical.com lucid Release [8215B]
Get:11 http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release [39.8kB]
Get:12 http://us-east-1.ec2.archive.ubuntu.com maverick/main Sources [829kB]
Get:13 http://archive.canonical.com lucid/partner amd64 Packages [13.0kB]
Get:14 http://security.ubuntu.com maverick-security/main Sources [106kB]
Get:15 http://us-east-1.ec2.archive.ubuntu.com maverick/universe Sources [4179kB]
Get:16 http://security.ubuntu.com maverick-security/universe Sources [34.2kB]
Get:17 http://security.ubuntu.com maverick-security/main amd64 Packages [330kB]
Get:18 http://us-east-1.ec2.archive.ubuntu.com maverick/main amd64 Packages [1491kB]
Get:19 http://us-east-1.ec2.archive.ubuntu.com maverick/universe amd64 Packages [5771kB]
Get:20 http://security.ubuntu.com maverick-security/universe amd64 Packages [116kB]
Get:21 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main Sources [179kB]
Get:22 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe Sources [65.2kB]
Get:23 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main amd64 Packages [466kB]
Get:24 http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe amd64 Packages [202kB]
Fetched 13.9MB in 8s (1667kB/s)
Reading package lists...

[EXEC] 11/07/12-11:16:14 sudo apt-get update:
Hit http://us-east-1.ec2.archive.ubuntu.com maverick Release.gpg
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/main Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick/universe Translation-en
Get:1 http://debian.datastax.com stable Release.gpg [490B]
Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release.gpg
Ign http://debian.datastax.com/community/ stable/main Translation-en
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/main Translation-en
Get:2 http://debian.datastax.com stable Release [2859B]
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/universe Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com maverick Release
Hit http://security.ubuntu.com maverick-security Release.gpg
Ign http://security.ubuntu.com/ubuntu/ maverick-security/main Translation-en
Ign http://security.ubuntu.com/ubuntu/ maverick-security/universe Translation-en
Get:3 http://debian.datastax.com stable/main amd64 Packages [5522B]
Hit http://archive.canonical.com lucid Release.gpg
Ign http://archive.canonical.com/ lucid/partner Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates Release
Hit http://us-east-1.ec2.archive.ubuntu.com maverick/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com maverick/universe Sources
Hit http://security.ubuntu.com maverick-security Release
Hit http://archive.canonical.com lucid Release
Hit http://us-east-1.ec2.archive.ubuntu.com maverick/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com maverick/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main Sources
Hit http://security.ubuntu.com maverick-security/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com maverick-updates/universe amd64 Packages
Hit http://archive.canonical.com lucid/partner amd64 Packages
Hit http://security.ubuntu.com maverick-security/universe Sources
Hit http://security.ubuntu.com maverick-security/main amd64 Packages
Hit http://security.ubuntu.com maverick-security/universe amd64 Packages
Fetched 8871B in 0s (22.2kB/s)
Reading package lists...

[INFO] Performing deployment install...
[EXEC] 11/07/12-11:16:20 sudo apt-get install -y cassandra=1.0 dsc:
Reading package lists...
Building dependency tree...
Reading state information...

[ERROR] 11/07/12-11:16:21 sudo service cassandra stop:
cassandra: unrecognized service

[EXEC] 11/07/12-11:16:21 sudo rm -rf /var/lib/cassandra
[EXEC] 11/07/12-11:16:21 sudo rm -rf /var/log/cassandra
[EXEC] 11/07/12-11:16:21 sudo mkdir -p /var/lib/cassandra
[EXEC] 11/07/12-11:16:21 sudo mkdir -p /var/log/cassandra
[ERROR] 11/07/12-11:16:21 sudo chown -R cassandra:cassandra /var/lib/cassandra:
chown: invalid user: `cassandra:cassandra'

[ERROR] 11/07/12-11:16:21 sudo chown -R cassandra:cassandra /var/log/cassandra:
chown: invalid user: `cassandra:cassandra'

[INFO] Installing OpsCenter...
[EXEC] 11/07/12-11:16:41 sudo apt-get -y install opscenter-free:
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
opscenter-free
0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 47.7MB of archives.
After this operation, 70.5MB of additional disk space will be used.
Get:1 http://debian.datastax.com/community/ stable/main opscenter-free all 2.1.2-1 [47.7MB]
Fetched 47.7MB in 9s (5117kB/s)
Selecting previously deselected package opscenter-free.
(Reading database ... 36947 files and directories currently installed.)
Unpacking opscenter-free (from .../opscenter-free_2.1.2-1_all.deb) ...
Processing triggers for ureadahead ...
Setting up opscenter-free (2.1.2-1) ...
Generating a 1024 bit RSA private key
........++++++
.................++++++

writing new private key to '/var/lib/opscenter/ssl/opscenter.key'

MAC verified OK
Certificate was added to keystore

[EXEC] 11/07/12-11:16:41 sudo service opscenterd stop
[INFO] Reflector loop...
[INFO] Seed list: set([u'10.83.46.56'])
[INFO] OpsCenter: 10.83.46.56
[INFO] Options: {'username': None, 'cfsreplication': None, 'reflector': None, 'clustername': 'TokCluster', 'analyticsnodes': 0, 'seed_indexes': [0, 3, 3], 'realtimenodes': 3, 'opscenter': None, 'totalnodes': 3, 'searchnodes': 0, 'opscenterinterface': None, 'version': 'community', 'dev': None, 'release': '1.0', 'password': None, 'email': None, 'raidonly': None, 'heapsize': None}
[ERROR] Exception seen in ds1_launcher.py:
Traceback (most recent call last):
File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 20, in initial_configurations
ds2_configure.run()
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 799, in run
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 389, in construct_yaml
IOError: [Errno 2] No such file or directory: '/etc/cassandra/cassandra.yaml'

Hello,

I feel like the choice between using this AMI as-is or forking a new AMI is too stark. For example, the way the RAID is built is such that it's always RAID0 on all available storage devices. I would like to create a RAID only on EBS storage, which should be a few lines to change in the Python scripts.

I wonder if the following would be possible:

• Allow specifying keys rather than relying on @joaquincasares's hard-coded one
• Allow specifying git repository to pull from rather than relying on the one in the AMI (currently "2.5")

For example, user data could be --keys KEY1,KEY2,KEY3 --repo GITREPO and then anyone would be able to modify behavior.

This would mostly involve simple changes to ds0_updater.py that I'm willing to help with if there's a chance they'll be merged.

Thoughts? Comments?

I have a VPC network which uses 3 AZ's .The DataStax AMI doesn't span across availability zones. Its using SimpleSnitch . Any plans to add EC2Snitch and EC2MultiRegionSnitch

http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/architecture/architectureSnitchesAbout_c.html#concept_ds_mj2_3qf_fk

Seems like new cassandra release breaking the cluster formation.

I want to set up a complete new cluster with release 2.0.8, but it failed.
For me the problem seems to be the release 2.0.8. Tests with --release 2.0.9 worked fine.

For testing I use this configuration.
--clustername test --totalnodes 1 --version community --release 2.0.8 --opscenter no

Console:
Exception seen in ds1_launcher.py. Please check ~/datastax_ami/ami.log for more info.

Please visit http://datastax.com/ami for this AMI's feature set.

ami.log:
[EXEC] 07/24/14-15:26:11 git pull:
Updating 5f722d6..e38ae0d
Fast-forward
README.md | 9 +-
SWITCHES.md | 7 +-
ami_ids.json | 22 +++++
community_cloud/README.md | 5 +
community_cloud/cloudformation/README.md | 35 +++++++
.../cloudformation/cassandra-vpc.template | 103 ++++++++++++++++++++
ds1_launcher.py | 2 +
ds2_configure.py | 103 ++++++++++++++------
ds4_motd.py | 7 +-
logger.py | 2 +-
presetup/setup.md | 4 +-
11 files changed, 261 insertions(+), 38 deletions(-)
create mode 100644 ami_ids.json
create mode 100644 community_cloud/README.md
create mode 100644 community_cloud/cloudformation/README.md
create mode 100644 community_cloud/cloudformation/cassandra-vpc.template

[EXEC] 07/24/14-15:26:11 git reset --hard:
HEAD is now at e38ae0d In some VPC cases meta-data/public-hostname returns blanks. Account for these cases.

[EXEC:E] 07/24/14-15:26:11 gpg --import /home/ubuntu/datastax_ami/repo_keys/DataStax_AMI.7123CDFD.key:
gpg: directory /root/.gnupg' created gpg: new configuration file/root/.gnupg/gpg.conf' created
gpg: WARNING: options in /root/.gnupg/gpg.conf' are not yet active during this run gpg: keyring/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 7123CDFD: public key "Joaquin Casares (DataStax AMI) [email protected]" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)

[EXEC] 07/24/14-15:26:11 git log --pretty="format:%G?" --show-signature HEAD^..HEAD:
gpg: Signature made Fri Jul 18 22:50:44 2014 UTC using RSA key ID 7123CDFD
gpg: Good signature from "Joaquin Casares (DataStax AMI) [email protected]"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6273 EED3 28B4 B09B 2087 1C0E E711 FCAF 7123 CDFD
G
[EXEC] 07/24/14-15:26:11 sudo rm ds2_configure.py
[INFO] Deleting ds2_configure.py now. This AMI will never change any configs after this first run.
[EXEC] 07/24/14-15:26:11 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/Launchpad_VLC.C2518248EEA14886.key:
OK

[EXEC] 07/24/14-15:26:11 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/Ubuntu_Archive.40976EAF437D05B5.key:
OK

[EXEC] 07/24/14-15:26:11 sudo rm -rf /etc/motd
[EXEC] 07/24/14-15:26:11 sudo touch /etc/motd
[INFO] Started with user data set to:
[INFO] --clustername test --totalnodes 1 --version community --release 2.0.8 --opscenter no
[INFO] Using instance type: m3.large
[INFO] meta-data:instance-type: m3.large
[INFO] meta-data:local-ipv4: 10.107.24.50
[INFO] meta-data:public-hostname: ec2-54-74-10-193.eu-west-1.compute.amazonaws.com
[INFO] meta-data:ami-launch-index: 0
[INFO] meta-data:reservation-id: r-ecae08ae
[EXEC] 07/24/14-15:26:11 sudo ls:
ami.conf
ami_ids.json
AMI_LICENSE.txt
ami.log
CHANGES.md
community_cloud
conf.py
conf.pyc
ds0_updater.py
ds0_utils.py
ds0_utils.pyc
ds1_launcher.py
ds1_launcher.pyc
ds2_configure.pyc
ds3_after_init.py
ds4_motd.py
EXPANDING.md
FILES.md
logger.py
logger.pyc
presetup
README.md
reflector.php
repo_keys
SWITCHES.md
tokentool.html
tokentoolv2.py

[INFO] Using cluster name: test
[INFO] Using cluster size: 1
[INFO] Using seed indexes: [0, 1, 1]
[PIPE] 07/24/14-15:26:11 echo "deb http://debian.datastax.com/community stable main" | sudo tee /etc/apt/sources.list.d/datastax.sources.list:
deb http://debian.datastax.com/community stable main

[PIPE] 07/24/14-15:26:11 echo "deb http://debian.datastax.com/community stable main" | sudo tee /etc/apt/sources.list.d/datastax.sources.list:
deb http://debian.datastax.com/community stable main

[EXEC] 07/24/14-15:26:11 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/DataStax.key:
OK

[EXEC] 07/24/14-15:26:26 sudo apt-get update:
Get:1 http://eu-west-1.ec2.archive.ubuntu.com precise Release.gpg [198 B]
Get:2 http://eu-west-1.ec2.archive.ubuntu.com precise-updates Release.gpg [198 B]
Get:3 http://ppa.launchpad.net precise Release.gpg [316 B]
Hit http://eu-west-1.ec2.archive.ubuntu.com precise Release
Get:4 http://ppa.launchpad.net precise Release [11.9 kB]
Get:5 http://eu-west-1.ec2.archive.ubuntu.com precise-updates Release [49.6 kB]
Get:6 http://ppa.launchpad.net precise/main Sources [1,253 B]
Get:7 http://eu-west-1.ec2.archive.ubuntu.com precise/main Sources [934 kB]
Get:8 http://ppa.launchpad.net precise/main amd64 Packages [2,826 B]
Get:9 http://ppa.launchpad.net precise/main i386 Packages [2,826 B]
Ign http://ppa.launchpad.net precise/main TranslationIndex
Get:10 http://eu-west-1.ec2.archive.ubuntu.com precise/universe Sources [5,019 kB]
Get:11 http://security.ubuntu.com precise-security Release.gpg [198 B]
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Get:12 http://debian.datastax.com stable Release.gpg [280 B]
Get:13 http://security.ubuntu.com precise-security Release [49.6 kB]
Get:14 http://eu-west-1.ec2.archive.ubuntu.com precise/main amd64 Packages [1,273 kB]
Get:15 http://debian.datastax.com stable Release [2,870 B]
Get:16 http://eu-west-1.ec2.archive.ubuntu.com precise/universe amd64 Packages [4,786 kB]
Get:17 http://debian.datastax.com stable/main amd64 Packages [14.3 kB]
Get:18 http://security.ubuntu.com precise-security/main Sources [107 kB]
Get:19 http://debian.datastax.com stable/main i386 Packages [14.3 kB]
Get:20 http://security.ubuntu.com precise-security/universe Sources [30.7 kB]
Ign http://debian.datastax.com stable/main TranslationIndex
Get:21 http://eu-west-1.ec2.archive.ubuntu.com precise/main i386 Packages [1,274 kB]
Get:22 http://security.ubuntu.com precise-security/main amd64 Packages [414 kB]
Get:23 http://eu-west-1.ec2.archive.ubuntu.com precise/universe i386 Packages [4,796 kB]
Get:24 http://eu-west-1.ec2.archive.ubuntu.com precise/main TranslationIndex [3,706 B]
Get:25 http://eu-west-1.ec2.archive.ubuntu.com precise/universe TranslationIndex [2,922 B]
Get:26 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main Sources [475 kB]
Get:27 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe Sources [108 kB]
Get:28 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main amd64 Packages [821 kB]
Get:29 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe amd64 Packages [244 kB]
Get:30 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main i386 Packages [853 kB]
Get:31 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe i386 Packages [251 kB]
Get:32 http://security.ubuntu.com precise-security/universe amd64 Packages [94.3 kB]
Ign http://debian.datastax.com stable/main Translation-en_US
Get:33 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main TranslationIndex [3,564 B]
Get:34 http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe TranslationIndex [2,850 B]
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/main Translation-en
Get:35 http://security.ubuntu.com precise-security/main i386 Packages [442 kB]
Ign http://debian.datastax.com stable/main Translation-en
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/universe Translation-en
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main Translation-en
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe Translation-en
Get:36 http://security.ubuntu.com precise-security/universe i386 Packages [99.7 kB]
Get:37 http://security.ubuntu.com precise-security/main TranslationIndex [74 B]
Get:38 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B]
Get:39 http://security.ubuntu.com precise-security/main Translation-en [189 kB]
Get:40 http://security.ubuntu.com precise-security/universe Translation-en [57.8 kB]
Fetched 22.4 MB in 8s (2,624 kB/s)
Reading package lists...

[EXEC] 07/24/14-15:26:31 sudo apt-get update:
Hit http://eu-west-1.ec2.archive.ubuntu.com precise Release.gpg
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates Release.gpg
Hit http://eu-west-1.ec2.archive.ubuntu.com precise Release
Hit http://ppa.launchpad.net precise Release.gpg
Hit http://security.ubuntu.com precise-security Release.gpg
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates Release
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/main Sources
Hit http://ppa.launchpad.net precise Release
Hit http://security.ubuntu.com precise-security Release
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/universe Sources
Hit http://ppa.launchpad.net precise/main Sources
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/main amd64 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/universe amd64 Packages
Hit http://security.ubuntu.com precise-security/main Sources
Hit http://ppa.launchpad.net precise/main amd64 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/main i386 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/universe i386 Packages
Hit http://security.ubuntu.com precise-security/universe Sources
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/main TranslationIndex
Hit http://ppa.launchpad.net precise/main i386 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/universe TranslationIndex
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main Sources
Hit http://security.ubuntu.com precise-security/main amd64 Packages
Ign http://ppa.launchpad.net precise/main TranslationIndex
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe Sources
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main amd64 Packages
Hit http://security.ubuntu.com precise-security/universe amd64 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe amd64 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main i386 Packages
Hit http://security.ubuntu.com precise-security/main i386 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe i386 Packages
Hit http://security.ubuntu.com precise-security/universe i386 Packages
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main TranslationIndex
Hit http://security.ubuntu.com precise-security/main TranslationIndex
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe TranslationIndex
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/main Translation-en
Hit http://eu-west-1.ec2.archive.ubuntu.com precise/universe Translation-en
Hit http://security.ubuntu.com precise-security/universe TranslationIndex
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/main Translation-en
Hit http://eu-west-1.ec2.archive.ubuntu.com precise-updates/universe Translation-en
Hit http://security.ubuntu.com precise-security/main Translation-en
Hit http://security.ubuntu.com precise-security/universe Translation-en
Ign http://ppa.launchpad.net precise/main Translation-en_US
Ign http://ppa.launchpad.net precise/main Translation-en
Get:1 http://debian.datastax.com stable Release.gpg [280 B]
Get:2 http://debian.datastax.com stable Release [2,870 B]
Get:3 http://debian.datastax.com stable/main amd64 Packages [14.3 kB]
Get:4 http://debian.datastax.com stable/main i386 Packages [14.3 kB]
Ign http://debian.datastax.com stable/main TranslationIndex
Ign http://debian.datastax.com stable/main Translation-en_US
Ign http://debian.datastax.com stable/main Translation-en
Fetched 31.8 kB in 2s (15.9 kB/s)
Reading package lists...

[INFO] Performing deployment install...
[EXEC] 07/24/14-15:26:36 sudo mv /etc/security/limits.d/cassandra.conf /etc/security/limits.d/cassandra.conf.bak
[EXEC] 07/24/14-15:26:36 sudo apt-get install -y python-cql datastax-agent cassandra=2.0.8 dsc20=2.0.8-1:
Reading package lists...
Building dependency tree...
Reading state information...

[ERROR] 07/24/14-15:26:36 sudo service cassandra stop:
cassandra: unrecognized service

[EXEC] 07/24/14-15:26:36 sudo rm -rf /var/lib/cassandra
[EXEC] 07/24/14-15:26:36 sudo rm -rf /var/log/cassandra
[EXEC] 07/24/14-15:26:36 sudo mkdir -p /var/lib/cassandra
[EXEC] 07/24/14-15:26:36 sudo mkdir -p /var/log/cassandra
[ERROR] 07/24/14-15:26:36 sudo chown -R cassandra:cassandra /var/lib/cassandra:
chown: invalid user: `cassandra:cassandra'

[ERROR] 07/24/14-15:26:36 sudo chown -R cassandra:cassandra /var/log/cassandra:
chown: invalid user: `cassandra:cassandra'

[EXEC] 07/24/14-15:26:36 sudo mv /etc/security/limits.d/cassandra.conf.bak /etc/security/limits.d/cassandra.conf
[INFO] Reflector loop...
[INFO] 07/24/14-15:26:37 Reflector: Received 1 of 1 responses from: [u'10.107.24.50']
[INFO] Seed list: set([u'10.107.24.50'])
[INFO] OpsCenter: 10.107.24.50
[INFO] Options: Namespace(analyticsnodes=0, base64postscript=None, cfsreplication=None, clustername='test', customreservation=None, email=None, hadoop=False, heapsize=None, opscenter='no', opscenterinterface=None, opscenteronly=False, opscenterssl=False, password=None, raidonly=False, realtimenodes=1, reflector=None, release='2.0.8', rpcbinding=False, searchnodes=0, seed_indexes=[0, 1, 1], totalnodes=1, username=None, version='community')
[ERROR] Exception seen in ds1_launcher.py:
Traceback (most recent call last):
File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 22, in initial_configurations
ds2_configure.run()
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 1051, in run
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 513, in construct_yaml
IOError: [Errno 2] No such file or directory: '/etc/cassandra/cassandra.yaml'

I set up an OpenSwan firewall/VPC between 2 regions.

Now I am ready to launch a datastax community cassandra AMI.

However, "sudo apt-get update" seems to go into a loop for a long time:

https://github.com/riptano/ComboAMI/blob/2.5/ds2_configure.py
also break on this "unable to resolve host" case
Starting line 315:

logger.exe('sudo apt-get update')
while True:
    output = logger.exe('sudo apt-get update')
    if not output[1] and not 'err' in output[0].lower() and not 'failed' in output[0].lower():
        break

Workaround:

-- put 127.0.1.1 ip-my-private-ip in /etc/hosts, so that the while True logger.exe("sudo apt-get update") output[1] check passes.

If not, the loop won't break on this "unable to resolve host" case.

When provisioning this ami via OpsCenter, which specifies raidonly=true, the following stack trace shows up in ami.log:

[ERROR] Exception seen in ds0_updater.py:
Traceback (most recent call last):
  File "ds0_updater.py", line 14, in <module>
    ds1_launcher.run()
  File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 156, in run
    wait_for_seed()
  File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 122, in wait_for_seed
    nodetoolStatement = "nodetool -h " + conf.get_config("AMI", "LeadingSeed") + " ring"
TypeError: cannot concatenate 'str' and 'bool' objects

There don't seem to be any actual problems related to it, but it'd be nice to fix to avoid red herrings

I looked at the deb repository, and it still have dsc version 1.0.10 which runs Cassandra 1.0.10. I want to install it to test upgrade between minor versions of 1.0.x.

BUT, when I try to create AMI that runs 1.0.10 using the following user data, it fails with errors:

--clustername TokCluster --totalnodes 1 --version community --release 1.0.10 --opscenter no

I traced the reason to the code, in ds2_configure.py, it will call

logger.exe('sudo apt-get install -y cassandra={0} dsc'.format(options.release))

so it will try to install cassandra=1.0.10, but for dsc it will take the latest which is 1.0.11-1 and it depends on cassandra 1.0.11.

In order to fix it, you need to do something similar to what you do for 1.1

elif options.release and options.release.startswith('1.1'):
            dse_release = cassandra_release = options.release
            if dse_release == '1.1.6':
                dse_release = '1.1.6-1'
            logger.exe('sudo apt-get install -y cassandra={0} dsc1.1={1}'.format(cassandra_release, dse_release))

Can you please update the AMI so it will be possible to create older versions of cassandra 1.0.x?

Thanks,
Tamar

Hi I just faced an issue trying to use the following image to create a I2.2xLarge Cassandra 1.2.8 node:

DataStax Auto-Clustering AMI 2.5.1-hvm - ami-ada2b6c4 with User Data : "--clustername Expand --totalnodes 14 --version community --opscenter no --release 1.2.18"

In the error log I can see this

Traceback (most recent call last):
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 501, in get_seed_list
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

and this from ~/datastax_ami/ami.log

INFO] Reflector loop...
[INFO] 11/17/14-23:12:26 Reflector: Received 0 of 1 responses from: []
[ERROR] EC2 is experiencing some issues and has not allocated all of the resources in under 10 minutes.

Aborting the clustering of this reservation. Please try again.
[ERROR] Exception seen in ds1_launcher.py:
Traceback (most recent call last):
  File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 22, in initial_configurations
    ds2_configure.run()
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 1106, in run
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 483, in get_seed_list
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 57, in exit_path
AttributeError: EC2 is experiencing some issues and has not allocated all of the resources in under 10 minutes.

Aborting the clustering of this reservation. Please try again.

FWIW, There is 2 things new for me there:

• It is my first deployment in us-east-1
• It was my first deployment inside a fully private VPC (no public IP/DNS).

I feel like this has something to see with the second point since "https://github.com/riptano/ComboAMI/blob/2.5/ds2_configure.py" is pointed on line 501 and the request seems to try getting "instance_data['publichostname']" at some point, which, I believe, does not exist.

Any clue @joaquincasares ?

The dependency-resolver for the apt tools don't make it easy to install old versions of metapackages (as documented at http://docs.datastax.com/en/datastax_enterprise/4.6/datastax_enterprise/install/installPrevious.html).

sudo apt-get install -y dse-full=4.6.1-1 fails if 4.6.1-1 isn't the latest version available in the repository. apt-get tries to install the latest version of deps, even though dse-full requires it's deps to version-match itself exactly.

The workaround is to specify all the deps and versions in a single apt-get install call, like so:

sudo apt-get install -y dse-full=4.6.1-1 dse=4.6.1-1 dse-demos=4.6.1-1 dse-hive=4.6.1-1 dse-libcassandra=4.6.1-1 dse-libhadoop=4.6.1-1 dse-libhive=4.6.1-1 dse-libpig=4.6.1-1 dse-pig=4.6.1-1 dse-liblog4j=4.6.1-1 dse-libsolr=4.6.1-1 dse-libsqoop=4.6.1-1 dse-libtomcat=4.6.1-1 dse-libmahout=4.6.1-1 dse-libhadoop-native=4.6.1-1

We currently hardcode the package list in the ami code:

It would be better to run apt-cache depends dse-full=version or apt-cache show dse-full=version and parse the dependencies out of that output rather than hard-code it and have to update every version and add custom logic if the package-list changes.

Seems like the upgrade is broken on the instance, and it doesn't happen on a vanilla 12.04:

$ sudo apt-get dist-upgrade
Preparing to replace apt 0.8.16~exp12ubuntu10.16 (using .../apt_0.8.16~exp12ubuntu10.17_amd64.deb) ...
Unpacking replacement apt ...
Processing triggers for man-db ...
Setting up apt (0.8.16~exp12ubuntu10.17) ...
ERROR: Can't find the archive-keyring
Is the ubuntu-keyring package installed?
dpkg: error processing apt (--configure):
 subprocess installed post-installation script returned error exit status 1
No apport report written because MaxReports is reached already
                                                              Errors were encountered while processing:
 apt
E: Sub-process /usr/bin/dpkg returned an error code (1)

The solution I found is:

sudo apt-get --reinstall install ubuntu-keyring

But that shouldn't happen by default

ComboAMI checks that commits are signed by 7123CDFD, which is labelled as belonging to Joaquin individually. Add a second valid key with an ownership label that indicates it's the ComboAMI signing key (I might even avoid Datastax affiliation in the name, to possibly enable a potential future community-maintainer to sign commits).

Code context:

This will require a rebake.

Seen when only the --base64 parameter is used.

[ERROR] Exception seen in ds1_launcher.py:
Traceback (most recent call last):
  File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 22, in initial_configurations
    ds2_configure.run()
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 997, in run
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 221, in parse_ec2_userdata
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

Looking at

This reduces the ports required by configurations that restrict outbound connections.

Since LOCAL_JMX is yes, other nodes in the cluster failed at running "nodetool -h ", then can not launch cassandra service.

These messages are seen at the end of the motd when clusters are launched with settings seen in #51 .

[ERROR] 07/18/14-23:23:52 sudo rm ds2_configure.py:
[ERROR] 07/18/14-23:23:52 sudo rm -rf /etc/motd:
[ERROR] 07/18/14-23:23:52 sudo touch /etc/motd:

The #51 fix should either happen before the time that it does, or we should filter out the error messages later. The former fix being preferred.

Does this work with 1.0 release yet of C?

After provisioning via opscenter (raidonly=true)

[automaton@ip-10-196-24-69 ~]$ ssh -i US_West_\(Northern_California\)-OpsCenterProvisioningKeyPair.pem [email protected]
Last login: Wed Feb 27 01:10:10 2013 from ip-10-196-24-69.us-west-1.compute.internal

Cluster started with these options:
--raidonly


Waiting for nodetool...
The cluster is now in it's finalization phase. This should only take a moment...

Note: You can also use CTRL+C to view the logs if desired:
    AMI log: ~/datastax_ami/ami.log
    Cassandra log: /var/log/cassandra/system.log


Traceback (most recent call last):
  File "datastax_ami/ds4_motd.py", line 196, in <module>
    run()
  File "datastax_ami/ds4_motd.py", line 187, in run
    waiting_for_nodetool()
  File "datastax_ami/ds4_motd.py", line 84, in waiting_for_nodetool
    retcode = subprocess.call(shlex.split(config_data['nodetool_statement']), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

The guidelines for dealing with insufficient user resource limits errors published here
suggest that

For all installations, add the following line to /etc/sysctl.conf:
vm.max_map_count = 131072

Maybe we should make this the default in the AMI?

Hi,

Not really an issue but more of a custom workflow problem.

So I use and love the Datastax AMI, and I launch it using a set of cloudformation templates, along with the other instances in my tiers.
I use ansible in pull mode for configuring my instances, so what I do is embedding a two-line script in the userdata section, that will just install ansible-pull on my instance.
Ansible will then take care of the rest of the configuration (for cassandra, it will do some minor things like setting the right seed node, increasing the JVM size, registering the IP address in Route 53).

Of course with the Datastax AMI I can't do this because the userdata is reserved to the various switches.

I understand this is not a big problem as I can find another way to do it (e.g. launching ansible from another machine once the deploy is done, or creating my own AMI), but I was looking if there's some "hidden" way (e.g. using a switch like "--base64-script" to be executed after cassandra gets installed) to do that with your AMI.

Thanks for the good work.

Waiting for nodetool...
The cluster is now in it's finalization phase. This should only take a moment...

Will stay on the motd for a while if the service is not running. Simplify that screen to allow for those running the cluster not to see that.

This makes cassandra unable to create /raid0/cassandra/logs for instance since cassandra is ran as "cassandra" user.

"Unable to redirect to /var/log/cassandra/output.log
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/log/cassandra/system.log (No such file or directory)"

I've been fighting a bug all day and I think it turns out you just pushed DSE4.0 to your debian repo since now it's trying to install DSE4.0 on the 2.4 branch but it also installs java 6 so there's a version mismatch there and everything fails to start.

should the apt-get install command be locked down to a target version for the branches?
2.4 in ds2_configure.py would have:
sudo apt-get install -y dse-full=3.2.5-1

https://github.com/riptano/ComboAMI/blob/2.4/ds2_configure.py#L323 needs to be updated for DSE 3

seeing these in the ./pre_1.sh script run apparently when hitting the maven install line in pre_2.py

[EXEC] sudo -u ubuntu mvn install

[EXEC] sudo -u ubuntu mvn install
stdout:
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Default Project
[INFO] task-segment: [install]
[INFO] ------------------------------------------------------------------------
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.3/maven-resources-plugin-2.3.pom
4K downloaded (maven-resources-plugin-2.3.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-plugins/12/maven-plugins-12.pom
11K downloaded (maven-plugins-12.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-parent/9/maven-parent-9.pom
32K downloaded (maven-parent-9.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/apache/4/apache-4.pom
4K downloaded (apache-4.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.3/maven-resources-plugin-2.3.jar
20K downloaded (maven-resources-plugin-2.3.jar)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/2.0.2/maven-compiler-plugin-2.0.2.pom
2K downloaded (maven-compiler-plugin-2.0.2.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-plugins/8/maven-plugins-8.pom
5K downloaded (maven-plugins-8.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-parent/5/maven-parent-5.pom
14K downloaded (maven-parent-5.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/apache/3/apache-3.pom
3K downloaded (apache-3.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/2.0.2/maven-compiler-plugin-2.0.2.jar
17K downloaded (maven-compiler-plugin-2.0.2.jar)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.10/maven-surefire-plugin-2.10.pom
10K downloaded (maven-surefire-plugin-2.10.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire/2.10/surefire-2.10.pom
11K downloaded (surefire-2.10.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-parent/20/maven-parent-20.pom
24K downloaded (maven-parent-20.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/apache/9/apache-9.pom
14K downloaded (apache-9.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.10/maven-surefire-plugin-2.10.jar
29K downloaded (maven-surefire-plugin-2.10.jar)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2/maven-jar-plugin-2.2.pom
8K downloaded (maven-jar-plugin-2.2.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-plugins/10/maven-plugins-10.pom
7K downloaded (maven-plugins-10.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-parent/7/maven-parent-7.pom
20K downloaded (maven-parent-7.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2/maven-jar-plugin-2.2.jar
26K downloaded (maven-jar-plugin-2.2.jar)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.3/maven-install-plugin-2.3.pom
4K downloaded (maven-install-plugin-2.3.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-plugins/13/maven-plugins-13.pom
11K downloaded (maven-plugins-13.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-parent/11/maven-parent-11.pom
31K downloaded (maven-parent-11.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/apache/5/apache-5.pom
4K downloaded (apache-5.pom)
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.3/maven-install-plugin-2.3.jar
22K downloaded (maven-install-plugin-2.3.jar)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Cannot execute mojo: resources. It requires a project with an existing pom.xml, but the build is not using one.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 second
[INFO] Finished at: Wed Feb 26 22:49:49 UTC 2014
[INFO] Final Memory: 5M/111M
[INFO] ------------------------------------------------------------------------

[EXEC] git clone -b cassandra-1.2 https://github.com/apache/cassandra.git

currently the commit and data logs are on the same mount point in the prepare_for_raid method of ds2_configure.py.

Given the recommendations are to store those on separate drives, would it make sense if I work on a pull request to allow a new user data option to run the commit log on one of the disks available if > 1 available?

/raid0
/raid0_commitlog

or do you have other suggestions there?

thanks,
Jim

The new i2 instances types are perfect for Cassandra workloads. They require HVM instances, though. It'd be awesome to have an HVM AMI as well.

I am trying to deploy into a VPC with outbound connection limitations. 80 and 443 are open, but I am still running into problems.

This is an install stripped down to a single node to simplify debugging. The following is coming in the ami.log:

~/datastax_ami$ cat ami.log
[ERROR] 01/17/14-23:52:41 git pull:
fatal: unable to connect to github.com:
github.com[0: 192.30.252.130]: errno=Connection timed out

[EXEC] 01/17/14-23:52:42 sudo rm ds2_configure.py
[INFO] Deleting ds2_configure.py now. This AMI will never change any configs after this first run.
[EXEC] 01/17/14-23:53:45 gpg --keyserver pgp.mit.edu --recv-keys 40976EAF437D05B5:
?: pgp.mit.edu: Connection timed out
gpgkeys: HTTP fetch error 7: couldn't connect: Connection timed out

[PIPE] 01/17/14-23:53:45 gpg --export --armor 40976EAF437D05B5 | sudo apt-key add -
[EXEC] 01/17/14-23:53:45 sudo rm -rf /etc/motd
[EXEC] 01/17/14-23:53:45 sudo touch /etc/motd
[INFO] Started with user data set to:
[INFO] --clustername cassandra-test-vpc --totalnodes 1 --version Community --release 2.0.3
[INFO] Using instance type: m1.xlarge
[ERROR] Exception seen in ds1_launcher.py:
Traceback (most recent call last):
File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 20, in initial_configurations
ds2_configure.run()
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 786, in run
File "/home/ubuntu/datastax_ami/ds2_configure.py", line 97, in get_ec2_data
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found

It is possible that I need more allowed egress routes/ports, but I haven't found those requirements anywhere yet...

I am able to connect to the reflector and pull w/o problems using curl from the command line on the instance.

This possibility was removed before, due to security reasons, but perhaps a hidden option to make this happen would be nice for advanced users.

Getting an error during mvm install. Do not seem to get a working image with it.
How do I resolve? Thanks.

[EXEC] sudo -u ubuntu mvn install
stdout:
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Default Project
[INFO] task-segment: [install]
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Cannot execute mojo: resources. It requires a project with an existing pom.xml, but the build is not using one.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 second
[INFO] Finished at: Tue Nov 04 13:03:23 UTC 2014
[INFO] Final Memory: 5M/143M
[INFO] ------------------------------------------------------------------------

Creation of initial tokens throw intermittent ConfigurationException resulting in node failure -- seemingly in creation of large clusters. Reference trace below:

INFO [main] 2013-07-23 16:22:05,012 CLibrary.java (line 125) JNA mlockall successful
INFO [main] 2013-07-23 16:22:05,037 DatabaseDescriptor.java (line 137) Loading settings from file:/etc/dse/cassandra/cassandra.yaml
INFO [main] 2013-07-23 16:22:05,570 DatabaseDescriptor.java (line 155) Data files directories: [/raid0/cassandra/data]
INFO [main] 2013-07-23 16:22:05,570 DatabaseDescriptor.java (line 156) Commit log directory: /raid0/cassandra/commitlog
INFO [main] 2013-07-23 16:22:05,570 DatabaseDescriptor.java (line 196) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2013-07-23 16:22:05,573 DatabaseDescriptor.java (line 210) disk_failure_policy is stop
INFO [main] 2013-07-23 16:22:05,574 DatabaseDescriptor.java (line 222) Legacy authentication config found. Existing authentication data will be migrated to a system keyspace.
INFO [main] 2013-07-23 16:22:05,591 DatabaseDescriptor.java (line 313) Global memtable threshold is enabled at 614MB
INFO [main] 2013-07-23 16:22:05,600 DseDelegateSnitch.java (line 47) Detected Hadoop trackers are enabled, setting my role to Analytics
INFO [main] 2013-07-23 16:22:05,604 DseConfig.java (line 88) Loading settings from file:/etc/dse/dse.yaml
INFO [main] 2013-07-23 16:22:05,621 DseSimpleSnitch.java (line 38) Detected Hadoop trackers are enabled, setting my DC to Analytics
INFO [main] 2013-07-23 16:22:05,623 DseConfig.java (line 229) Load of settings is done.
INFO [main] 2013-07-23 16:22:05,833 DatabaseDescriptor.java (line 441) Not using multi-threaded compaction
ERROR [main] 2013-07-23 16:22:05,835 DatabaseDescriptor.java (line 540) Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: For input string: "9223372036854785804"
at org.apache.cassandra.dht.Murmur3Partitioner$1.validate(Murmur3Partitioner.java:182)
at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:480)
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:128)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:211)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:164)
at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:375)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)

Launching instances in a VPC (at least as our security guys set it up) requires that you specify the subnet that your nodes will use. The subnets don't span availability zones.

To bring a cluster up from the ec2 manger in multiple availability zones in the VPC you need to use the customreservation as you will be running multiple instance launch commands.

I just received an email from Datastax today:

"DATASTAX SUPPORT ALERT - May 5, 2015

• Please ensure you are prepared for the Leap Second on June 30, 2015 *
  [...]
  Required Actions:
  Ensure that you are running Cassandra and its drivers on kernel version 3.4 or higher. We also recommend using JDK version 7u60 or higher. This should protect you from the livelock problems users experienced in 2012."

On an instance created from the AMI "datastax_clustering_ami_2.5.1_hvm.manifest.xml (ami-7f33cd08)" which I believe depend on this code, I have this:

"root@ip-10-0-2-56:# uname -r
3.8.0-37-generic
root@ip-10-0-2-56:# java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)"

So the kernel is quite old but new enough to handle the leap second (and I already opened an issue about it #58).

Yet the JDK version seems to be bellow the one advised by Datastax. I guess we should bump it here directly. I am not sure what version we should use, so I son't propose a PR.

If you don't do it in here I will bump it in my deploy process, but I wanted to let you know.

C*heers,

Alain

Hi,

I've reported an issue with raid on the cassandra mailling list :

http://grokbase.com/t/cassandra/user/13bwye5433/raid-issue-on-ec2-datastax-ami-1-2-11

Have you already seen something like this?

Thank you,

Philippe

Leaving this here for future work.

We've seen occasional CI EC2 deployments fail in the EU-West (Ireland) AWS region.

The contents of ~/datastax_ami/ami.log suggests apt authentication failures when installing the DataStax/Cassandra packages:

~/datastax_ami/ami.log:

...
[EXEC] 02/12/15-16:29:19 sudo apt-get install -y python-cql datastax-agent cassandra=2.0.11 dsc20=2.0.11-1:
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  python-thrift-basic
The following NEW packages will be installed:
  cassandra datastax-agent dsc20 python-cql python-thrift-basic
0 upgraded, 5 newly installed, 0 to remove and 122 not upgraded.
Need to get 40.6 MB of archives.
After this operation, 46.8 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  cassandra datastax-agent dsc20 python-thrift-basic python-cql

[ERROR] 02/12/15-16:29:19 sudo service cassandra stop:
cassandra: unrecognized service
...

We're using the ami-8932ccfe AMI and supplying the following user data parameters:

--clustername myTestCluster --totalnodes 1 --version community --release 2.0.11

This is not easily repeatable and we only seem to see it during deployments which kick off shortly after midnight (GMT), but this timing way well be a coincidence.

'apt-get update' results in:

W: GPG error: http://debian.datastax.com stable Release: The following signatures were invalid: KEYEXPIRED 1344722043 KEYEXPIRED 1344722043 KEYEXPIRED 1344722043

Looks like the repository key has expired, which results in an endless loop on the machine, and the machine not configuring.
This due to lines 244-248 in ds2_configure.py

https://github.com/riptano/ComboAMI/blob/2.2/ds2_configure.py#L72

This python script doesn't work in VPC due to the lack of resolution on 'instance-data' hostname within a VPC. You can definitely change this hostname to 169.254.169.254 and it works in both EC2 and Instances in VPC[Ec2]

I installed a two-node cluster cassandra. When you try to add a third node(from OpsCenter), an error occurs: FAILURE: Timed out waiting to finish installing DataStax AMI.

Additional information:

node 0

Cluster started with these options:
--clustername maincluster --totalnodes 2 --version community

Raiding complete                                                                                                         
Waiting for nodetool...
The cluster is now in it's finalization phase. This should only take a moment...

Note: You can also use CTRL+C to view the logs if desired:
    AMI log: ~/datastax_ami/ami.log
    Cassandra log: /var/log/cassandra/system.log


Datacenter: eu-west
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Owns (effective)  Host ID                               Token                                    Rack
UN  172.31.27.33  40.94 KB   100.0%            df7786b4-a925-4f74-8866-795e915f8db7  -9220662582844771385                     1b
UN  172.31.27.34  41.08 KB   100.0%            8a4dac75-f774-44a7-a067-dc8143ceb96a  -9213724707703533005                     1b

Opscenter: http://ec2-54-194-xx-xx.eu-west-1.compute.amazonaws.com:8888/
    Please wait 60 seconds if this is the cluster's first start...


Tools:
    Run: datastax_tools
Demos:
    Run: datastax_demos
Support:
    Run: datastax_support


------------------------------------
DataStax AMI for DataStax Enterprise
and DataStax Community
AMI version 2.5
DataStax Community version 2.0.7-1

------------------------------------

node 3

ubuntu@ip-172-31-13-214:~$ cat ~/datastax_ami/ami.log
[EXEC] 05/06/14-11:44:46 git pull:
Updating 5f722d6..fbd0e3e
Fast-forward
 SWITCHES.md                                        |    1 -
 ami_ids.json                                       |   22 +++++
 community_cloud/README.md                          |    5 +
 community_cloud/cloudformation/README.md           |   35 +++++++
 .../cloudformation/cassandra-vpc.template          |  103 ++++++++++++++++++++
 ds1_launcher.py                                    |    2 +
 ds2_configure.py                                   |   74 +++++++++-----
 ds4_motd.py                                        |    7 +-
 presetup/setup.md                                  |    4 +-
 9 files changed, 225 insertions(+), 28 deletions(-)
 create mode 100644 ami_ids.json
 create mode 100644 community_cloud/README.md
 create mode 100644 community_cloud/cloudformation/README.md
 create mode 100644 community_cloud/cloudformation/cassandra-vpc.template

[EXEC] 05/06/14-11:44:46 git reset --hard:
HEAD is now at fbd0e3e Signing commit

[EXEC:E] 05/06/14-11:44:47 gpg --import /home/ubuntu/datastax_ami/repo_keys/DataStax_AMI.7123CDFD.key:
gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/secring.gpg' created
gpg: keyring `/root/.gnupg/pubring.gpg' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 7123CDFD: public key "Joaquin Casares (DataStax AMI) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

[EXEC] 05/06/14-11:44:47 git log --pretty="format:%G?" --show-signature HEAD^..HEAD:
gpg: Signature made Thu Apr 24 04:10:39 2014 UTC using RSA key ID 7123CDFD
gpg: Good signature from "Joaquin Casares (DataStax AMI) <[email protected]>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 6273 EED3 28B4 B09B 2087  1C0E E711 FCAF 7123 CDFD
G
[EXEC] 05/06/14-11:44:47 sudo rm ds2_configure.py
[INFO] Deleting ds2_configure.py now. This AMI will never change any configs after this first run.
[EXEC] 05/06/14-11:44:47 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/Launchpad_VLC.C2518248EEA14886.key:
OK

[EXEC] 05/06/14-11:44:47 sudo apt-key add /home/ubuntu/datastax_ami/repo_keys/Ubuntu_Archive.40976EAF437D05B5.key:
OK

[EXEC] 05/06/14-11:44:47 sudo rm -rf /etc/motd
[EXEC] 05/06/14-11:44:47 sudo touch /etc/motd
[INFO] Started with user data set to:
[INFO] None
[INFO] Using instance type: m1.large
[INFO] meta-data:instance-type: m1.large
[INFO] meta-data:local-ipv4: 172.31.13.214
[INFO] meta-data:public-hostname: ec2-54-194-xx-xx.eu-west-1.compute.amazonaws.com
[INFO] meta-data:ami-launch-index: 0
[INFO] meta-data:reservation-id: r-8c19ffce
[ERROR] Missing required --totalnodes (-n) switch. Aborting installation.

Please verify your settings:
None
[ERROR] Exception seen in ds1_launcher.py:
Traceback (most recent call last):
  File "/home/ubuntu/datastax_ami/ds1_launcher.py", line 22, in initial_configurations
    ds2_configure.run()
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 1015, in run
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 242, in use_ec2_userdata
  File "/home/ubuntu/datastax_ami/ds2_configure.py", line 57, in exit_path
AttributeError: Missing required --totalnodes (-n) switch. Aborting installation.

Please verify your settings:
None

Hi, we are still using Precise (LTS 12.04) while new LTS 14.04 is available. Shouldn't we migrate for security (security updates) and stability ?

Kernel is quite old and it is a ".0" minor version, not sure how bad this is. Still, there is a lot of new kernel versions http://packages.ubuntu.com/precise/kernel/linux-image. Current is 3.8.0-37.

We are seeing weird things in kernel logs and think that maybe a release update or at least a kern update could help. Here is a syslog extract: https://gist.github.com/arodrime/79d9e9879f49de0db880. This trace comes from AWS EC2 (VPC) on PV mode.

Is there any reason that makes us stick to 12.04 (or to this old kernel) ?

Use the reflector code to grab all IPs and set the listen_address and seed_lists appropriately, if ami.conf shows a completed first launch.

Currently this is broken due to argparse getting more than expected. Will cause a rebake.

The process of building ami's with the presetup code is complex and unnecessary. Packer handles this elegantly.