oliverlloyd / jmeter-ec2 Goto Github PK

They're slow and tend to cause issues

Right now code and files are sent up and down between the computer running the script and each instance. This can create a lot of traffic when the volume of instances increases and doesn't scale well because upload speeds tend to be slow from people's PCs.

A better solution would be to use an S3 bucket. Files could be uploaded there once and then the instances told to read from that location.

In the same way results could be written to this location and the client given a url to the file but this would mean it would not be possible to concatenate and sort the files.

Hi,

Using keypairs with spaces in their name makes the jmeter-ec2.sh hang on the ec2-run-instances call, with a message "waiting for stdin".

I tried a number of escaping/quoting hacks ("${AMAZON_KEYPAIR_NAME}", "${AMAZON_KEYPAIR_NAME}" etc) and they worked fine in bash but I unable to get them to work inside the script.

When I generated a new keypair without spaces in the name it worked like a charm.

Thanks,

• Jan

Return a msg something like: 1 of 20 installs complete while checking

Hello! Love the script.

I'm getting jmeter-ec2 to work in vagant so I can debug my jmx files and such before I spend the $$ on ec2 instances, it's working but I had to do a few things.

The version of ssh (at least on Mac OS 10.8 (OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011)) doesn't like the syntax host:port

$ ssh -i $HOME/.vagrant.d/insecure_private_key [email protected]:2222
ssh: Could not resolve hostname 127.0.0.1:2222: nodename nor servname provided, or not known

But does like host -p port

$ ssh -i $HOME/.vagrant.d/insecure_private_key [email protected] -p2222
Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-17-generic x86_64)

I have a patch for this (and its silly ssh uses -p and scp uses -P!) but it's not flexible if you have multiple remote machines with multiple ports (i just added a $REMOTE_PORT var)

I haven't attached the code as I'm still playing and wanted to chat about the right approach.

When I request one Instance all works really well. When I request more than one instance I have recently been receiving the following error: Client.InvalidInstanceID.Malformed: Invalid id: (instance id's)
This appears to be an issue with the following line of code:
count_passed=$(ec2-describe-instance-status --region $REGION ${attempted_instanceids[@]} | awk '/INSTANCESTATUS/ {print $3}' | grep -c passed)

The instances are actually initiated. However the script fails due to the error above. I am trying to debug but without too much success at present.- any ideas?

I have a meta jmx file that uses the Include Controller that includes all scripts for individual projects. This a useful layout for when you have many projects going through a single gateway. I use the individual jmx files for testing individual APIs, then include them in the meta jmx file for the biblical, full load tests.

jmeter-ec2 doesn't recognize the Include Controller for things like thread management/etc, and thats probably ok as it may add a lot of complexity. Also since it copies the jmx file from the $project/jmx/meta.jmx to /tmp/execute.jmx on the remote host, the include controller gets confused as the layout locally is different than the layout remotely. (../data/ vs data/)

Work around is to copy the included jmx files into the data/ dir and as that layout matches the layout on the remote server. I have a simple bash script to copy the jmx files and the csv files into the data dir.

Eg:

project/
 - meta.jmx
 data/
  - foo.jmx
  - bar.jmx
  - foo.csv
  - bar.csv
 results/
 jmx/
  - meta.jmx # Same as above

Not really a feature request but more a work around if anyone has the same issue and to show how i worked around the issue.

Hello,

During test runs, the running totals on the load controller are always 0. If I SSH into one of the load generators, and tail the jmeter.log file, I do see the correct running totals. @oliverlloyd, any idea what would be causing this?

Load Controller:

> 19:13:04: [RUNNING TOTALS] total count: 0, current avg: 0 (ms), average tps: 0 (p/sec), recent tps: 0 (p/sec), total errors: 0

jmeter.log on load generator:

2014/07/14 19:15:00 INFO  - jmeter.reporters.Summariser: summary +     13 in    14s =    0.9/s Avg:   136 Min:   117 Max:   188 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
2014/07/14 19:15:00 INFO  - jmeter.reporters.Summariser: summary =    154 in   190s =    0.8/s Avg:   148 Min:    44 Max:   766 Err:     0 (0.00%)

Talk about:

• security groups
• AMIs
• key files
• zone

This issue lists the various things that can cause this error but it's closed and the information is hidden. Better would be to add a section to the readme and maybe print a reference to it in the script output.

Note: The results and tests tables were created successfully, and a new test was inserted into the tests table, however I got the following error when the results were being imported:

processing results...(standard_in) 2: syntax error
copying import-results.sh to database...done....uploading jtl file to database..done....importing jtl file....ERROR 1148 (42000) at line 1: The used command is not allowed with this MySQL version
.done

MySQL Version
Server version: 5.5.37-0ubuntu0.14.04.1 (Ubuntu)

I use Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-5189a661 .

It starting correctly and being active during test, but all the time I see only

waiting for the test to start...to stop the test while it is running, press CTRL-C. This message doesn't disappear, and it can hang there one hour.

If I do ctrl+c, script exiting successfully, destroying temporary instance, and returns empty report.

Any suggestions what can cause this problem?

The instructions are not available:
http://www.http503.com/2012/jmeter-ec2/

Make sure we handle the autoflush feature in jmeter.properties

# AutoFlush on each line written in XML or CSV output
# Setting this to true will result in less test results data loss in case of Crash
# but with impact on performances, particularly for intensive tests (low or no pauses)
# Since JMeter 2.10, this is false by default
# jmeter.save.saveservice.autoflush=false

Hi,
it would be nice if this article link http://www.http503.com/2012/jmeter-ec2/ could be updated. I'm new to was so this article would be really helpful for me to set up the environment.

Thanks!

This is always being read as 100. The progress bar code stole the variable name 'percent' and overwrote it.

Hi Oliver,

Congrats on creating an exceptional batch file which helps us a lot. I am trying to build a script which also incorporates the InfluxDB + Grafana Dashboard as well.

I have not tested this, but I would like to know if you have any insight if I had a GraphiteBackendListener into the script and make the test push the data into InfluxDB and I want to be able to view the dashboards in Grafana?

I am looking at tweaking your scripts, but meanwhile wanted to understand your views.

Sorry, I am new to jmeter, so maybe this is a silly question. I have the files in the correct directory and they upload to the remote host, but I am getting ERR in my tests as it does not seem to be loading the files correctly. I am using: /${_StringFromFile()}, to pass in content dynamically. Any idea why this wouldn't be working? Am I missing a jar file or something?

Also: Thanks for this repo.. sure beats the distributed jmeter!

Move from the legacy tools that are currently being used to the new api.

As per #10, connection errors sometimes occur and can be caused by things that could have been checked by the script.

Intermittently the install process for 1 or more AWS instances fails and the script hangs filling up the page with endless periods.

The problem is the location where jmeter is being downloaded from is becoming very slow. We need to identify a better location from which to download the jmeter bundle.

This is a feature request rather than a bug. Currently when using this I'm allocating elastic ip's before a test run, then editing the jmeter-ec2.properties file with the ip's, running the test, then releasing them after (so we're not charged by amazon for unused elastic ip's).

Was wondering if it could be built into this somehow, like take the number of instances parameter at the start and allocate that many elastic ip's automatically (and throw an error if you've tried to allocate more than your account is allowed), and then release them automatically after the test finishes/instances are terminated.

Might be a problem with how jmeter is reading the host from the OS.

If the jmeter process is out of memory then sending StopTestNow calls to it can fail. This hangs the script and prevents tidy up and termination.

Maybe send the calls to the background and then sleep a bit, and then continue with termination?

First - thank you so much for such an impressive script. It has already saved me hours of work.

I was using this script successfully for days, but started seeing errors when running install.sh... I updated all my scripts to the latest version and found an appropriate ubuntu image for the agents. I'm now getting the following and just wondering if anyone else is experiencing something similar. Thanks again and I'll update this issue as I find out more. Best regards, Hal

   jmeter-ec2 Automation Script - Running myproject.jmx over 2 AWS Instance(s)

requesting 2 instance(s)...success

waiting for instance status checks to pass (this can take several minutes).....
all hosts ready

assigning tags...
TAG instance i-8cd856f5 ProjectName myproject
TAG instance i-8ed856f7 ProjectName myproject

TAG instance i-8cd856f5 Name jmeter-ec2-myproject
TAG instance i-8ed856f7 Name jmeter-ec2-myproject
complete
copying install.sh to 2 server(s)....complete

Extracting templates from packages: 100%
Extracting templates from packages: 100%
..........tar: This does not look like a tar archive

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
mv: cannot move /tmp/JMeterPlugins.jar' to/tmp/apache-jmeter-2.6/lib/ext/': No such file or directory
mv: cannot move /tmp/mysql-connector-java-5.1.16-bin.jar' to/tmp/apache-jmeter-2.6/lib/': No such file or directory
.....tar: This does not look like a tar archive

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
mv: cannot move /tmp/JMeterPlugins.jar' to/tmp/apache-jmeter-2.6/lib/ext/': No such file or directory
mv: cannot move /tmp/mysql-connector-java-5.1.16-bin.jar' to/tmp/apache-jmeter-2.6/lib/': No such file or directory
.complete

Hello,

I am using the 'Ultimate Thread Group' plugin which should be available with the 'JMeterPlugins.jar' file. However, an error is thrown (see below) unless 'JMeterPlugins-Extras.jar' file is also added.

Uncaught Exception java.lang.AbstractMethodError: org.apache.jmeter.threads.AbstractThreadGroup.start(ILorg/apache/jmeter/threads/ListenerNotifier;Lorg/apache/jorphan/collections/ListedHashTree;Lorg/apache/jmeter/engine/StandardJMeterEngine;)V. See log file for details.

Could the JMeterPlugins-Extras.jar file be added to the script as the same way as JMeterPlugins.jar?

The summariser has a new time format and the results file has field names added by default now.

Just tried running this script, was giving me errors, took me ages to debug but I think this is the issue:

https://s3.amazonaws.com/jmeter-ec2/JMeterPlugins.jar

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>AllAccessDisabled</Code>
    <Message>All access to this object has been disabled</Message>
    <RequestId>74BA2D3C3841B0BD</RequestId>
    <HostId>e98OUXoHbq07QNr4kpBEsBIdjQONQCXSGveSDQ508X6q4oecj/eOFjxn2LbBzGTP</HostId>    
</Error>

Can we have a mirror to this link or have it fixed please? - I tried installing another plugins.jar but kept getting errors with missing dependancies and such.

Thanks!

Don't add so many tags and don't print out so much when we do add them

Don't wait for instance checks, check for ssh access and use this to decide ready state.

    # E.g.
    for host in ${hosts[@]} ; do
      if [ ! "$(ssh -q \
        -o StrictHostKeyChecking=no \
        -o "BatchMode=yes" \
        -o "ConnectTimeout=15" \
        -i "$PEM_PATH/$PEM_FILE" \
        -p $REMOTE_PORT \
        $USER@$host echo up)" == "up" ] ; then
        echo "Host $host is not responding"
      fi
    done

Hi Oliver – Nice work on the script. I am experiencing the same issues with lost connection when copying over the install.sh.

copying install.sh to 1 server(s)…lost connection
.grep: /Users/workspace/testing-ws/jmeter-ec2/sample-project/1364519238*scpinstall.out: No such file or directory

I have verified my security group is configured correctly and can ssh into the box by doing a telnet ping on 22

telnet ec2-23-23-69-1.compute-1.amazonaws.com 22 ⏎
Trying 23.23.69.1…
 Connected to ec2-23-23-69-1.compute-1.amazonaws.com.
 Escape character is ‘^]’.
 SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
  ^C

However, the scp of install.sh fails and I can’t get past that point. I have even put a sleep 60s as suggested by Zack on (http://www.http503.com/2012/run-jmeter-on-amazon-ec2-cloud/), to no avail.

Any ideas why this might be happening? Is there a work around?

Thanks in advance.

Hello,

The plugins directory does not appear to get copied over from the $project_home directory, even though the jmeter-ec2.sh script does appear to have a conditional check for it

            # scp any project specific custom jar files
            if [ -d $project_home/plugins ] && [ -n $(ls $project_home/plugins/) ] ; then # don't try to upload any files if none present
                echo -n "project specific jar file(s)..."
                for host in ${hosts[@]} ; do
                    (scp -q -C -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no \
                                                  -i "$PEM_PATH/$PEM_FILE" -P $REMOTE_PORT \
                                                  $project_home/plugins/*.jar \
                                                  $USER@$host:$REMOTE_HOME/$JMETER_VERSION/lib/ext/) &
                done
                wait
                echo -n "done...."
            fi

An error is returned

copying test files to 1 server(s)...jmx files..done....data dir..done....jmeter.properties..done....jmeter execution file...done..../home/ubuntu/projects/jmeter-ec2/jmeter-ec2.sh: line 569: [: too many arguments
all files uploaded

Line 569 of jmeter-ec2.sh

if [ -d $project_home/plugins ] && [ -n $(ls $project_home/plugins/) ] ; then # don't try to upload any files if none present

Note: I have about 10 jars I need to get into apache/lib/ext.

When trying to use Elastic IPs, I'm seeing the following:

assigning elastic ips...
ADDRESS xx.xx.xx.xx i-xxxxxxxxx             
complete

checking elastic ips...complete

copying install.sh to 1 server(s)...lost connection
.grep: /home/ubuntu/tests/perftest/1404744761*scpinstall.out: No such file or directory
.grep: /home/ubuntu/tests/perftest/1404744761*scpinstall.out: No such file or directory
.grep: /home/ubuntu/tests/perftest/1404744761*scpinstall.out: No such file or directory

this missing variable in the example vagrant cfg file is causing this error to appear in while running a test:

./jmeter-ec2.sh: line 854: [: ==: unary operator expected

Hi, I want to perform load test using master-slave.

For Eg :

I have 1 master and 10 slaves running and 1000 threads which are specified in master jmx file which should run in such a manner so that thread can be distributed among 10 slaves equally.

Is there any way to modify jmx file in such a manner ? or any other alternatives ?

Thanks !!

These should be removed by the cleanup function but they and also the working_n files are left behind.

Hi

Everything is working fine but no real-time aggregated results is displayed on console.
Generate Summary Results listener is added to the test plan so it should not be this problem.

Still no output when using the provided 'myproject'

I was trying to run jmeter-ec2.sh from an amazon instance (which uses ec2-user by default), deploying an ubuntu base AMI (ami-018c9568). Although I specified "ubuntu" (the default ubuntu user) in the jmeter-ec2.properties file, it fails at the install.sh script, and repeatedly outputs 'grep /home/ec2-user/...' It should be using /home/ubuntu.

To get around this, I setup an ubuntu instance instead, and am running jmeter-ec2.sh from there.

This is no longer used and confusing

Sometimes the install process can fail on an instance which currently stalls the entire test. We should time out the checks and let the test continue with the remaining hardware so that a failure on 1 instance does not waste the entire test run.

The delay happens before the check loop runs. I suspect we need to run this command in the background.

Can you explain why this is the case: File paths cannot be dynamic, any variables in the filepath will be ignored. What would it take to allow this behaviour?

Not passing this value means the zone is chosen automatically which is actually better for load testing as Amazon will choose the zone best able to serve requests are that time.

looks like the recommended amazon instance as per instructions is not available. Can you please provide a public instance which we can use

Things have moved on a bit and there's been a few things patched onto the readme to reference various changes but now it's a little confusing.

Hi,

Whenever i run the script it wont go past waiting for "instance status checks to pass", ive left it on for about an hour and it hasnt moved from there. i can confirm from my amazon panel that the instances are running and the checks are 2/2. Some help please?

Create a suite of recommended AMIs (over all regions) that already have Java, JMeter, etc. pre-installed. The install.sh script can still check and install things as required but like this the default path will be substantially faster.