azure-samples / jmeter-aci-terraform Goto Github PK

After running the pipeline over and over we get the following error upon terraform apply
Error: Error Creating/Updating Virtual Network "<name-goes-here>" (Resource Group "<name-goes-here>"): network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InUseSubnetCannotBeDeleted" Message="Subnet <name-goes-here> is in use by /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Network/networkProfiles/<net-profile>/containerNetworkInterfaceConfigurations/<name-goes-here>/ipConfigurations/<name-goes-here> and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet." Details=[]

Terraform destroy appears to destroy everything but somehow the ipConfigurations is staying but not showing in portal.

Has anybody else undergone this error?

When using a single worker, we should optimize to not use JMeter in distributed mode, since it currently creates 1 controller and 1 worker. Instead, we should spawn just a single jmeter instance.

Need some way to identify if a load test is failing because ACI is being pushed beyond it's limits or if the site being tested is failing.

The documentation should give an example on how to do a large scale load test across multiple jMeter workers Guidance should be included as to how many virtual users can be handled per worker (ACI).

Add a note on the Create Variable Groups section on README about not adding capital letters on the beginning of the ACR name. Otherwise, the Azure CLI will broke when pushing JMeter image to ACR.

Example:

Instead of Myacr.azurecr.io, use myacr.azurecr.io

when in my local docker, I can pass out the count variable(or some others) but in this sample pipeline can't recognize the outer variables.

what I have changed:

1. In sample.jmx add a Count variable to control loop count using ${__P(Count)}

2. In main.tf pass it using -JCount=5
   commands = [ "/bin/sh", "-c", "cd /jmeter; /entrypoint.sh -n -J server.rmi.ssl.disable=true -t ${var.JMETER_JMX_FILE} -l ${var.JMETER_RESULTS_FILE} -JCount=5 -e -o ${var.JMETER_DASHBOARD_FOLDER} -R ${join(",", "${azurerm_container_group.jmeter_workers.*.ip_address}")} ${var.JMETER_EXTRA_CLI_ARGUMENTS}", ]

it not works as expect but works well in my local docker. can I get some suggestions ?

Hello,

I see that there is another issue that is talking about there test looping although I don't know if its similar.

I have a test plan that is hitting a certain site with the thread group a follows:

<ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <intProp name="LoopController.loops">-1</intProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">1000</stringProp>
        <stringProp name="ThreadGroup.ramp_time">5</stringProp>
        <boolProp name="ThreadGroup.scheduler">true</boolProp>
        <stringProp name="ThreadGroup.duration">300</stringProp>
        <stringProp name="ThreadGroup.delay">0</stringProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">false</boolProp>
      </ThreadGroup>

Now the problem is once the test is finished it never stops. It is on a constant loop despite the duration being set to 5 minutes. I can tell the test has stopped via azures application insights.
When I stop the worker and controller container the test will stop.

Any help would be appreciated thanks.

Running the pipeline and getting this error on the AZCopy step,
pipelines/azure-pipelines.load-test.yml Line 62
`- script: |
azcopy
--source $(JMETER_DIRECTORY_INPUT)
--destination $(terraform output storage_file_share_url)
--dest-key $(terraform output storage_account_key)
--recursive
workingDirectory: ./terraform
displayName: 'SETUP: Transfer JMeter Files to Storage Account'`

And the error:
`
jobs Sub-commands related to managing jobs
list List the entities in a given resource
load Sub-commands related to transferring data in specific formats
login Log in to Azure Active Directory (AD) to access Azure Storage resources.
logout Log out to terminate access to Azure Storage resources.
make Create a container or file share.
remove Delete blobs or files from an Azure storage account
sync Replicate source to the destination location

Flags:
--cap-mbps float Caps the transfer rate, in megabits per second. Moment-by-moment throughput might vary slightly from the cap. If this option is set to zero, or it is omitted, the throughput isn't capped.
-h, --help help for azcopy
--output-type string Format of the command's output. The choices include: text, json. The default value is 'text'. (default "text")
--trusted-microsoft-suffixes string Specifies additional domain suffixes where Azure Active Directory login tokens may be sent. The default is '.core.windows.net;.core.chinacloudapi.cn;.core.cloudapi.de;.core.usgovcloudapi.net;*.storage.azure.net'. Any listed here are added to the default. For security, you should only put Microsoft Azure domains here. Separate multiple entries with semi-colons.
-v, --version version for azcopy

py [command] --help" for more information about a command.

unknown flag: --source
##[error]Bash exited with code '1'.
Finishing: SETUP: Transfer JMeter Files to Storage Account
Bash exited with code '1'.
`

When we click on python script link in https://docs.microsoft.com/en-us/samples/azure-samples/jmeter-aci-terraform/jmeter-aci-terraform/?source=docs 404 is coming.

This is fixed by PR #51

Improve the documentation by adding context about what is a service principal and how to fill out the following info:

CLIENT_ID=
CLIENT_SECRET=
TENANT_ID=
SUBSCRIPTION_ID=

Hi,
ERROR: The subscription of 'subscription_id' doesn't exist in cloud 'AzureCloud'. is thrown at 'Login on Azure CLI' job when running docker pipeline. I have same tenant id for 5 of my azure subscriptions.

Settings do not match the block on Bash and JMETER_AZURE_PRINCIPAL

CLIENT_ID : $ARM_CLIENT_ID
CLIENT_SECRET : $ARM_CLIENT_SECRET
TENANT_ID : $ARM_TENANT_ID
SUBSCRIPTION_ID : $ARM_SUBSCRIPTION_ID

Block on Bash

CLIENT_ID=
CLIENT_SECRET=
TENANT_ID=
SUBSCRIPTION_ID=

JMETER_AZURE_PRINCIPAL

PRIN_GROUP_ID=$(az pipelines variable-group create  --name JMETER_AZURE_PRINCIPAL --authorize \
                                                    --variables ARM_CLIENT_ID=$ARM_CLIENT_ID \
                                                                ARM_TENANT_ID=$ARM_TENANT_ID \
                                                                ARM_SUBSCRIPTION_ID=$ARM_SUBSCRIPTION_ID \
                                                                | jq .id)

az pipelines variable-group variable create --group-id $PRIN_GROUP_ID --secret true \
                                            --name ARM_CLIENT_SECRET \
                                            --value $ARM_CLIENT_SECRET

The documentation should outline how to calculate the costs in Azure for running tests. This should include costs for the Azure Container Registry, Azure Container Instances, Storage, and any other related services. The calculation examples should show for different load test scales with multiple jMeter ACI's

As per issue #32 and #38 , there are users who need to pass additional parameters to JMeter controller CLI. This will allow it be done using an environment variable (TF_VAR_JMETER_EXTRA_CLI_ARGUMENTS)

First of all I'm missing the correct TF flow: Tf Plan, Apply to finally destroy.
About the issue I don't get why to use TF Apply with -target azurerm_storage_share.jmeter_share I believe because of this I'm receiving the following error:

Error: A resource with the ID "/subscriptions/subsidxxxxx/resourceGroups/RG/providers/Microsoft.Network/virtualNetworks/jmetervnet" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_virtual_network" for more information.

I'll try apply all first before continue to move files and tests, but please give a feedback.

Thanks!

I am trying to run performance test on 2 different worker by using Azure triggers functionality, so for example running test for 1hr and then keeping 30 min gap and then another run for 1hr,by creating triggers in Azure.
1st Run always run fine but during 2nd run I always see issue mentioned below.

Error: Error creating/updating container group "jmeter-worker1" (Resource Group "jmeter"): containerinstance.ContainerGroupsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="NetworkProfileNotFound" Message="Network profile '/subscriptions/abcdefg-tttt-54we-6589-eeee98g65236/resourceGroups/jmeter/providers/Microsoft.Network/networkProfiles/jmeternetprofile' is not found for container group 'jmeter-worker1'."

on main.tf line 75, in resource "azurerm_container_group" "jmeter_workers":
75: resource "azurerm_container_group" "jmeter_workers" {

Error: Error creating/updating container group "jmeter-worker0" (Resource Group "jmeter"): containerinstance.ContainerGroupsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="NetworkProfileNotFound" Message="Network profile '/subscriptions/abcdefg-tttt-54we-6589-eeee98g65236/resourceGroups/jmeter/providers/Microsoft.Network/networkProfiles/jmeternetprofile' is not found for container group 'jmeter-worker0'."

on main.tf line 75, in resource "azurerm_container_group" "jmeter_workers":
75: resource "azurerm_container_group" "jmeter_workers" {

Any Idea how to resolve this issue?

Here's my tf command and I'm passing in the following value to JMETER_SETTING_PROTOCOL: https

commands = [ "/bin/sh", "-c", "cd /jmeter; /entrypoint.sh -n -Jserver.rmi.ssl.disable=true -t ${var.JMETER_JMX_FILE} -l ${var.JMETER_RESULTS_FILE} -e -o ${var.JMETER_DASHBOARD_FOLDER} -R ${join(",", "${azurerm_container_group.jmeter_workers.*.ip_address}")} -Jprotocol=${var.JMETER_SETTING_PROTOCOL}" ]

Here's the output from the "RESULTS: Show JMeter Controller Logs" in the pipeline:
jmeter args=-n -Jserver.rmi.ssl.disable=true -t BasicLoadStatic.jmx -l results.jtl -e -o dashboard -R 10.0.0.4 -Jprotocol=https

Here's the error showing up in my dashboard:
Non HTTP response code: java.net.MalformedURLException/Non HTTP response message: unknown protocol: 1

Here's the variable I'm using in my jmx file: post: ${__P(protocol)}

I've tried a bunch of stuff and looked everywhere for solutions and haven't figured out why it isn't accepting this parameter. I can run the same cmdline arguments locally without problems so it's either the tf command line syntax or perhaps the handling in entrypoint.sh (which I'm off to look at now). I wasn't clear on how it handles the parameters but it looked like it supported additional params. Any ideas?

I am trying to run the jmx file which has CSV Data Set Config, in this case I have added my testdata.csv file in the jmeter repository and tried to run the the jmx file, but looks like test failed with null pointer exception, I suspect jmx file is not getting data from testdata.csv file in this case. Could you please let me know how can I solve this issue, where shall I add the testdata.csv file so that jmx can read the data from it?

Hi all,

I have a test: 2 users are running about 25 rest calls for 50 times. I think that this takes about almost 1 hour but it breaks before it ends.

It runs fine if I execute the same jmx file for 5 users 20 times.
my tests run fine for 20 minutes but when I want to run it for about an hour I get this error:
TEST: Wait Test Execution
->
/usr/bin/az account set --subscription 4fe8cc8e-d9ed-4b5a-91c4-e2471d71e675
/bin/bash /home/vsts/work/_temp/azureclitaskscript1622822721406.sh
Fri Jun 4 16:05:24 UTC 2021: Started!
Fri Jun 4 16:05:25 UTC 2021: Still Running...
...
Fri Jun 4 16:58:57 UTC 2021: Still Running...
##[error]The operation was canceled.
Finishing: TEST: Wait Test Execution

So the test was actually running 53 minutes.

It happened before:
usr/bin/az account set --subscription 4fe8cc8e-d9ed-4b5a-91c4-e2471d71e675
/bin/bash /home/vsts/work/_temp/azureclitaskscript1622816548628.sh
Fri Jun 4 14:22:31 UTC 2021: Started!
Fri Jun 4 14:22:32 UTC 2021: Still Running...
Fri Jun 4 14:22:54 UTC 2021: Still Running...
...

Fri Jun 4 15:15:54 UTC 2021: Still Running...
##[error]The operation was canceled.
Finishing: TEST: Wait Test Execution

So also 53 minutes.

Does anybody know debug this issue?
So 53 minutes also

We should have some unit tests for jtl_junit_converter.py

It should guarantee that no errors are introduced by new PRs. We should verify:

• Docker images are building
• Terraform code is valid

It's not a issue, only a suggestion.

When I tried to copy and past the information "az login && az extension add --name azure-devops"
the && is not copying correctly. So I needed to break the command and two stages.

1. Logging on Azure
2. Set Up which subscription I want to work
3. Added the DevOps extension.

Thank you for your work.

AZcopy does not take in account key anymore which breaks the pipeline. I'll eventually get to submitting a PR, but a quick fix is below

`

• task: AzureCLI@2
  displayName: 'SETUP: Transfer JMeter Files to Storage Account'
  inputs:
  azureSubscription: $(AZURE_SERVICE_CONNECTION_NAME)
  scriptType: 'bash'
  workingDirectory: ./terraform
  scriptLocation: 'inlineScript'
  inlineScript: |
  SHARENAME=$(terraform output storage_file_share_name)
  ACCOUNTNAME=$(terraform output storage_account_name)
  STGKEY=$(terraform output storage_account_key)
  az storage file upload-batch --account-key $STGKEY --account-name $ACCOUNTNAME --destination $SHARENAME --source $(JMETER_DIRECTORY_INPUT)
  `

`

• task: AzureCLI@2
  displayName: 'RESULTS: Download Jmeter results'
  inputs:
  azureSubscription: $(AZURE_SERVICE_CONNECTION_NAME)
  scriptType: 'bash'
  workingDirectory: ./terraform
  scriptLocation: 'inlineScript'
  inlineScript: |
  RG=$(terraform output resource_group_name)
  SHARENAME=$(terraform output storage_file_share_name)
  STGKEY=$(terraform output storage_account_key)
  ACCOUNTNAME=$(terraform output storage_account_name)
  mkdir $(System.DefaultWorkingDirectory)/results
  az storage file download-batch --account-key $STGKEY --account-name $ACCOUNTNAME --destination $(JMETER_DIRECTORY_OUTPUT) --no-progress --source $SHARENAME
  `

The current main.tf file gives us the following warning when creating a azurerm_subnet:

Warning: "address_prefix": [DEPRECATED] Use the `address_prefixes` property instead.

We should use the new property.

the website we want to test is not public, and we have to whitelist IPs. for this reason we cant use the normal azure load tests services, in addition to the azure load test jmeter offering being too limited (no custom listeners or samplyers, and only jmeter 3.2).

Does this tool have any way to use it with fixed outgoing IPs?

Hello,

I was wondering can anybody help me. Currently when launching the pipeline it is using jmeter version 5.1.1 is there a way to get a newer version. I can see that you are pulling from justb4/jmeter although the current version they are using is 5.4.
Thanks

Create pipelines that support GitHub actions.
Vote +👍 if you need this feature.

During a load test I had some HTTP requests that failed due to the remote system unable to handle the load. Once the test completed the "Publish Load Testing Results" step shows the error "##[error]There are one or more test failures detected in result files. Detailed summary of published test results can be viewed in the Tests tab." which is expected since some http requests failed. The issue is the following step which should upload the jMeter results (RESULTS: Publish Load Test Artifacts) does not run due to the prior step's error. This means there is no way to see the actual jMeter dashboard when any failure occurs.

Hello,
My intention is to Run multiple jmx file on different agents but at the same time, so for example as show below.

1. JMX 1 which has 10000 user should run on agent 1
2. JMX2 which has 10000 user should run on agent 2
3. JMX3 which has 10000 user should run on agent 3 likewise
   Could you please guide me how can I achieve this?

Add the az extension add --name azure-devops step in the Getting Started section on README.

When noting the prerequisites please add that ACR needs admin access enabled. Instructions for both CLI and UI are here: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication#admin-account

Otherwise great repo, I forked it to tailor it for Azure Cognitive Search latency and concurrency tests

We should replace those terms in order to use inclusive language.

to achieve this jmeter command -
jmeter -n -t "sample.jmx" -Jemail=*** -Jpassword=*** -Juser.classpath="somejar.jar" -l results.jtl -j log.txt

I updated main.tf line 161.
"cd /jmeter; /entrypoint.sh -n -J server.rmi.ssl.disable=true -t ${var.JMETER_JMX_FILE} -l ${var.JMETER_RESULTS_FILE} -e -o ${var.JMETER_DASHBOARD_FOLDER} -R ${join(",", "${azurerm_container_group.jmeter_workers.*.ip_address}")} ${var.JMETER_EXTRA_CLI_ARGUMENTS}"

to
"cd /jmeter; /entrypoint.sh -n -J server.rmi.ssl.disable=true -t ${var.JMETER_JMX_FILE} -l ${var.JMETER_RESULTS_FILE} -e -o ${var.JMETER_DASHBOARD_FOLDER} -R ${join(",", "${azurerm_container_group.jmeter_workers.*.ip_address}")} ${var.JMETER_EXTRA_CLI_ARGUMENTS} -Jemail=${var.EMAIL} -Jpassword=${var.PASSWORD} -Juser.classpath=${var.SOMEJAR} '

but at time of execution this parameter is not getting invoked.
Can anyone let me know if i am doing anything wrong in this

Teardown steps should be always executed especially when some error occurs in the pipeline.

For example, we execute Run Terraform Apply (target=file share) step, and then we face an error after that. Pipeline at this point is not usable due to resources are already created and the user needs to go and manually delete resources that pipeline created in order to unplug pipeline.

Hi,
Sometimes the execution of tests is going into an infinite loop.
In the build agent, we are getting a message at the execution of the test step as
Still Running...
Still Running...
We have configured it to execute it for a Duration of 600 Seconds.
Can you please let us know how to debug this issue further?

Hello everyone, how are you? So, I was a heavy user from this project on the last weeks (thanks a lot for that) and there are two things that I took some time to find since the docs doesn't say anything about:

• In order to use Azure cli with Azure DevOps, you need to install an extension for that.
• The command to add parameters on the pipelines seem wrong, since default value is also a mandatory argument on the command line.

I'll open a PR in order to fix this things I told. Do you it makes sense?
Thanks a lot, again!

I didn't understand what is necessary to perform the most basic test and publish.
At Jmeter folder I just let the sample.jmx I'm receiving the following error on the moment of Convert Jmeter Resullts:

I understand the script python don't found results.jtl , but the tasks before works without errors and generate this files on artifact:

What I'm missing out?

Hi there,

I need some help with some specific task.

We need to run 4 worker nodes with each 500 threads = unique users
The users will be calculated based on the current thread number and node_index.
Also users are unique as we calculate the offset based on the number of users per node and node index.

Here is the script if someone is interested

This works fine when running locally with following command for first node:

jmeter -n -t loadtest.jmx -l testresults.jtl -Jnode_index=0 -Jusers_per_node=500

for second node

jmeter -n -t loadtest.jmx -l testresults.jtl -Jnode_index=1 -Jusers_per_node=500

How can I make this work with this terraform setup?
I need to make sure that every user is unique across all nodes and ideally I do not need to maintain some CSV, as we have the need to vary between test cases and go up to 10000 users as well.

Thank you in Advance!

I get the following error when I try to run the load-test pipeline

Warning: "address_prefix": [DEPRECATED] Use the address_prefixes property instead.

on main.tf line 22, in resource "azurerm_subnet" "jmeter_subnet":
22: resource "azurerm_subnet" "jmeter_subnet" {

Error: Error ensuring Resource Providers are registered.

Terraform automatically attempts to register the Resource Providers it supports to
ensure it's able to provision resources.

Is this something on my end? I tried multiple stuff(giving all permission to user, etc.)

I used to run Jmeter using the command line non gui mode and passing the needed property variables from there to have configurable test plans instead of having multiple jmx scripts.

jmeter -JnoThreads=2 -JrampUp=1 -n -t sample.jmx

I simply passed the number of threads value and ramp up values.

Is there a way to replicate that? Or do I need to update the pipeline settings?

If I need to do updating on the pipeline, can you provide examples?

We need to create a setup.sh script that provisions and setup everything accordantly to README.md.

This should be a Shell script. The developer can also implement a PowerShell version if he/she/they wants. (as #27 already reported problems)

I am looking for a business context of when I should use this sample (e.g. doing unit tests, integration tests using http, or load testing for messaging system and performing). This info will help me to decide when to use this.

Hi,
When I tried to create more slaves (6 slaves), SETUP: Run Terrafarm Apply task keeps running more than 45 minutes and not finished. When I checked my list of container instances, only one slave was in Running state and rest of the 5 were in Pending state. What could be the issue?

When I set up a cluster with below, my JMeter Master CPU utilization is 3.77 GB which is above 90% of the available resource. I cant create a master with more CPU since 4 GB is the maximum allowed CPU. I have the cluster created in West Europe Azure region. Moreover, Neither JMeter Artefacts are generated nor Pipeline run results. So, I dont have any detailed logs to analyse exactly what caused the failure

My load profile: 800 Concurrent Threads with 200 seconds ramp up time and 300 seconds run duration
Master: 4 CPU, 16 GB Memory;
4 Slaves each with 4 vCPU, 16 GB Memory