Is your feature request related to a problem? Please describe.

The k-NN plugin adds support for a new field type, knn_vector, which can be thought of as an array of floating point numbers. The plugin then also adds support for running approximate k nearest neighbor search on these fields.

For benchmarking the plugin, we are interested in several metrics including:

1. Query latency
2. Indexing throughput
3. Refresh time
4. Recall (the ratio of neighbors returned by an Approximate search that are actually in the ground truth nearest neighbors)
5. Training latency
6. Native memory footprint
7. Disk utilization

Currently, we have our own custom code to get these metrics: https://github.com/opensearch-project/k-NN/tree/main/benchmarks/perf-tool. The reason we decided to build our own tool was that we needed some functionality that was not easily available in Rally/OpenSearch Benchmark: ability to compute the recall from a set of queries, integration of our own custom APIs and metrics, using datasets in alternative forms, etc.

Describe the solution you'd like

We would prefer to use OpenSearch Benchmarks to collect the metrics above so that we don't have to maintain our own tool and that allows customers to not have to adopt another tool other than OpenSearch Benchmarks. I saw #98 was created and I imagine we may need that in order to reach our goal. I would be interested in helping contribute this feature.

Describe alternatives you've considered

The alternative is to continue to use our own benchmarking tool. However, this has several drawbacks mentioned above.

Describe the bug
In CONTRIBUTING.md it says "If we accept the PR, a [maintainer] will merge" and links maintainers to https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/MAINTAINERS.md, but this file 404s.

To Reproduce
Read CONTRIBUTING.md, click on link as above, see the famous Octocat / Landspeeder 404 screen.

Expected behavior
It goes to a real file.

Should be 'OpenSearch-Benchmarks'

Change the pypi artifact name from osbenchmark to opensearch-benchmark

Create a service layer around OpenSearch Benchmark. This will allow users to run OpenSearch Benchmark as a service which could enable to run performance tests with just an API call for various scenarios. This will allow developers to setup OpenSearch Benchmark as a service and not have every user worry about setting up infrastructure to run performance tests. Alongside the service endpoint users should also be able to execute performance tests with OpenSearch Benchmark using command line cli.

OpenSearch Benchmark currently reports various performance metrics and creates a detailed report which can be published to an OpenSearch instance. The goal here is to:

• Create aggregations across many of these metrics to provide a summary report.
• Publish this report in a data store
• Analysis : Report should have insights about the quality of test, if it was anomalous or the test ran successfully, etc.

Acceptance Criteria

• We have a comprehensive dashboard that records the performance stats on regular basis that can be reproduced by anyone (Ex: using CDK to create the entire stack)
• We can view associated PR's and other changes corresponding to a nightly build run
• We can view aggregate stats of performance test runs at any point in time
• We can analyze the raw stats corresponding to a specific performance run for deeper analysis at any point in time
• The dashboard is updated automatically after ever performance test runs
• The dashboard is accessible by everyone ( Read only access enabled for anonymous user )
• Automated alerting is enabled to report on regressions / anomalies
• Admin users have ability to create new dashboards / visualizations as needed ( integrate with OIDC ? )
• The dashboard is maintained on regular basis ( up to date patches, upgrades etc..)

When I try to run benchmarks using http_logs workload, I get an error on index_append task execution.

To Reproduce
I'm using a dockerized environment (specifically the official docker-compose example).

docker-compose up -d 

Then I add to the network a container with Python and I install this library:

docker run -it --entrypoint bash --network=opensearch_opensearch-net python:3.8

# Inside Python container:
root@80f98d9b4de2:/# pip3 install opensearch-benchmark
root@80f98d9b4de2:/# opensearch-benchmark execute_test --workload=http_logs --target-hosts=opensearch-node1:9200,opensearch-node2:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'admin',basic_auth_password:'admin'"

Expected behavior
Get a clean execution with benchmarking results from this specific type of data.

Logs

root@80f98d9b4de2:/# opensearch-benchmark execute_test --workload=http_logs --target-hosts=opensearch-node1:9200,opensearch-node2:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'admin',basic_auth_password:'admin'"

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Executing test with workload [http_logs], test_procedure [append-no-conflicts] and provision_config_instance ['external'] with version [1.2.3].

[WARNING] merges_total_time is 105643 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_throttled_time is 2889 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 1096703 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 117937 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 135 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [  0% done]



[ERROR] Cannot execute_test. Error in load generator [0]
	Cannot run task [index-append]: expected str, bytes or os.PathLike object, not NoneType

Getting further help:
*********************
* Check the log files in /root/.benchmark/logs for errors.
* Read the documentation at https://opensearch.org/docs.
* Ask a question on the forum at https://discuss.opendistrocommunity.dev/.
* Raise an issue at https://github.com/opensearch-project/OpenSearch-Benchmark/issues and include the log files in /root/.benchmark/logs.

--------------------------------
[INFO] FAILURE (took 32 seconds)
--------------------------------

Here you can find also the log file /root/.benchmark/logs/benchmark.log.

More Context:

Versions:

• python:3.8
• opensearchproject/opensearch-dashboards:1.2.0
• opensearchproject/opensearch:1.2.3
• opensearch-benchmark 0.0.1

Additional context

If I use different workloads, benchmarking is working fine. I've only tried with geonames and percolator(but this last one seems to not have a index-append task).

Describe the bug

In the Developer Guide "Setup" section it says:

source .venv/bin/activate

That won't work on a non-unix machine (e.g. Windows). Is there another way of setting up this development environment for windows machines?

DEVELOPER_GUIDE should be updated to include common issues experienced and their respective fixes when running integration tests or using OpenSearch-Benchmark. A lot of the error messages that come out of OpenSearch-Benchmark don't always point users to the right direction.

Is your feature request related to a problem? Please describe.

Repo names are lowercased in this org, except for OpenSearch and OpenSearch Dashboards, by convention.

Describe the solution you'd like

Rename the repo to lowercase.

Copy rally-teams folder to remove need for separate external repository

Create documentation for OpenSearch Benchmark. This should help users understand various nuances inside OpenSearch Benchmark

Describe the bug
I followed https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md to run the benchmark on Amazon Linux Developer Desktop and it failed with following error (logs attached):

[ERROR] Cannot execute_test. Error in task executor
	Download of [/home/ankikala/.benchmark/benchmarks/data/eventdata/eventdata.json.bz2] is corrupt. Downloaded [1159168] bytes but [792768300] bytes are expected. Please retry.

To Reproduce

• Checkout opensearch-benchmark.
• Setup all the prereq in Developer Guide linked above.
• execute make prereq
• execute make install
• source .venv/bin/activate
• Run the benchmark test. sample command: opensearch-benchmark execute_test --workload=eventdata --target-hosts=endpoint --pipeline=benchmark-only --workload-params="number_of_replicas:1" --client-options="use_ssl:true,basic_auth_user:USERNAME,basic_auth_password:PASSWORD,verify_certs:false" --kill-running-processes

Expected behavior
Benchmark test should execute successfully.

Logs
log-devdesktop.txt

Traceback (most recent call last):

 File "/home/ankikala/.pyenv/versions/3.8.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
   result = self.fn(*self.args, **self.kwargs)

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/workload/loader.py", line 418, in prepare_docs
   preparator.prepare_document_set(document_set, data_root[0])

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/workload/loader.py", line 575, in prepare_document_set
   self.downloader.download(document_set.base_url, target_path, expected_size)

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/workload/loader.py", line 490, in download
   net.download(data_url, target_path, size_in_bytes, progress_indicator=progress)

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/utils/net.py", line 239, in download
   raise exceptions.DataError("Download of [%s] is corrupt. Downloaded [%d] bytes but [%d] bytes are expected. Please retry." %

osbenchmark.exceptions.DataError: Download of [/home/ankikala/.benchmark/benchmarks/data/eventdata/eventdata.json.bz2] is corrupt. Downloaded [1159168] bytes but [792768300] bytes are expected. Please retry.```

Update code to pull in the OpenSearch Docker image

Update license file to incldue exact text from apache.org link

Update the Developer guide to the include the following :

• Installation brief
• How to run rally for various use cases
• Running unit and integration tests
• Troubleshooting common issues

Publish a road map to capture list of intended projects to be completed.

Is your feature request related to a problem? Please describe.

In early 2022, Docker Desktop will no longer be free to use. The development of Benchmarks seems to be tied to docker-compose as per the Developer Guide. Docker Desktop is used on Windows and MacOS to provide access to both the Docker Engine and Docker Compose. Meaning many potential developers will not be able to contribute.

Describe the solution you'd like

Link to resources and alter the developer guide to include alternate methods of developing Benchmarks without Docker Desktop (example).

Describe alternatives you've considered

1. Move to Helm chart or similar
2. Change to non-containerized execution somehow.

Describe the bug
Many of the links in the README file are invalid:

• http://osbenchmark.readthedocs.io/en/stable/recipes.html#benchmarking-an-existing-cluster (404)
• https://osbenchmark.readthedocs.io/en/latest/developing.html#installation-instructions-for-development (404)
• https://www.elastic.co/support/matrix#matrix_jvm (do not link to other company)
• https://osbenchmark.readthedocs.io/en/latest/install.html (404)
• https://github.com/opensearch-project/OpenSearch-Benchmark-tracks/tree/master/geonames (404)
• https://osbenchmark.readthedocs.io/ (404)
• https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/benchmark (do not link to other company, link to OpenSearch forums)
• https://www.elastic.co (do not link to other company - you might want to follow the format used by OpenSearch https://github.com/opensearch-project/OpenSearch#copyright )

Describe the bug
The section "Importing the project into an IDE" the sentences that start with "This is typically created via" and "In order to run tests within the IDE" use code markup in an usual way - these are terminal commands or code. I feel like this is referring to a specific IDE but it's not clear.

Change OpenSearch Copyright from 2021 to 2022

Describe the bug
Running processes are not being killed the --kill-running-processes flag is added to a test execution. I think this is a result of changes to as part of #81 not being propogated to the process class:

To Reproduce

1. Execute a test and immediately cancel it using Ctrl+C opensearch-benchmark execute_test --workload=geonames --target-hosts=<host> --pipeline=benchmark-only --test-mode
2. Run ps -aef | grep opensearch-benchmark
3. Execute another test using the flag. Let this run to completion opensearch-benchmark execute_test --workload=geonames --target-hosts=<host> --pipeline=benchmark-only --test-mode --kill-running-processes
4. Run ps -aef | grep opensearch-benchmark again and the same processes from step 2 should still be running

Expected behavior
After step 3 completes there should not be any running OpenSearch-Benchmark processes

Temporary Workaround
Run kill -9 <pid> on each PID output by ps -aef | grep opensearch-benchmark

Add the following license header to opensearch-benchmark-provisionconfigs source files :

# SPDX-License-Identifier: Apache-2.0
#
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.
# Modifications Copyright OpenSearch Contributors. See
# GitHub history for details.

opensearch-benchmark/opensearch-benchmark-provisionconfigs/1.0/plugins/v1/repository_s3/plugin.py
opensearch-benchmark/opensearch-benchmark-provisionconfigs/1.0/plugins/v1/repository_gcs/plugin.py
opensearch-benchmark/opensearch-benchmark-provisionconfigs/1.0/plugins/v1/repository_azure/plugin.py
opensearch-benchmark/opensearch-benchmark-provisionconfigs/main/plugins/v1/repository_s3/plugin.py
opensearch-benchmark/opensearch-benchmark-provisionconfigs/main/plugins/v1/repository_gcs/plugin.py
opensearch-benchmark/opensearch-benchmark-provisionconfigs/main/plugins/v1/repository_azure/plugin.py

Update release.sh and setup.py to publish Benchmark to PyPI

Background

OpenSearch Benchmark supports creating a self-managed OpenSearch cluster on the same machine running OpenSearch Benchmark. Clusters can be created by building OpenSearch from source, downloading a distribution, or using a docker image. A variety of cluster settings can be specified such as heap size, JVM flags, and the garbage collection algorithm.

Proposal

OpenSearch Benchmark’s system under test provisioning capabilities will be expanded to support cloud infrastructure creation. This will allow clusters to be provisioned on infrastructure external to the machine running OpenSearch Benchmark.

Implementations for self-managed OpenSearch running on EC2 instances and managed Amazon OpenSearch Service #100 are in scope for this project. The existing functionality to provision a cluster on the local machine will continue to be supported.

Design

There will be no change to the existing execute_test workflow. The BuilderActor will continue to be invoked as the first step of the BenchmarkActor’s workflow. The WorkerCoordinatorActor will not receive a request to start the test until the BuilderActor has notified the BenchmarkActor that the cluster is ready. Once the test has completed, the BuilderActor will receive a request to delete the cluster from the BenchmarkActor.

The Builder System will consist of the core logic required to provision a local cluster running self-managed OpenSearch. Additional plugins can be defined to provision cloud infrastructure when creating a cluster. Eventually these additional plugins will be moved to a separate repository and fetched via a PluginLoader #98. In the short term, any infrastructure-provider plugins will be implemented as modules within OpenSearch Benchmark’s Builder System.

The existing NodeBuilderActor class will be removed in favor of handling the provisioning logic within the Core Builder. The Dispatcher will remain an intermediary step between the builder plugins and the BuilderActor in order to prevent the CreateCluster call from blocking the BuilderActor while provisioning is underway. Based on the user input, the Core Builder will determine if a provisioning plugin is required and load it for use by the Builder System.

Builder System

The Builder System is responsible for orchestrating the logic to create and delete a system under test. It consists of multiple components that are executed sequentially to perform the provisioning logic. The top-level class and the interface into the Builder System is a Builder.

A Builder has four primary components that handle the provisioning and deprovisioning work. The three existing components, the Downloader, Installer, and Launcher will continue to be used. A fourth component, called the Provisioner, will be created to handle the infrastructure logic.

Builder plugins can create new concrete implementations of the Provisioner, Downloader, Installer, and Launcher for use in the Builder class. The below class diagram shows a sample of the Core Builder and AWS Builder Plugin’s component implementations that feed into the Builder class.

User Experience

A new version (v2) of provision config instances will be created for the Builder System expansion. The v2 configs will incorporate all of the existing options available in v1 configs as well as new options necessary to provision cloud infrastructure. v2 configs will be relied on as the central location for defining a cluster, removing any and all definitions in other locations, such as benchmark.ini and base config.ini files.

The existing CLI options related to provisioning will remain unchanged, with the exception of the --pipeline parameter. If the new builder system is specified by the user, --pipeline=benchmark-only will continue to be used to specify a pre-provisioned cluster. Any other value, or the omission of this parameter, will result in a cluster being provisioned and the parameter value being ignored in favor of the cluster definition in the provision config instance.

Provision Config Instances

In order to simplify the user experience, all mandatory provision config options will have a sensible default value defined in the Core Builder. Each Builder Plugin will have a unique default provision config instance file where the variables section can defined. These variables will be additive to the default provision config instance file in the Core Builder. Users will be able to define their own provision config instances and mixins with as few or as many config options as needed to define their use case.

Additional CLI Commands

The existing top-level OSB CLI commands related to provisioning a cluster outside of a test will remain unchanged. Two new commands will be added to invoke the infrastructure provisioning and deprovisioning workflow.

opensearch-benchmark provision ...

opensearch-benchmark tear_down ...

The list CLI parameter will be expanded to support listing the registered builders.

opensearch-benchmark list builders

Tasks

• Create Core Builder Interfaces
• Create/Refactor Builder Models
• Create/Refactor PCI Parsing Logic
• Migrate Existing Builder Component Implementations
• Write Core Builder Default PCI
• Remove NodeBuilderActor
• Implement Module Loader for Plugins
• Deprecate/Update Builder CLI Parameters
• Implement New CLI Commands
• Add Integration Tests
• Add Error Handling
• [AWS Plugin] Implement Permissions Handling
• [AWS Plugin] Implement SSMExecutor
• [AWS Plugin] Implement EC2Provisioner
• [AWS Plugin] Implement AOSProvisioner
• [AWS Plugin] Write Default PCI
• [AWS Plugin] Wire Into Core Builder

Appendix

Naming Changes

• StartEngine → CreateCluster
• EngineStarted → ClusterCreated
• StopEngine → DeleteCluster
• EngineStopped → ClusterDeleted
• Supplier → Downloader
• Provisioner → Installer

Is your feature request related to a problem? Please describe.

It would be useful to be able to run tests for a specified amount of time in order to test the stability of a cluster over time. My specific use case would be for generating load against a cluster for several days at a time.

Describe the solution you'd like

Run a given workload for a specified amount of time. This could be accomplished by repeating the same workload over and over until the time limit is reached.

Describe alternatives you've considered

Run a given workload for a specified amount in immediate succession. This allows for testing similar use cases but is not as specific since one has to have an estimate of how long a workload takes to run on their specific cluster configuration.

Additional context

N/A

Describe the bug
When OpenSearch-Benchmark executes a test, it writes docs to an index called benchmark-metrics-. For every doc written into benchmark-metrics- index, there is a field called operation-type that is read in from a workload's challenge and is suppose to be formatted with the first letter being uppercase and the rest lowercase. However, the operation-type is currently being written into the index without being formatted.

This enum has a method called from_hyphenated_string() that is suppose to convert the operation-type from the workload to the correct format. However, somehow it's not reaching it. Will need to dive deeper to see.

To Reproduce

1. Download opensearch-benchmark: pip3 install opensearch-benchmark
2. Set results-publishing section to an external opensearch-cluster that will serve as the metrics data store
3. Execute a test

opensearch-benchmark execute_test --pipeline=benchmark-only --workload=nyc_taxis --test-execution-id=12345 --target-hosts=<cluster ip or endpoint>--client-options=basic_auth_user:'<username>',basic_auth_password:'<password>,timeout:60 --telemetry=node-stats --telemetry-params='{\"node-stats-sample-interval\":60}' --kill-running-processes --workload-repository=default --workload-params=number_of_shards:5,number_of_replicas:0,ingest_percentage:10"

4. Check metrics data store via OpenSearch-Dashboards for benchmark-metrics-*. Filter the operation-type.

Expected behavior
Expected Operation-Type --> Actual Operation-Type

• "Bulk" --> "bulk"
• "Search" --> "search"
• "ClusterHealth" --> "cluster-health"

Logs
No logs applicable yet. Will need to deep dive and output to console.log

More Context (please complete the following information):

• Workload(Share link for custom workloads) = default workloads
• Service(E.g OpenSearch) = OpenSearch
• Version (E.g. 1.0) = 1.0 - 1.1

remove xpack references, telemetry, marketing, and replace subcomponent names

OpenSearch Benchmark will not be available as a Docker image with the initial release. The existing Docker related scripts should be removed.

Integration tests that rely on a Benchmark Docker image should also be removed.

Add DCO check to PRs in alignment with OpenSearch

Enable OpenSearch Benchmark to be extensible by having a plugin framework - this framework should allow the following :

• Developers should be able to write their own plugins for specific features like load generator instance provisioning, cluster provisioning, publishing reports to data store, formatting reports in specific formats, etc
• There should be a plugin registry which would allow developers to register plugins

Update the notice file to include opensearch license.

Describe the bug
I followed https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md to run the benchmark locally and it failed with following error (logs attached):

[ERROR] Cannot execute_test. This test_execution ended with a fatal crash.

To Reproduce

• Checkout opensearch-benchmark on OSX.
• Setup all the prereq in Developer Guide linked above.
• execute make prereq
• execute make install
• source .venv/bin/activate
• Run the benchmark test. sample command: opensearch-benchmark execute_test --workload=eventdata --target-hosts=endpoint --pipeline=benchmark-only --workload-params="number_of_replicas:1" --client-options="use_ssl:true,basic_auth_user:USERNAME,basic_auth_password:PASSWORD,verify_certs:false" --kill-running-processes

Expected behavior
Benchmark test should execute successfully.

Logs
osx-log.txt

Additional info

• I verified that there are no listener on 1900 port using lsof -P -n -r 1 -i :1900
• I also tried restarting Mac but it didn't help.
• I see the following error in the logs . Not sure if there is any alternative.

2021-12-15 04:29:48,233 -not-actor-/PID:35422 osbenchmark.actor ERROR Could not initialize internal actor system.
Traceback (most recent call last):
 File "/Volumes/ws/opensearch-benchmark/osbenchmark/actor.py", line 254, in bootstrap_actor_system
   return thespian.actors.ActorSystem(system_base,
 File "/Volumes/ws/opensearch-benchmark/.venv/lib/python3.8/site-packages/thespian/actors.py", line 635, in __init__
   systemBase = self._startupActorSys(
 File "/Volumes/ws/opensearch-benchmark/.venv/lib/python3.8/site-packages/thespian/actors.py", line 676, in _startupActorSys
   systemBase = sbc(self, logDefs=logDefs)
 File "/Volumes/ws/opensearch-benchmark/.venv/lib/python3.8/site-packages/thespian/system/multiprocTCPBase.py", line 28, in __init__
   super(ActorSystemBase, self).__init__(system, logDefs)
 File "/Volumes/ws/opensearch-benchmark/.venv/lib/python3.8/site-packages/thespian/system/multiprocCommon.py", line 86, in __init__
   super(multiprocessCommon, self).__init__(system, logDefs)
 File "/Volumes/ws/opensearch-benchmark/.venv/lib/python3.8/site-packages/thespian/system/systemBase.py", line 326, in __init__
   self._startAdmin(self.adminAddr,
 File "/Volumes/ws/opensearch-benchmark/.venv/lib/python3.8/site-packages/thespian/system/multiprocCommon.py", line 114, in _startAdmin
   raise InvalidActorAddress(adminAddr,
thespian.actors.InvalidActorAddress: ActorAddr-(T|:1900) is not a valid ActorSystem admin```

Build support to provision Amazon OpenSearch Service domains. This should be built as a plugin allowing users to selectively use plugins of their choice.

In compliance with the standards described at semver update the version for alpha release to 0.0.1

The test_list_workloads integration test attempts to load the available workloads from a specific commit hash: 000f2af53b9c34a2e83277e936dac113d497a80f. The challenges directory had not been renamed to test_procedures in that tree, causing the below error:

it/list_test.py::test_list_workloads[in-memory-it]
...

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[ERROR] Cannot list. Workload 'so' is invalid.

Error details: [] is too short
Instance: []
Path: deque(['test_procedures'])
Schema path: deque(['properties', 'test_procedures', 'minItems'])

Getting further help:
*********************
* Check the log files in /Users/engechas/.benchmark/logs for errors.
* Read the documentation at https://opensearch.org/docs.
* Ask a question on the forum at https://discuss.opendistrocommunity.dev/.
* Raise an issue at https://github.com/opensearch-project/OpenSearch-Benchmark/issues and include the log files in /Users/engechas/.benchmark/logs.

-------------------------------
[INFO] FAILURE (took 0 seconds)
-------------------------------
FAILED

Update the issue/PR template to align them for OpenSearch Benchmark.

Describe the bug
In the developer guide under the setup section, running make prereq as described produces this message:

IMPORTANT: please add eval "$(pyenv init -)" to your bash profile and restart your terminal before proceeding any further.

This doesn't appear to be relevant for the setup.

Expected behavior
If there is a message that says to stop (e.g. 'before proceeding') and you don't need to stop then this should be disclosed.

Description

The configuration of Jinja2 enables the application developers to escape any HTML content, thus mitigating attacks like XSS.
However, the auto-escaping is not enabled by default in Jinja2.

With regards to OpenSearch-Benchmark, the following source files have jinja2 with autoescape enabled :

CodePackage: https://github.com/opensearch-project/opensearch-benchmark (

Resolution :

Change jinja2 environment instantiation to enable autoescaped . E.g.

jinja2.Environment(loader=jinja2.FileSystemLoader(root))

with

jinja2.Environment(loader=jinja2.FileSystemLoader(root), autoescape=jinja2.select_autoescape(['html', 'xml']))

Is your feature request related to a problem? Please describe.
OpenSearch Benchmark has automated way to run unit tests, however, no such support exists for integration tests. The goal of this issue is to build an automated trigger to run integration tests.

Describe the solution you'd like
This script is currently used to run unit tests on every pull request. We can extend the same to provide support for integration tests.

Reference : Instruction on running integration tests can be found here.

Describe alternatives you've considered

• Create automation using a separate jenkins infrastructure

Dependency Tree for the opensearch benchmark package https://paste.amazon.com/show/hoangia/1632167121

• Sphinx is used primarily for documentation and no other packages depended on them

(.venv) hoangia@3c22fbd0d988 OpenSearch-Benchmark % pipdeptree --reverse --packages alabaster
alabaster==0.7.12
 - Sphinx==2.2.0 [requires: alabaster>=0.7,<0.8]
  - sphinx-rtd-theme==0.5.1 [requires: sphinx]
(.venv) hoangia@3c22fbd0d988 OpenSearch-Benchmark % pipdeptree --reverse --packages Sphinx  
sphinx==2.2.0
 - sphinx-rtd-theme==0.5.1 [requires: sphinx]

Describe the bug
Running the following on an arm (Graviton) system fails to find version of elastic search because Opensearch uses arm64 and the script runs sysstats.cpu_arch() which returns aarch64

To Reproduce

opensearch-benchmark execute_test --distribution-version=1.2.4 --workload=geonames

Expected behavior
The benchmark runs

Logs

[ERROR] Cannot execute_test. Cannot download OpenSearch distribution from [https://artifacts.opensearch.org/releases/bundle/opensearch/1.0.0/opensearch-1.0.0-linux-aarch64.tar.gz]. Please check that the specified version [1.0.0] is correct.
	Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/osbenchmark/builder/supplier.py", line 536, in fetch
    net.download(download_url, distribution_path, progress_indicator=progress)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/osbenchmark/utils/net.py", line 229, in download
    expected_size_in_bytes = download_http(url, tmp_data_set_path, expected_size_in_bytes, progress_indicator)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/osbenchmark/utils/net.py", line 184, in download_http
    raise urllib.error.HTTPError(url, r.status, "", None, None)
urllib.error.HTTPError: HTTP Error 403:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/osbenchmark/builder/builder.py", line 563, in receiveMsg_StartNodes
    self.builder.start_engine()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/osbenchmark/builder/builder.py", line 678, in start_engine
    binaries = self.supply()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/osbenchmark/builder/supplier.py", line 288, in __call__
    supplier.fetch()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/osbenchmark/builder/supplier.py", line 541, in fetch
    raise exceptions.SystemSetupError("Cannot download OpenSearch distribution from [%s]. Please check that the specified "
osbenchmark.exceptions.SystemSetupError: Cannot download OpenSearch distribution from [https://artifacts.opensearch.org/releases/bundle/opensearch/1.0.0/opensearch-1.0.0-linux-aarch64.tar.gz]. Please check that the specified version [1.0.0] is correct.

More Context (please complete the following information):

• Workload(Share link for custom workloads): Opensearch-benchmark
• Service(E.g OpenSearch)
• Version (E.g. 1.0)

Additional context

The problem happens here.

A fix would be to add the following:

self.arch = sysstats.cpu_arch().lower()
if self.arch == "aarch64":
   self.arch = "arm64"

or to offer downloads for open search under both the arm64 and aarch64 names.

Update contributing file to align with OpenSearch Benchmark.

Describe the bug
Running OpenSearch Benchmark after a fresh pip installation fails because of a markupsafe versioning issue. A similar issue can be found here: aws/aws-sam-cli#3661

It looks like MarkupSafe 2.1 was just released yesterday: https://markupsafe.palletsprojects.com/en/2.1.x/changes/

Temporary workaround is to manually install markupsafe 2.0.1 after installing OpenSearch Benchmark using pip install markupsafe==2.0.1

To Reproduce
pip install opensearch-benchmark
opensearch-benchmark list telemetry

Expected behavior
OpenSearch Benchmark should run after being installed.

Logs

opensearch-benchmark list telemetry
Traceback (most recent call last):
  File "/home/ec2-user/.pyenv/versions/3.8.0/bin/opensearch-benchmark", line 5, in <module>
    from osbenchmark.benchmark import main
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/osbenchmark/benchmark.py", line 37, in <module>
    from osbenchmark import version, actor, config, paths, \
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/osbenchmark/test_execution_orchestrator.py", line 33, in <module>
    from osbenchmark import actor, config, doc_link, \
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/osbenchmark/worker_coordinator/__init__.py", line 26, in <module>
    from .worker_coordinator import (
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/osbenchmark/worker_coordinator/worker_coordinator.py", line 43, in <module>
    from osbenchmark import actor, config, exceptions, metrics, workload, client, paths, PROGRAM_NAME, telemetry
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/osbenchmark/workload/__init__.py", line 25, in <module>
    from .loader import (
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/osbenchmark/workload/loader.py", line 33, in <module>
    import jinja2
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/home/ec2-user/.pyenv/versions/3.8.0/lib/python3.8/site-packages/markupsafe/__init__.py)

OpenSearch Benchmark workloads currently use an existing dataset to ingest records into OpenSearch/ElasticSearch. The goal for this task is to build support for automatically generated datasets for a workload. This would enable workloads to have large auto-generated datasets without requiring any specific data.

Acceptance Criteria

• We have generated 3 to 5 datasets for custom workloads based on real world use cases
• The generated workloads can be used by OpenSearch benchmark to measure specific performance stats for OpenSearch cluster
• The custom workloads are specific to the needs (Ex: smaller workloads for executing against each PR, medium workloads for more frequent runs, complex workloads to capture specific issues / regression for longevity tests etc..)

Automatically run unit tests as part of every PR

Is your feature request related to a problem? Please describe.
We want to use opensearch benchmark for triggering performance tests for cross cluster replication. This issue is to track the changes required.

Describe the solution you'd like

• Support using a telemetry device which can read the cross cluster replication status and log into the metric store:
  Status API
• Support passing multiple endpoints using Opensearch-benchmark.

Additional context

• Since opensearch-benchmark is a fork of Elasticsearch's rally, it already has support for CCR on elasticsearch.
• Ex: https://www.elastic.co/blog/benchmarking-elasticsearch-cross-cluster-replication
• Telemetry device: https://github.com/opensearch-project/opensearch-benchmark/blob/main/osbenchmark/telemetry.py#L317
• Rally recipe: https://github.com/elastic/rally/blob/master/recipes/ccr/start.sh

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior.

• Download OpenSearch-Benchmark via pip3 install opensearch-benchmark
• Run the following command:opensearch-benchmark execute_test --distribution-version=1.0.0 --workload=http_logs --test-mode

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[ERROR] Cannot execute_test. Error in builder ([Errno 2] No such file or directory: '/home/ec2-user/.benchmark/benchmarks/workloads/default/http_logs/test_procedures/common/opensearch-benchmark-provisionconfigs')

Expected behavior
It should print out the test-mode report. It requires accessing the opensearch-benchmark-provisionconfigs directory within the opensearch-benchmark repository. Two ways to get pass this is either following the setup in the DEVELOPER_GUIDE.md or cloning the repository and cd-ing into that repository and running the opensearch-benchmark command once more.

Logs
https://pastebin.pl/view/2045f7d8

More Context (please complete the following information):

• Workload = any workload (i.e. geonames or http_logs)
• Service = OpenSearch
• Version = OpenSearch-Benchmark 1.0.0

Is your feature request related to a problem? Please describe.

OpenSearch Benchmark currently uses Elasticsearch client. This goal of this issue is to replace the Elaticsearch client with the OpenSearch Python client

Describe the solution you'd like

Replace the Elasticsearch client with OpenSearch.

Describe alternatives you've considered

None

Additional context