Git Product home page Git Product logo

apm-perf's Introduction

apm-perf

apm-perf contains server performance testing tools.

apmsoak

Soak testing involves testing apm-server against a continuous, sustained workload to identify performance and stability issues that occur over an extended period. apmsoak command can be used for both classic APM Server and managed APM service(in development) for the purpose of soak testing:

❯ make build
❯ ./dist/apmsoak run --help
Run apmsoak

Usage:
  apmsoak run [flags]

Flags:
      --api-keys project_id_1:my_api_key,project_id_2:my_key   API keys for managed service. Specify key value pairs as project_id_1:my_api_key,project_id_2:my_key
      --bypass-proxy                                           Detach from proxy dependency and provide projectID via header. Useful when testing locally
  -f, --file string                                            Path to scenarios file (default "./scenarios.yml")
  -h, --help                                                   help for run
      --scenario string                                        Specify which scenario to use. the value should match one of the scenario key defined in given scenarios YAML file (default "steady")
      --secret-token string                                    Secret token for APM Server. Managed intake service doesn't support secret token
      --server-url string                                      Server URL (default http://127.0.0.1:8200), if specified <project_id>, it will be replaced with the project_id provided by the config, (example: https://<project_id>.apm.elastic.cloud)

All the flags are optional, but it's expected to provide a scenario flag to start a specific scenario from pre-defined scenarios file, it is steady by default.

# start soaktest that generates 2000 events per second
./apmsoak run --scenario=steady

The configs in scenarios.yml file inherits loadgen configs, with extra fields such as project_id and api_key that can be used for the managed APM service.

Note that the managed service will use API key based auth and one soaktest can target multiple projects, so we provide key-value pairs(projectID and api key respectively). the api key can also be provided as an ENV variable.

./apmsoak run --scenario=fairness --api-keys="project_1:key-to-project_1,project_2:key-to-project_2"

internal/telemetrygen

This package is from https://github.com/open-telemetry/opentelemetry-collector-contrib but slightly modified so that it can be used within apm-perf.

apm-perf's People

Contributors

amannocci avatar axw avatar carsonip avatar dependabot[bot] avatar endorama avatar inge4pres avatar kruskall avatar kyungeunni avatar lahsivjar avatar lamrecognitions avatar pazone avatar v1v avatar vikmik avatar

Stargazers

avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

kyungeunni marclop carsonip v1v lahsivjar kruskall endorama inge4pres reakaleek 1337cybersecurity lamrecognitions axw

apm-perf's Issues

remove apmsoak binary

Looks like the apmsoak binary was pushed by mistake: https://github.com/elastic/apm-perf/blob/main/cmd/apmsoak/apmsoak

The file is 42MB and is significantly increasing the size of the repo. We should probably delete it.

We could also remove it completely from the git history to reduce the size even further (I'm did not investigate this so I don't know if it's going to make a sensible difference). This would also require a force push but since the repo only has a few commits from a single person, it's not going to be an issue.

[apmbench/apmsoak] Allow scaling of load generation with meaningful benchmark results

Using apm{bench|soak} in a distributed way to scale load generation would cause the benchmark results to be broken. We should handle this by introducing a mechanism to select a leader/coordinator. The leader will be in charge of collecting and reporting benchmark metrics while the non-leader nodes will generate load and communicate their state to the leader. The communication can be explicit (using a gossip protocol) or implicit (by monitoring benchmark metrics).

Separate running otel collector as part of `apmbench`

Instead of running otel collector as part of apmbench, we should run apmbench and otel-collector separately. The collector should expose endpoints to query and reset the aggregated metrics and the apmbench should use them.

feat: add support for otlp endpoints

The soaktest runner sends events to the intake v2 endpoints and is able to generate load from the major agents however there's no option to send events to the otlp endpoint.

We should add support for that

`apmperf`: panic slice out of bounds

Describe the bug

Encountered this panic when running apmsoak in the tilt/AWS setup

{"log.level":"error","@timestamp":"2024-03-19T14:54:09.768Z","log.logger":"soaktest","log.origin":{"function":"github.com/elastic/apm-perf/internal/loadgen/eventhandler.logResponseOutcome","file.name":"eventhandler/apm.go","file.line":90},"message":"request failed","status_code":401,"response":"{\"message\":\"Unauthorized\"}\n","ecs.version":"1.6.0"}
panic: runtime error: slice bounds out of range [1267810:759964]

goroutine 21 [running]:
bytes.(*Buffer).Bytes(...)
	/usr/local/go/src/bytes/buffer.go:54
net/http.NewRequestWithContext({0x26893d8?, 0x40001440f0}, {0x43673f?, 0x0?}, {0x400012a1b0?, 0x30?}, {0x2685380, 0x40000b4040?})
	/usr/local/go/src/net/http/request.go:876 +0x670
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*APMTransport).SendEvents(0x400011ca80, {0x26893d8?, 0x40001440f0?}, {0x2685380, 0x40000b4040}, 0x0?)
	/opt/apm-perf/internal/loadgen/eventhandler/apm.go:50 +0x60
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*Handler).sendBatch(0x400014a0c0, {0x26893d8, 0x40001440f0}, 0x40000a5b60, {{0x4001ab2240, 0x23b, 0x23b}, {0x4004e00000, 0xea9, 0x1300}}, ...)
	/opt/apm-perf/internal/loadgen/eventhandler/handler.go:358 +0x450
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*Handler).sendBatches(0x400014a0c0, {0x26893d8, 0x40001440f0}, 0x40000a5b60)
	/opt/apm-perf/internal/loadgen/eventhandler/handler.go:307 +0x390
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*Handler).SendBatchesInLoop(0x400014a0c0, {0x26893d8, 0x40001440f0})
	/opt/apm-perf/internal/loadgen/eventhandler/handler.go:269 +0x74
github.com/elastic/apm-perf/internal/soaktest.runAgent({0x26893d8, 0x40001440f0}, 0x4000126480, {{0x400013c140, 0x20}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
	/opt/apm-perf/internal/soaktest/runner.go:127 +0x204
github.com/elastic/apm-perf/internal/soaktest.(*Runner).Run.func1()
	/opt/apm-perf/internal/soaktest/runner.go:103 +0x128
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x5c
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x9c

Make benchmark cleanup deterministic

Currently, we sleep for a random duration of 1m between benchmarks to allow any remaining entries in the queue to be consumed. This is not ideal, we should have a more deterministic way of detecting a clean slate after a benchmark has run by monitoring some metric.

Ref:

apm-perf/cmd/apmbench/run.go

Lines 94 to 97 in 4ce0d0c

// TODO (lahsivjar): Make this deterministic by introducing cleanup
// metrics. We can watch the cleanup metrics to reach to a specified
// threshold and then run the next benchmark.
time.Sleep(time.Minute)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.