elastic / apm-perf Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Using apm{bench|soak}
in a distributed way to scale load generation would cause the benchmark results to be broken. We should handle this by introducing a mechanism to select a leader/coordinator. The leader will be in charge of collecting and reporting benchmark metrics while the non-leader nodes will generate load and communicate their state to the leader. The communication can be explicit (using a gossip protocol) or implicit (by monitoring benchmark metrics).
Looks like the apmsoak binary was pushed by mistake: https://github.com/elastic/apm-perf/blob/main/cmd/apmsoak/apmsoak
The file is 42MB and is significantly increasing the size of the repo. We should probably delete it.
We could also remove it completely from the git history to reduce the size even further (I'm did not investigate this so I don't know if it's going to make a sensible difference). This would also require a force push but since the repo only has a few commits from a single person, it's not going to be an issue.
Describe the bug
Encountered this panic when running apmsoak in the tilt/AWS setup
{"log.level":"error","@timestamp":"2024-03-19T14:54:09.768Z","log.logger":"soaktest","log.origin":{"function":"github.com/elastic/apm-perf/internal/loadgen/eventhandler.logResponseOutcome","file.name":"eventhandler/apm.go","file.line":90},"message":"request failed","status_code":401,"response":"{\"message\":\"Unauthorized\"}\n","ecs.version":"1.6.0"}
panic: runtime error: slice bounds out of range [1267810:759964]
goroutine 21 [running]:
bytes.(*Buffer).Bytes(...)
/usr/local/go/src/bytes/buffer.go:54
net/http.NewRequestWithContext({0x26893d8?, 0x40001440f0}, {0x43673f?, 0x0?}, {0x400012a1b0?, 0x30?}, {0x2685380, 0x40000b4040?})
/usr/local/go/src/net/http/request.go:876 +0x670
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*APMTransport).SendEvents(0x400011ca80, {0x26893d8?, 0x40001440f0?}, {0x2685380, 0x40000b4040}, 0x0?)
/opt/apm-perf/internal/loadgen/eventhandler/apm.go:50 +0x60
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*Handler).sendBatch(0x400014a0c0, {0x26893d8, 0x40001440f0}, 0x40000a5b60, {{0x4001ab2240, 0x23b, 0x23b}, {0x4004e00000, 0xea9, 0x1300}}, ...)
/opt/apm-perf/internal/loadgen/eventhandler/handler.go:358 +0x450
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*Handler).sendBatches(0x400014a0c0, {0x26893d8, 0x40001440f0}, 0x40000a5b60)
/opt/apm-perf/internal/loadgen/eventhandler/handler.go:307 +0x390
github.com/elastic/apm-perf/internal/loadgen/eventhandler.(*Handler).SendBatchesInLoop(0x400014a0c0, {0x26893d8, 0x40001440f0})
/opt/apm-perf/internal/loadgen/eventhandler/handler.go:269 +0x74
github.com/elastic/apm-perf/internal/soaktest.runAgent({0x26893d8, 0x40001440f0}, 0x4000126480, {{0x400013c140, 0x20}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
/opt/apm-perf/internal/soaktest/runner.go:127 +0x204
github.com/elastic/apm-perf/internal/soaktest.(*Runner).Run.func1()
/opt/apm-perf/internal/soaktest/runner.go:103 +0x128
golang.org/x/sync/errgroup.(*Group).Go.func1()
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x5c
created by golang.org/x/sync/errgroup.(*Group).Go
/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x9c
Instead of running otel collector as part of apmbench
, we should run apmbench
and otel-collector
separately. The collector should expose endpoints to query and reset the aggregated metrics and the apmbench
should use them.
Make a new, dedicated home for server performance testing tools. The repo should have a license added, and be made public.
The soaktest runner sends events to the intake v2 endpoints and is able to generate load from the major agents however there's no option to send events to the otlp endpoint.
We should add support for that
Currently, we sleep for a random duration of 1m
between benchmarks to allow any remaining entries in the queue to be consumed. This is not ideal, we should have a more deterministic way of detecting a clean slate after a benchmark has run by monitoring some metric.
Ref:
Lines 94 to 97 in 4ce0d0c
As of #24, we have 2 commands with apm-perf
: apmsoak
and apmbench
. Consider using buildkite with ko image builder (ref)
Looks like we put something in a nil map
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.