Comments (9)
Dropping the parallelism to 75 (on my laptop at least) means that packets don't drop -- so it does seem like an overwhelming of the system issue.
I am also realising that i am not smart, and I'm reading this jitter results badly. I read 0.288 ms
as 200ms, not 0.2ms - so it's not actually that bad at all. 🤦🏻 Please excuse my panic.
from quilkin.
Suggestion: Stick it in front of a game and see what happens.
Could use Xonotic or Supertuxkart from Agones as an example (in fact, could be an interesting example to have on hand anyway).
from quilkin.
Working on an example with https://iperf.fr/iperf-doc.php#3doc to demo sending lots of packets through - this can be then turned into a proper perf test.
from quilkin.
About to submit a PR, so that everyone can replicate locally, but wanted to capture this.
When running iperf3 against itself on my local machine (no quilkin proxy), I get this result:
[SUM] 0.00-60.00 sec 2.10 GBytes 300 Mbits/sec 0.000 ms 0/68700 (0%) sender
[SUM] 0.00-60.00 sec 2.10 GBytes 300 Mbits/sec 0.009 ms 0/68700 (0%) receiver
Very little jitter, and nothing dropped.
Unfortunately, when I do the same test with Quilkin in between I see the the following:
[SUM] 0.00-60.00 sec 2.10 GBytes 300 Mbits/sec 0.000 ms 0/68700 (0%) sender
[SUM] 0.00-60.00 sec 1.40 GBytes 200 Mbits/sec 0.288 ms 22635/68521 (33%) receiver
As you can see, quite a bit of packet loss and some pretty nasty jitter. I've run this a few times and seen similar results.
I'm not 100% sure it's Quilkin just because I had to do some TCP tunnelling to get iperf3 to work with Quilkin between, and I'm assuming I got it all correct, but worth noting.
I capture Quilkin's prometheus metrics, along with all the logs, and they list no packets dropped, or errors. iperf3.zip
@iffyio when you did you load tests, did you get reports back on packets dropped?
The good new is - CPU is low for the number of packets I'm sending, and memory is very stable.
from quilkin.
Packet loss would be expected if quilkin isn't keeping up with what iperf3 is sending - its recv buffer gets full and the os drops any new packets, which would be fine. As long as quilkin is forwarding all packets it receives which sounds like thats the case. In my previous tests I didn't explicitly check for packet loss but I was pretty sure some where being dropped since the sender there sent as fast as was possible
from quilkin.
Another idea benchmarking: https://www.jibbow.com/posts/criterion-flamegraphs/
In general https://github.com/tikv/pprof-rs looks pretty awesome for looking into things. Shame it doesn't do heap analysis as well.
from quilkin.
Packet loss would be expected if quilkin isn't keeping up with what iperf3 is sending
🤔 CPU is definitely not a bottleneck here. But the point of setting this up is so we can dig into these type of issues. We can keep building out tooling and see where we end up.
I'll adjust the PR to be something that passes a bit better, rather than an overload, and we can take things from there.
from quilkin.
Moving this to high priority because I'd to have to a criterion
benchmark in the repository before release.
from quilkin.
Going to close this, as we now have a couple of benchmarks, and we can track any of them in newer more specific issues.
from quilkin.
Related Issues (20)
- proxy::sessions::tests::same_address_uses_different_sockets deadlocks HOT 3
- Move `Filter::ReadContext` to `ClusterMap`
- Redis Provider/Agent
- Benchmark error: "Resource temporarily unavailable" HOT 1
- Evaluate LTO, CGU=1, Profile-Guided Optimization (PGO) and LLVM BOLT HOT 13
- Replace iperf loadtest with fortio
- Consider Alternatives To DashMap
- Add build-step check for proto submodules HOT 1
- Add Support for Kubernetes native service routing via Gateway API v1 HOT 1
- ClusterMap::PartialEq is incredibly slow HOT 2
- Add licence.html to release quilkin.zip
- Agent <-> Relay communication is failing HOT 4
- Add lz4 compression HOT 2
- `Concatenate.Strategy.Prepend` should maybe be removed HOT 1
- Rust code generated from protobufs should be in source control
- CI is very slow HOT 1
- Cargo.lock should be committed
- Flaky Test: relay::tests::agones_token_router HOT 11
- Make delta xDS more efficient
- Documentation for xDS Delta API HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from quilkin.