awr / kafkaclient Goto Github PK

Many of the producer/consumer tests have issues

I see that this repository is not updated in a long time, can you tell us how far this client from production quality ?

This can be included as part of the nuget package (in tools), and ideally would mimic the canonical tool from a parameter point of view (at least where it makes sense to do so).

When decoding objects received from the server, it is possible to run into scenarios where we allocate too much memory (because of bad transmission, a bug on the server, man-in-the-middle, etc). There should be a way for the user to specify a maximum size for arrays that are allocated when decoding.

This could be a single value across the board, or could be more specific (perhaps message numbers are much larger than topic response numbers, for example).

Simulation of load with a multi server environment is necessary to ensure that the client doesn't fall down.

Currently, only gzip is supported.

There are updated protocols -- see http://kafka.apache.org/protocol.html for details

Producer.SendBatchWithCodecAsync should retry when failing because of metadata changes.
It should likely take a similar approach to BrokerRouterExtensions.GetTopicOffsetsAsync

A lot of complexity in here, which requires testing. In particular, the retry capabilities with multiple partitions going to different leaders (when leader arrangement is in the middle of changing).

Currently, this is just some code that you can build. Not very discoverable.

This only works on .net 4.5.2 at this point.

The simple solution is to move to .net 4.6 and core 1.1. Note that this will require adopting a versions of AsyncEx.* projects.

This will require heartbeats, group join semantics, etc.

It seems this lib lacks the SSL support.
I need it, so I would hack it & contribute.
Any suggestions, ideas, maybe some guidance?

The purpose is to be able to add/remove topics and consumer groups.
This can be used for tooling, but more also for test setup (currently integration tests fail on first time through because of the autocreation lag).

Ideally this is done with the "internal" apis so it's legit and there are no other external dependencies for tests and operational usage.

For running integration tests on commit against CI

This will be a 0.10 server

The goal is to keep coverage for tests on CI >= 90% in unit + integration tests, and >= 80% for unit tests.

To verify that this is happening, we need coverage running from CI.

Currently, opencover is failing with the new pdb format -- see OpenCover/opencover#732. Once they update the nuget package available and we adopt it, we should start seeing coverage numbers.

Fall back to default ssl setup if none configured and url specified is in the format http://

The library needs to be tuned from a performance point of view, especially wrt sending and receiving messages at high volume.

There is likely some low hanging fruit if profiled. It would also be good to benchmark against a couple other c# libraries, at least for the simple scenarios.

This may require refactoring the ReconnectingSocket code so the SslTransport can participate in the retry cycle directly.

To close this, we should have tests to prove this is working when ssl auth fails -> succeeds.

There are a number of new APIs (21-33) as well as many new versions of existing APIs. The on-the-wire representation needs to be fleshed out for these, along with tests.

A dependent of #16

Encoding of ProduceRequest

Using random, consistent message bytes.

BenchmarkDotNet=v0.10.1, OS=Windows
Processor=?, ProcessorCount=8
Frequency=2435767 Hz, Resolution=410.5483 ns, Timer=TSC
dotnet cli version=1.0.0-preview2-1-003177
  [Host]     : .NET Core 4.6.24628.01, 64bit RyuJIT
  DefaultJob : .NET Core 4.6.24628.01, 64bit RyuJIT

Benchmark Baseline (CodecNone)

Note that speed is important, but so is memory allocation and byte size. Byte size will directly impact network traffic costs (especially in terms of overall speed).

Benchmark Gzip (CodecGzip with byte[])

Benchmark Gzip (CodecGzip with Stream)

The stream approach seems to be marginally better for memory allocation (consistent across runs), with timing in the same ballpark (varying across runs). I'm going to opt for the streaming approach.

The level isn't quite as clear -- it appears that Optimal is likely worth using, but I'd like to see what happens when unzipping first.

There is a need for SASL support at the Connection level (to be triggered right after API version support is triggered), with a default implementation of PLAIN. This should be extensible so any user of the client can provide their own implementation if it isn't included in the library.

Consumer:

• do not lose message when blocked
• querying should move to the next offset automatically
• cancellation interrupts consumption
• coordinator state changes are managed correctly (when heartbeating, etc)
• when in the middle of processing a message and changing state, resolution for current position is satisfied

Assignment:

• only assigned partitions are read from
• if more members are subscribed than partitions, additional members gracefully exit (and do not cause excessive chatter)
• if same cardinality, one member is assigned to one partition
• if more partitions than members, all partitions are still assigned (as evenly as possible)
• works with 1 or many subscriptions
• extensible to a priority based assignment

Management / monitoring:

• can list topics
• can list consumer groups and offsets
• throughput telemetry numbers lie in correct range