awr / kafkaclient Goto Github PK
View Code? Open in Web Editor NEWThis project forked from gigya/kafkanetclient
Native C# client for Kafka
License: Apache License 2.0
This project forked from gigya/kafkanetclient
Native C# client for Kafka
License: Apache License 2.0
Many of the producer/consumer tests have issues
I see that this repository is not updated in a long time, can you tell us how far this client from production quality ?
This can be included as part of the nuget package (in tools), and ideally would mimic the canonical tool from a parameter point of view (at least where it makes sense to do so).
When decoding objects received from the server, it is possible to run into scenarios where we allocate too much memory (because of bad transmission, a bug on the server, man-in-the-middle, etc). There should be a way for the user to specify a maximum size for arrays that are allocated when decoding.
This could be a single value across the board, or could be more specific (perhaps message numbers are much larger than topic response numbers, for example).
Simulation of load with a multi server environment is necessary to ensure that the client doesn't fall down.
Currently, only gzip is supported.
There are updated protocols -- see http://kafka.apache.org/protocol.html for details
Producer.SendBatchWithCodecAsync should retry when failing because of metadata changes.
It should likely take a similar approach to BrokerRouterExtensions.GetTopicOffsetsAsync
A lot of complexity in here, which requires testing. In particular, the retry capabilities with multiple partitions going to different leaders (when leader arrangement is in the middle of changing).
Currently, this is just some code that you can build. Not very discoverable.
This only works on .net 4.5.2 at this point.
The simple solution is to move to .net 4.6 and core 1.1. Note that this will require adopting a versions of AsyncEx.* projects.
This will require heartbeats, group join semantics, etc.
It seems this lib lacks the SSL support.
I need it, so I would hack it & contribute.
Any suggestions, ideas, maybe some guidance?
The purpose is to be able to add/remove topics and consumer groups.
This can be used for tooling, but more also for test setup (currently integration tests fail on first time through because of the autocreation lag).
Ideally this is done with the "internal" apis so it's legit and there are no other external dependencies for tests and operational usage.
For running integration tests on commit against CI
This will be a 0.10 server
The goal is to keep coverage for tests on CI >= 90% in unit + integration tests, and >= 80% for unit tests.
To verify that this is happening, we need coverage running from CI.
Currently, opencover is failing with the new pdb format -- see OpenCover/opencover#732. Once they update the nuget package available and we adopt it, we should start seeing coverage numbers.
Fall back to default ssl setup if none configured and url specified is in the format http://
The library needs to be tuned from a performance point of view, especially wrt sending and receiving messages at high volume.
There is likely some low hanging fruit if profiled. It would also be good to benchmark against a couple other c# libraries, at least for the simple scenarios.
This may require refactoring the ReconnectingSocket code so the SslTransport can participate in the retry cycle directly.
To close this, we should have tests to prove this is working when ssl auth fails -> succeeds.
There are a number of new APIs (21-33) as well as many new versions of existing APIs. The on-the-wire representation needs to be fleshed out for these, along with tests.
A dependent of #16
Using random, consistent message bytes.
BenchmarkDotNet=v0.10.1, OS=Windows
Processor=?, ProcessorCount=8
Frequency=2435767 Hz, Resolution=410.5483 ns, Timer=TSC
dotnet cli version=1.0.0-preview2-1-003177
[Host] : .NET Core 4.6.24628.01, 64bit RyuJIT
DefaultJob : .NET Core 4.6.24628.01, 64bit RyuJIT
CodecNone
)Note that speed is important, but so is memory allocation and byte size. Byte size will directly impact network traffic costs (especially in terms of overall speed).
Method | Messages | MessageSize | Codec | Mean | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
Encode | 100 | 1 | CodecNone | 82.9727 us | 0.8513 us | 7.4056 | - | - | 63.63 kB |
Encode | 100 | 1000 | CodecNone | 1,211.5400 us | 9.5125 us | 31.7708 | 31.7708 | 31.7708 | 429.46 kB |
Encode | 10000 | 1 | CodecNone | 8,695.0426 us | 46.7663 us | 702.0833 | 47.9167 | 45.8333 | 6.43 MB |
Encode | 10000 | 1000 | CodecNone | 131,300.6830 us | 845.3136 us | 1454.1667 | 641.6667 | 641.6667 | 46.84 MB |
Codec | Level | Messages | MessageSize | Bytes |
---|---|---|---|---|
CodecNone | - | 100 | 1 | 2746 |
CodecNone | - | 100 | 1000 | 102646 |
CodecNone | - | 10000 | 1 | 270046 |
CodecNone | - | 10000 | 1000 | 10260046 |
CodecGzip | Fastest | 100 | 1 | 147 |
CodecGzip | Optimal | 100 | 1 | 132 |
CodecGzip | Fastest | 100 | 1000 | 2272 |
CodecGzip | Optimal | 100 | 1000 | 1816 |
CodecGzip | Fastest | 10000 | 1 | 1718 |
CodecGzip | Optimal | 10000 | 1 | 782 |
CodecGzip | Fastest | 10000 | 1000 | 100861 |
CodecGzip | Optimal | 10000 | 1000 | 56002 |
CodecGzip
with byte[]
)Method | Messages | MessageSize | Codec | Level | Mean | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|---|
Encode | 100 | 1 | CodecGzip | Fastest | 108.3579 us | 0.5767 us | 7.0313 | - | - | 73.96 kB |
Encode | 100 | 1 | CodecGzip | Optimal | 103.9860 us | 0.2018 us | 7.0313 | - | - | 73.94 kB |
Encode | 100 | 1000 | CodecGzip | Fastest | 1,304.7041 us | 9.7604 us | 33.8542 | 33.8542 | 33.8542 | 442.34 kB |
Encode | 100 | 1000 | CodecGzip | Optimal | 1,331.5181 us | 8.1201 us | 36.9792 | 36.9792 | 36.9792 | 442.37 kB |
Encode | 10000 | 1 | CodecGzip | Fastest | 8,978.8525 us | 78.0883 us | 710.4167 | 56.2500 | 54.1667 | 6.45 MB |
Encode | 10000 | 1 | CodecGzip | Optimal | 8,897.3779 us | 88.5309 us | 718.7500 | 64.5833 | 62.5000 | 6.45 MB |
Encode | 10000 | 1000 | CodecGzip | Fastest | 133,817.3749 us | 1,517.0871 us | 1075.0000 | 262.5000 | 262.5000 | 45.02 MB |
Encode | 10000 | 1000 | CodecGzip | Optimal | 138,365.5674 us | 305.4618 us | 1104.1667 | 291.6667 | 291.6667 | 46.15 MB |
CodecGzip
with Stream
)Method | Messages | MessageSize | Codec | Level | Mean | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|---|
Encode | 100 | 1 | CodecGzip | Fastest | 108.4666 us | 1.6359 us | 6.2826 | - | - | 73.51 kB |
Encode | 100 | 1 | CodecGzip | Optimal | 108.6534 us | 1.1747 us | 6.5430 | - | - | 73.51 kB |
Encode | 100 | 1000 | CodecGzip | Fastest | 1,291.8933 us | 21.6539 us | 32.8125 | 32.8125 | 32.8125 | 437.78 kB |
Encode | 100 | 1000 | CodecGzip | Optimal | 1,288.3538 us | 9.6701 us | 33.8542 | 33.8542 | 33.8542 | 437.78 kB |
Encode | 10000 | 1 | CodecGzip | Fastest | 8,635.0205 us | 74.1442 us | 718.7500 | 64.5833 | 62.5000 | 6.44 MB |
Encode | 10000 | 1 | CodecGzip | Optimal | 8,621.5494 us | 82.3327 us | 677.0833 | 22.9167 | 20.8333 | 6.44 MB |
Encode | 10000 | 1000 | CodecGzip | Fastest | 137,845.2882 us | 5,184.4836 us | 1158.3333 | 345.8333 | 345.8333 | 44.82 MB |
Encode | 10000 | 1000 | CodecGzip | Optimal | 137,451.9864 us | 1,243.2845 us | 1187.5000 | 375.0000 | 375.0000 | 45.72 MB |
The stream approach seems to be marginally better for memory allocation (consistent across runs), with timing in the same ballpark (varying across runs). I'm going to opt for the streaming approach.
The level isn't quite as clear -- it appears that Optimal is likely worth using, but I'd like to see what happens when unzipping first.
There is a need for SASL support at the Connection level (to be triggered right after API version support is triggered), with a default implementation of PLAIN. This should be extensible so any user of the client can provide their own implementation if it isn't included in the library.
Consumer:
Assignment:
Management / monitoring:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.