Comments (9)
How much does the synchronization delay the processing given relatively high cost of message parsing, disk IO, etc?
My concern about per-thread rate limiting is that it relies on even distribution of the load across threads which I don't think is a fair assumption. Ideally, it would be nice to have throttling at even higher level than process (e.g., a broker) but it's rather complicated to implement.
from secor.
How much does the synchronization delay the processing given relatively high cost of message parsing, disk IO, etc?
I would say it depends on number of threads and message rate among other things. According to my observation in a single secor process with 10 threads this particular lock was eating up as much as 70% of total execution time.
My concern about per-thread rate limiting is that it relies on even distribution of the load across threads which I don't think is a fair assumption. Ideally, it would be nice to have throttling at even higher level than process (e.g., a broker) but it's rather complicated to implement.
I agree. How about offering a thread-based limiting as an alternative to existing? Until a better solution comes up at least.
from secor.
I have to say that I'm shocked with this result. If it's indeed the case that most of the time is spent waiting for the lock, we should indeed consider alternatives such as per-thread locking or not locking at all if secor.messages.per.second is set to a negative value.
from secor.
I started implementing a NoRateLimiter
that extended the RateLimiter
class, but for some reason I couldn't really get it to work, probably due to my java-fu being a bit low nowadays. I'm gonna give it another go this weekend if anyone (besides me) is still interested in this. I also saw a lot of time being consumed by the rate limiting lock.
from secor.
I may be totally wrong but I still find it a bit hard to believe that locking in the rate limiter will have such a tremendous impact on the performance. Could the NoRateLimiter be 'implemented' simply by setting secor.messages.per.second to a ridiculously high value?
from secor.
Dunno. It was quite some weeks ago when I did performance tests, but I'll probably redo them this coming week without my previous changes, I'll report back the results.
from secor.
I ran a tiny bit of profiling last week and the rate limiting functionality didn't score high at all (though I'm fairly sure I've seen it hit higher numbers before..). The numbers were well below the time that is spent reloading the configuration file from disk, which tends to happen a lot, so I don't think the rate limiting is an issue (we set it waaaay high as we don't really want any rate limiting). Should mention that we're only running 2 threads per process, so it'll probably be different when running with ten threads.
from secor.
So from the benchmark tests I ran, the rate limiting was a big gating factor..I was able to circumvent it just by putting it to some really high number though. Didn't have to turn it off (both had almost the same effect).
from secor.
Thanks @mthssdrbrg for sharing findings from your test run.
@lefthandmagic it is expected that rate limiting will make your application slower while you are hitting the rate limit :-)
from secor.
Related Issues (20)
- Blocking Issue when running on K8s on GCP HOT 1
- manually assign avro schema to topic HOT 1
- Crashing exception after some time HOT 1
- Cannot parse timestamp with timezone HOT 1
- Racing condition with parquet writer HOT 3
- Cannot make secor work with AWS instance profile HOT 8
- Ignore parsing exceptions HOT 2
- Expose JVM metrics when using Micrometer HOT 3
- Secor uploads (different) files with the same "name" into different "days". HOT 2
- Kuberntes OOM Error. Very High memory usage > 15gb HOT 2
- Is there any alternative to AvroParquetWriter so that we can avoid using IndexedRecord? HOT 1
- ParquetWriter write method is taking too long to write single avro message HOT 1
- A Simple Study about Configuration Java Types
- PartitionFinalizer fails when there is no data produced in the topic eventhough there are other topics with data HOT 3
- Add headers while calling schema registry for decoding messages HOT 1
- Duplicate message when partition revoking or rebalancing HOT 1
- secor-monitor container not getting started. HOT 2
- Tune file size create by Secor
- Secor with zookeerper less Kafka HOT 1
- Secor filling up local disk space with intermittent data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from secor.