Comments (2)
I agree it's a bit confusing but it's there for a reason. I used to have the algorithm you're mentioning. meaning it did batching but no matter if the batch is full or not, flush would happen in intervals. but the performance of that hit a ceiling after 200k request per second.
At the moment my assumption is when you set up dnsmonster you know if you're in a low-volume env (flush interval) or a high-volume env (batch count).
from dnsmonster.
I find it a bit strange you get this performance issue. It is probably that when you are uploading to clickhouse you are not processing anything from the channel. Running multiple clickhouse consumer processes, I'd probably rather recommend spawning a new goroutine to handle the upload of the data at each interval so that the consumer loop continues running all the time. I'd guess this may be your bottleneck.
from dnsmonster.
Related Issues (20)
- SrcIP and DstIP in DNS_LOG table IPv4 addresses are incorrect HOT 3
- Question around pcap file behaviour HOT 12
- Potentially inaccurate packet count when reading from pcap file HOT 6
- Compilation failed on MacOS M1 HOT 1
- PCAP to JSON workflow - Split/maximum packets per output file?
- File output supporting compression HOT 2
- support for Parquet format output HOT 1
- (Illegal type DateTime64(3) [ while pushing to view default.DNS_PROTOCOL] Latest Version(v0.9.7) DNS-Monster Showing Error while send data to Click-House HOT 9
- How to do Multiple ClickHouse Ip address entry into Dnsmonster configuration file HOT 2
- how to make dns-monster listen multiple ports HOT 2
- Clickhouse Cloud HOT 3
- Decrypting DoT / DoH sessions via passive DNS approach itself HOT 1
- Clickhouse schema weirdness HOT 2
- Save dnstap `identity` as server name HOT 6
- Warning Error While logging data on DNStap socket to Click-House HOT 3
- Not logged all data from Interface Device HOT 5
- can dnsmonster handle 5,000,000 QPS? HOT 4
- Feature Request: support Kafka SASL_PLAINTEXT Protocol HOT 1
- Found wrong sql in grafana example panel.json HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dnsmonster.