mrpickles / dank-dns Goto Github PK

The third way of storing the query data is by using the ELK stack. The ELK stack is the combination of Elasticsearch, Logstash, and Kibana, which conveniently happens to be optimized for big data analytics in the cloud (and DNS query processing).

Analysis - Types of Queries (TOQ)

Records the total number of each type of query sent to both the old and new IP addresses. Malformed queries (in terms of structure, not due to the use of invalid types/classes) are placed in their own category.

Time it takes to import to DB and time it takes to query

Analysis - Source Counts and Swaps (SCS)

Records the number of queries sent to both the new and old IP addresses before/after the old server started to advertise the new IP address. In addition, this notes the number of times (if any) the source switched between querying the old and new IP addresses.

In addition, records the time when a source switches from initially querying the old IP address to new one. Note that this will not count a transition such as New -> Old -> New.

Also captures unique IP addresses that queried the old server more than 3 times after the switch occured. This lower bound (3) is used to avoid priming queries, which occur under correct resolver operation.

One way of having persistent and quick-to-read query data is to put all the data into C structs which will be written to disk. If a user wants to access the data quickly for scripting/analysis purposes, the user can write code to read from disk and manually set the data as the C structs.

JavaScript native dns parser repo

In its current state, the executable processes pcaps/packets one at a time on a single thread. This is utterly unacceptable when we are crunching big data on a beefy big data machine.

We can utilize the multiple cores via concurrent pcap processing or concurrent packet processing. Doing each file concurrently is probably a simpler task, though making the concurrency work on the packet level is probably better, especially if the number of files is low.

Another method of having accessible query data is to store the query pairs into a database (probably MongoDB). Therefore, users looking to quick access to the data should simply query the database for results.

naw

MongoDB aggregation for:

• Queries per second
• Top N source IP (note: IP is stored as binary in DB)
• Top N questions (note: questions are arrays)

Analysis - Traffic Validity

Determining the amount of valid/invalid traffic for the both the new and old IP address, before and after the old server started announcing the new IP address. In addition, for any invalid traffic breaking the TLD rule (3), we keep track of the number of infractions for each unique TLD.

I am basing the rules on valid/invalid traffic from the papers "A Day at the Root of the Internet" (SIGCOMM 08) and "Wow, That's a Lot of Packets" (PAM 03). The rules are in sorted order of priority in the structure below, meaning that we attribute only 1 invalidation reason to each query.