[Feature Suggestion] IPinfo's free IP database - IP to Country ASN about sniffnet HOT 11 CLOSED

I thank you again very much for your proposal and assistance in providing such a detailed ticket.

However, as anticipated I think that #365 is the best tradeoff between complexity and completeness.

from sniffnet.

@GyulyVGC Thank you very much again for considering us. I totally understand.

We will be cheering on from the sidelines :) Feel free to ping us anytime in the future, it will be an honor to help.

from sniffnet.

Hi @abdullahdevrel first of all thanks very much for the very detailed and precise report, I really appreciate that.

Some concerns I'm noticing:

• the size of the DB is a bit high (if I'm not wrong it's around 47 MB) and this would negatively impact on the packed binary size. I know that probably the size overhead wrt the DB I'm currently using is due to the presence of more entries/major accuracy, but it still feels a bit too much.
• Even if the DB is updated daily, I'm currently distributing the MMDBs embedded into the binary, so given a version of Sniffnet the MMDB file wouldn't be updated anyway.

from sniffnet.

I really appreciate you taking a look, @GyulyVGC

Would it be possible for us to write a PR for the project that implements a database download mechanism during the installation process?

This will be a great feature because:

• It will enable you to ship smaller binaries without packaging a database in the project.
• Users will get full accuracy and updated data routinely if they update the IP database themselves.
• It will simplify code maintenance issues as the IP database will be downloaded from the client side and the update mechanism will be done by the user.

The plan I would suggest:

• During the installation, the database from IPinfo gets downloaded.
• During the installation process, we mention where the IP database is stored by mentioning the file path to the DB.
• The user is provided some level of documentation so they can update the database routinely. The file path to the IP database should be transparent and it is just a file they can overwrite to keep the IP data fresh.

Downloading the IP database is as simple as making a curl/wget call (documentation):

curl -L https://ipinfo.io/data/free/country_asn.mmdb?token=<YOUR_TOKEN> -o country_asn.mmdb

Addressing your concerns

wrt the DB I'm currently using is due to the presence of more entries/major accuracy, but it still feels a bit too much.

The cost of high accuracy is that we are including more granular information, and thus, the size of the binary is relatively large.

The current two databases that you use account for a total of ~15 MB. While our database is significantly bigger.

The size discrepancy is due to IP range aggregation. IP range aggregation works by clustering IP ranges ( for example, /24) into one geolocation. So, even if IP addresses of an IP address range fall into multiple countries, by doing IP address range clustering, providers simply pick the majority location of each IP address and assign it to the entire range. That is an accuracy compromise. This creates a new dimension of the problem, as range clustering does not account for IP corrections on an individual level.

We do not do range clustering; the database is fully accurate even though it is free. IP range granularity goes to even one IP address level (/32 CIDR range).

Example:

SELECT *
FROM (
    SELECT *
    FROM ip_country_asn
    WHERE start_ip=end_ip
)
SAMPLE (5 ROWS)

Now, if we implement an IP database download feature, you can remove the database packaged with the binary, which accounts for 15 MB.

Then, when the user installs the program, they can download the database directly from us after installation. Moreover, if we keep the downloaded database path transparent, users can regularly download our database using their own free token to keep the database updated. They can write a cronjob of sort that overwrites the IP database on a regular schedule.

MMDBs embedded into the binary, so given a version of Sniffnet the MMDB file wouldn't be updated anyway.

I hope that you will consider implementing the download mechanism. The parent databases of the IP to Country ASN database are ASN database and IP to Geolocation database.

• The ASN database has changed by 0.116% since yesterday.
• The IP to Geolocation database changed by 0.004% since yesterday.

Considering the daily changes, these updates can add up. Sniffnet deals with sensitive information, so the project would greatly benefit from the increased accuracy provided by our database.

Please let me know what you think.

from sniffnet.

While I got your point and actually the solution could be beneficial under certain point of views, I still prefer bundling the DBs to keep things easy for the average user.
I think the normal user is not interested in having such a fine grained granularity and I don't want to leave the burden to create a token and install in a pre-defined path.

A better solution is an hybrid one: in #365 I'm already implementing the possibility to choose a custom MMDB file for users who have a commercial database or want more precision, otherwise the embedded one will be used.
In this way, the average user can enjoy the app without having to worry about further installations/configurations, while more advanced users can download the MMDB file they prefer and later set the path to it via the new Advanced settings.

from sniffnet.

@GyulyVGC How about integrating the access token into Sniffnet directly?

With the Sniffnet IPinfo access token, the IP database will be downloaded directly. Users don't have to set up a configuration file or provide their own access token. The data download mechanism will use the project's own access token.

from sniffnet.

Hey @abdullahdevrel I was wondering if there could be a chance for a sort of collaboration between IPinfo and Sniffnet.

#365 was merged some days ago and I thought it can be nice to include in Sniffnet's settings a link to the IPinfo website.

This would be beneficial for:

• my users to know where to get MMDB resources
• for IPinfo itself, since it'd be a sort of promoting chance to acquire new customers and to be highlighted as one of the leaders in this sector

Furthermore, if you are willing to start this collaboration, I'll be more than happy to highlight IPinfo as one of Sniffnet's sponsors in the README as well as the official website of the project.

I plan to release the next version of the app in a couple of months, so feel free to take your time to come to a decision 😃

from sniffnet.

Hey @GyulyVGC,

Apologies for the late response. That is a fantastic proposal. I have emailed you (gyul****@gmail.com) to discuss the suggestion further.

Thanks!

from sniffnet.

Hey @abdullahdevrel I'm super happy you find the proposal interesting!

I've just replied to your email.

I remain open for further discussion both here or via email.

from sniffnet.