Git Product home page Git Product logo

d4-project / ipasn-history Goto Github PK

View Code? Open in Web Editor NEW
88.0 8.0 11.0 668 KB

IP ASN History to find ASN announcing an IP and the closest prefix announcing it at a specific date

Home Page: https://www.d4-project.org/open%20source/asn/history/2019/01/09/IP-ASN-History.html

License: GNU Affero General Public License v3.0

Python 99.45% Shell 0.55%
ip-address security-tools historical-data ip-asn-history ip-whois d4-project

ipasn-history's Introduction

IP ASN History

IP ASN History is an open source project to figure out the ASN announcing an IP and the closest prefix announcing it at a specific date.

The default currently loads announces from CAIDA also RIPE dumps in MRT format.

IP ASN History is a complete open source project to run your own server locally.

If you don't want to run your own server, there is a public API operated by CIRCL.

Python client

See the PyIPASNHistory repository

API

The REST API has two entry points:

  • /meta (GET): returns meta informations about the information currently stored in the database

     {
       "sources": [
         "caida"
       ],
       "expected_interval": {
         "first": "2018-05-12",
         "last": "2018-11-08"
       },
       "cached_dates": {
         "caida": {
           "v4": {
     	"cached": [
     	  "2018-04-26T12:00:00",
     	  "2018-04-27T12:00:00",
     		  //...
     	  "2018-11-05T12:00:00",
     	  "2018-11-06T12:00:00"
     	],
     	"missing": [
     	  "2018-11-07"
     	],
     	"percent": 99.44444444444444
           },
           "v6": {
     	"cached": [
     	  "2018-04-26T12:00:00",
     	  "2018-04-27T12:00:00",
     		  //...
     	  "2018-11-06T12:00:00",
     	  "2018-11-07T12:00:00"
     	],
     	"missing": [],
     	"percent": 100.0
           }
         }
       }
     }

Note: the percentage will help 3rd party component to decide if they should query the service now or wait. It is expected to miss a few days and probably not important.

  • / (POST/GET): Runs a query. (if you want to resolve more than 1000 IPs, use mass_cache and mass_query)

    Parameters:

    • ip: (required) IP to lookup
    • source: (optional) Source to query (defaults to 'caida') - currently, only caida is supported
    • address_family: (optional) v4 or v6 (defaults to v4)
    • date: (optional) Exact date to lookup (defaults to most recent available)
    • first: (optional) First date in the interval
    • last: (optional) Last date in the interval
    • precision_delta: (optional) Max delta allowed between the date queried and the one we have in the database. Expects a dictionary to pass to timedelta. Example: {days=1, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0}

    Response:

     {
       "meta": {
         "source": "caida",
         "ip_version": "v4",
         "ip": "146.185.222.49"
       },
       "error": "Optional, only if there was an error",
       "info": "Optional, informational message if needed",
       "response": {
         "2018-11-01T12:00:00": {
           "asn": "44050",
           "prefix": "146.185.222.0/24"
         },
         "2018-11-02T16:00:00": {
           "asn": "44050",
           "prefix": "146.185.222.0/24"
         },
         "2018-11-03T12:00:00": {
           "asn": "44050",
           "prefix": "146.185.222.0/24"
         },
         "2018-11-04T12:00:00": {
           "asn": "44050",
           "prefix": "146.185.222.0/24"
         },
         "2018-11-05T12:00:00": {
           "asn": "44050",
           "prefix": "146.185.222.0/24"
         },
         "2018-11-06T12:00:00": {
           "asn": "44050",
           "prefix": "146.185.222.0/24"
         }
       }
     }

    Curl examples:

    It works fine for single requests, if you have batches of IPs to lookup, use the python API and mass_cache mass_query.

    curl https://bgpranking-ng.circl.lu/ipasn_history/?ip=8.8.8.8
    curl "https://bgpranking-ng.circl.lu/ipasn_history/?ip=2001:db8:a0b:12f0::1&address_family=v6"

    Note: The date returned may differ from the one queried: the system will figure out the closest available date to the one queried.

  • /mass_cache (POST): Caches a lot of queries at once, don't wait for the lookup.

    Parameters: A list of dictionaries with the same parameters as the default query.

    Response: A list of queries that IPASN History wasn't able to process.

    Note: Use this path when you have lots of query to run and (>1000) in order to resolve all of them at once.

  • /mass_query (POST): Caches a lot of queries at once. Either wait for the lookup to be done, or pick the data from cache.

    Parameters: A list of dictionaries with the same parameters as the default query.

    Response: A list of responses as the default query.

    Note: Use this path when you have lots of query to run and (>1000) in order to resolve all of them at once.

  • /asn_meta (POST): Returns meta informations about an ASN

    Parameters: A list of dictionaries with the same parameters as the default query.

    Response

    {
      "meta": {
      "address_family": "v4",
      "asn": "137342",
      "source": "caida"
      },
      "response": {
        "2019-01-01T12:00:00": {
          "137342": {
            "ipcount": "512",
            "prefixes": [
              "180.214.250.0/24",
              "103.113.3.0/24"
            ]
          }
        }
      }
    }

Installation

IMPORTANT: Use poetry

NOTE: Yes, it requires python3.8+. No, it will never support anything older.

Install redis

git clone https://github.com/antirez/redis.git
cd redis
git checkout 7.0
make
make test
cd ..

Note: If it fails, have a look at the documentation.

Install kvrocks

git clone https://github.com/KvrocksLabs/kvrocks.git
cd kvrocks
git checkout 2.5
./x.py build
cd ..

Note: If it fails, have a look at the documentation.

Install & run IP ASN History

git clone https://github.com/D4-project/IPASN-History.git
cd IPASN-History
poetry install
echo IPASNHISTORY_HOME="'`pwd`'" > .env
poetry shell
start

Shutdown IP ASN History

stop

(Optional) Build & install bgpdumpy, required to process dumps from RIPE

sudo apt install ca-certificates gcc make autoconf libbz2-dev zlib1g-dev libffi-dev
poetry install -E ripe

ipasn-history's People

Contributors

aaronkaplan avatar adulau avatar dependabot[bot] avatar fukusuket avatar rafiot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ipasn-history's Issues

Automatically guess ipv4/ipv6

2 options:

  1. On client side

    • Advantage: the responsibility is put on the client to figure out the data they send
    • Disadvantage: The client has to to figure it out somehow. People will tell the tool isn't working because they send ipv6 and the system defaults to ipv4. So it returns nothing.
  2. in the lookup process

    • Advantage: easier for the client
    • Disadvantage: May have an impact on performances if we use the ipaddress module. Need to figure it out.

Current approach: in the python module on client side.

Query recursively all sources, find best match

Right now, it is up to the user to query sources loaded in the database. It would be nice to have it done automatically on the backend:

  1. trigger a query against all the sources in the system
  2. find the smallest prefix announcing this IP
  3. return that

mypy cache issue in the github actions

Hi, when merging in #18 you will see that the github actions break in mypy.

I did a bit of research and it seems that this error is discussed also here python/mypy#2960.
One trick could be to pass the parameter --cache-dir=/dev/null to mypy.

You might want to consider this.

Unable to add `ASN` data to `kvrocks` during initial installation

Hello :)
I encountered a minor error while initial installation, so I will report it. (Sorry if it doesn't support the latest kvrocks yet.)

Describe the issue
Unable to add ASN data to kvrocks during initial installation with latest kvrocks

Step to Reproduce

  1. Install according to the following README and start
    https://github.com/D4-project/IPASN-History#installation
  2. kvrocks builds with x.py as follows
    #27

Actual behavior
Unable to add ASN data to kvrocks due to following error.

2023-08-10 00:55:05,504 RipeDownloader INFO:Unreachable: http://data.ris.ripe.net/rrc00/2023.08/bview.20230810.0000.gz
2023-08-10 00:55:05,510 RipeDownloader INFO:Launching RipeDownloader
2023-08-10 00:55:05,513 RipeDownloader INFO:New file to download: rrc00/2023.08/bview.20230810.0000.gz
2023-08-10 00:55:05,650 CaidaDownloader INFO:Launching CaidaDownloader
2023-08-10 00:55:06,876 RipeDownloader INFO:Unreachable: http://data.ris.ripe.net/rrc00/2023.08/bview.20230810.0000.gz
2023-08-10 00:55:33,333 CaidaLoader ERROR:Something went terribly wrong in CaidaLoader.
Traceback (most recent call last):
  File "/home/fukusuke/IPASN-History/ipasnhistory/default/abstractmanager.py", line 138, in run
    self._to_run_forever()
  File "/home/fukusuke/IPASN-History/bin/caida_loader.py", line 33, in _to_run_forever
    self.load_all()
  File "/home/fukusuke/IPASN-History/bin/caida_loader.py", line 53, in load_all
    if self.already_loaded(address_family, date):
  File "/home/fukusuke/IPASN-History/bin/caida_loader.py", line 36, in already_loaded
    return self.storagedb.sismember(f'{self.key_prefix}|{address_family}|dates', date)
  File "/home/fukusuke/.cache/pypoetry/virtualenvs/ipasnhistory-lVZCKYoT-py3.10/lib/python3.10/site-packages/redis/commands/core.py", line 3374, in sismember
    return self.execute_command("SISMEMBER", name, value)
  File "/home/fukusuke/.cache/pypoetry/virtualenvs/ipasnhistory-lVZCKYoT-py3.10/lib/python3.10/site-packages/redis/client.py", line 1269, in execute_command
    return conn.retry.call_with_retry(
  File "/home/fukusuke/.cache/pypoetry/virtualenvs/ipasnhistory-lVZCKYoT-py3.10/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/fukusuke/.cache/pypoetry/virtualenvs/ipasnhistory-lVZCKYoT-py3.10/lib/python3.10/site-packages/redis/client.py", line 1270, in <lambda>
    lambda: self._send_command_parse_response(
  File "/home/fukusuke/.cache/pypoetry/virtualenvs/ipasnhistory-lVZCKYoT-py3.10/lib/python3.10/site-packages/redis/client.py", line 1246, in _send_command_parse_response
    return self.parse_response(conn, command_name, **options)
  File "/home/fukusuke/.cache/pypoetry/virtualenvs/ipasnhistory-lVZCKYoT-py3.10/lib/python3.10/site-packages/redis/client.py", line 1286, in parse_response
    response = connection.read_response()
  File "/home/fukusuke/.cache/pypoetry/virtualenvs/ipasnhistory-lVZCKYoT-py3.10/lib/python3.10/site-packages/redis/connection.py", line 905, in read_response
    raise response
redis.exceptions.ResponseError: NotFound:

Expected behavior
Able to add ASN data to kvrocks

Environment
This issue occurred in the following environment.

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ python3 -V
Python 3.10.12
$ ./kvrocks -v
kvrocks unstable (commit 4fe7fb0)
$ ./redis-server -v
Redis server v=6.2.13 sha=24c0bf5c:0 malloc=jemalloc-5.1.0 bits=64 build=b5e184f9aef85bc1

Additional context
I checked the response to kvrocks sismember, it was the following result.
(In the data below, caida|v6|dates are already registered because I applied fix #29 before check)

$ redis-cli -p 5177
127.0.0.1:5177> sismember caida|v6|dates 2023-08-08T10:00:00
(integer) 1
127.0.0.1:5177> sismember caida|v6|dates 9999-01-01T00:00:00
(integer) 0
127.0.0.1:5177> sismember foo 2023-08-08T10:00:00
(error) ERR NotFound:
127.0.0.1:5177> sismember caida|v6|dates foo
(integer) 0
127.0.0.1:5177>

It seems to return an ERR NotFound instead of 0 when the key doesn't exist yet.

This seems to cause an exception on the line below.
https://github.com/D4-project/IPASN-History/blob/main/bin/caida_loader.py#L36

wishlist

  • UI
  • look at it again, update dependcies, framework, etc. refactor
  • clean up
  • differences in data feeds. Are there diffs?
  • exporting the client to a dedicated repo
  • make it stream ready / make an enricher
  • ardb --> replace by a new redis-interface compliant DB. ask @adulau which one that is
  • make the places of the respective databases configurable (example: kvrock's db/ dir, redis's DB on disk). Why? Because I would like to put those on an LVM logical volume, which I may expand if needed.
  • make the PROD version dockerized

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.