hadiasghari / pyasn Goto Github PK

Is it possible to get an organization name from ASN offline?

Got AssertionError while converting rib.20141014.0600.bz2 but rib.20141006.1600.bz2 works fine.

pyasn_util_convert.py --single rib.20141014.0600.bz2 ipasn.dat
MRT RIB log importer v1.5
parse_mrt_file(): starting  parse for MrtTable {ts:1413266401, type:13, sub-type:1, data-len:762, seq:None, prefix:None}
  mrt record 100000 @10s
  mrt record 200000 @21s
  mrt record 300000 @31s
Traceback (most recent call last):
  File "bin/pyasn_util_convert.py", line 63, in <module>
    dat = mrtx.parse_mrt_file(f, print_progress=True)
  File "pyasn-1.5.0b2-py2.7-macosx-10.9-x86_64.egg/pyasn/mrtx.py", line 80, in parse_mrt_file
    results[mrt.prefix] = mrt.as_path.origin_as
  File "pyasn-1.5.0b2-py2.7-macosx-10.9-x86_64.egg/pyasn/mrtx.py", line 218, in as_path
    path = a.path_detail()
  File "pyasn-1.5.0b2-py2.7-macosx-10.9-x86_64.egg/pyasn/mrtx.py", line 352, in path_detail
    self._detail = self.BgpAttrASPath(self.data, self._is32)
  File "pyasn-1.5.0b2-py2.7-macosx-10.9-x86_64.egg/pyasn/mrtx.py", line 362, in __init__
    seg = self.BgpPathSegment(buf, is32)
  File "pyasn-1.5.0b2-py2.7-macosx-10.9-x86_64.egg/pyasn/mrtx.py", line 430, in __init__
    assert asn > 0
AssertionError

Suggested by Arman.

for-ipv4 this is: sum([2**(32-int(s.split('/')[1])) for i in prefixes])

The --latest switch downloads the RIB file for today. However, in the early hours of the day in a time zone ahead of the server, the server has not yet created any files for that day. So the script aborts with "NOT FOUND" error, instead of downloading anythig.

A simple good alternative would be to just download the last RIB file available; another is to change the time to the RouteView server's time zone, and go back a day if necessary

Got AssertionError while converting rib.20161125.0600.bz2

Traceback (most recent call last):
  File "/usr/bin/pyasn_util_convert.py", line 5, in <module>
    pkg_resources.run_script('pyasn==1.5.0b7', 'pyasn_util_convert.py')
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 461, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1194, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/lib64/python2.6/site-packages/pyasn-1.5.0b7-py2.6-linux-x86_64.egg/EGG-INFO/scripts/pyasn_util_convert.py", line 75, in <module>
    dat = mrtx.parse_mrt_file(f, print_progress=print_progress)
  File "/usr/lib64/python2.6/site-packages/pyasn-1.5.0b7-py2.6-linux-x86_64.egg/pyasn/mrtx.py", line 96, in parse_mrt_file
    assert mrt.type == mrt.TYPE_TABLE_DUMP
AssertionError

20090519.0600...
Error parsing prefix '84.205.80.0/24'

20090609.0600...
Error parsing prefix '84.205.64.0/24'

Traceback (most recent call last):
File "/home/hadi/pyasn.git/pyasn-utils/pyasn_util_convert.py", line 86, in
dat = mrtx.parse_mrt_file(f)
File "/usr/lib64/python2.6/site-packages/pyasn-1.5.0b6-py2.6-linux-x86_64.egg/pyasn/mrtx.py", line 83, in parse_mrt_file
origin = mrt.as_path.origin_as
File "/usr/lib64/python2.6/site-packages/pyasn-1.5.0b6-py2.6-linux-x86_64.egg/pyasn/mrtx.py", line 240, in as_path
path = a.path_detail()
File "/usr/lib64/python2.6/site-packages/pyasn-1.5.0b6-py2.6-linux-x86_64.egg/pyasn/mrtx.py", line 376, in path_detail
self._detail = self.BgpAttrASPath(self.data, self._is32)
File "/usr/lib64/python2.6/site-packages/pyasn-1.5.0b6-py2.6-linux-x86_64.egg/pyasn/mrtx.py", line 386, in init
seg = self.BgpPathSegment(buf, is32)
File "/usr/lib64/python2.6/site-packages/pyasn-1.5.0b6-py2.6-linux-x86_64.egg/pyasn/mrtx.py", line 457, in init
assert self.seg_type in (self.AS_SET, self.AS_SEQUENCE)
AssertionError

I am trying to convert some RIB data from WIDE: http://archive.routeviews.org/route-views.wide/bgpdata/.

I am getting this assertion error :
Traceback (most recent call last):
File "/home/jmazel/.local/bin/pyasn_util_convert.py", line 53, in
dat = mrtx.parse_mrt_file(f, print_progress=print_progress)
File "/home/jmazel/.local/lib/python2.7/site-packages/pyasn/mrtx.py", line 64, in parse_mrt_file
mrt = MrtRecord.next_dump_table_record(mrt_file)
File "/home/jmazel/.local/lib/python2.7/site-packages/pyasn/mrtx.py", line 217, in next_dump_table_record
mrt.table = MrtTableDump1(buf, mrt.sub_type)
File "/home/jmazel/.local/lib/python2.7/site-packages/pyasn/mrtx.py", line 260, in init
assert sub_type1 == MrtRecord.T1_AFI_IPv4
AssertionError

Files that generate this error are:
rib.20040701.0000.bz2
rib.20050101.0132.bz2
rib.20050701.0022.bz2
rib.20060101.0118.bz2
rib.20060701.0051.bz2
rib.20070101.0035.bz2
rib.20070701.0145.bz2
rib.20080101.0106.bz2
rib.20080701.0019.bz2
rib.20090101.0119.bz2
rib.20090701.0141.bz2
rib.20100101.0050.bz2
rib.20100701.0056.bz2

I did not find official md5 hashes of the files so I cannot check if I have the correct file.
I however downloaded all of them a second time and I got the same error.

This error may be similar to #39 although the failing assert is not the same.

import pyasn
pyasn.version
'5.0.6'

Should return 1.5.xxx not sure where this strange number is coming frame; perhaps a string/tuple/thing. To check.

.get_as_prefixes_effective() gives an exception on a non existant AS (or perhaps one with no prefixes). The desired outcome will be for it to behave similar to .get_as_prefixes() in such cases (I think returning a None)

I'm trying to setup pyasn for private BGP AS and prefix co relation. I tried tweaking the prefix database downloaded from internet by adding RFC1918 prefixes and private AS numbers.
however, it didn't work.

Is it possible to use pyasn for private BGP setup ? If yes, is there a doc available using which we can try to create a private BGP AS and Prefix database ?

Thanks
GS

Both with Python 2 and Python 3 I encounter this error with IPv6 addresses, while IPv4 works fine:

>>> db.lookup('8.8.8.8')
(15169, '8.8.8.0/24')
>>> db.lookup('2001:500:88:200::8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/pyasn/__init__.py", line 83, in lookup
    rn = self.radix.search_best(ip_address)
ValueError: inet_pton() returned error

This IPv6 address is from iana.org, so should be valid.

pyasn==1.5.0b6

Database is also fresh: rib.20150901.1400.bz2

Hi, I was wondering if it would be possible for you to make a new pypi release with the changes you've made to the master branch.

My code is currently breaking when using the current pypi release of pyasn, and it seems that the changes you've made (namely the asn ranges in asn_is_bogus) resolve the issues.

As of now, get_as_name is a "DO NOT USE YET" method. Would a implementation using http://www.cidr-report.org/as2.0/autnums.html for a lookup be good enough for it? If so, I could do it.

I have noticed that some ASNs cannot be mapped to prefixe(s).Here is the list of ASNs :
11018
12895
13063
13289
34691
36383
393223
47448
48076
48496
52334

Here is the code I have tried:

import pyasn
asndb =pyasn.pyasn("/data/data/db.rviews/ipasn_20141028.dat")
asndb.lookup('82.212.250.250')
(13289, '82.212.192.0/18')
asndb.get_as_prefixes(13289)
pyasn.version
'1.5.0b4'
asndb.get_as_prefixes('13289')

Hello!

I tried to install the package on my computer, but the install fails

(base) C:\Users\john\Documents\packages\dist\pyasn-1.6.0b1>python setup.py build
running build
running build_py
creating build
creating build\lib.win-amd64-2.7
creating build\lib.win-amd64-2.7\pyasn
copying pyasn\mrtx.py -> build\lib.win-amd64-2.7\pyasn
copying pyasn\_version.py -> build\lib.win-amd64-2.7\pyasn
copying pyasn\__init__.py -> build\lib.win-amd64-2.7\pyasn
running build_ext
building 'pyasn.pyasn_radix' extension
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
creating build\temp.win-amd64-2.7\Release\pyasn
creating build\temp.win-amd64-2.7\Release\pyasn\_radix
C:\Users\john\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\john\Documents\packages\dist\pyasn-1.6.0b1\pyasn -IC:\Users\john\Anaconda2\include -IC:\Users\john\Anaconda2\PC /Tcpyasn/pyasn_radix.c /Fobuild\temp.win-amd64-2.7\Release\pyasn/pyasn_radix.obj
pyasn_radix.c
pyasn/pyasn_radix.c(662) : error C2143: syntax error : missing ';' before 'type'
pyasn/pyasn_radix.c(663) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(663) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'
pyasn/pyasn_radix.c(663) : warning C4024: 'fgets' : different types for formal and actual parameter 1
pyasn/pyasn_radix.c(665) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(665) : error C2109: subscript requires array or pointer type
pyasn/pyasn_radix.c(665) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(665) : error C2109: subscript requires array or pointer type
pyasn/pyasn_radix.c(665) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(665) : error C2109: subscript requires array or pointer type
pyasn/pyasn_radix.c(665) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(665) : error C2109: subscript requires array or pointer type
pyasn/pyasn_radix.c(668) : error C2065: 'p1' : undeclared identifier
pyasn/pyasn_radix.c(668) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(668) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
pyasn/pyasn_radix.c(668) : warning C4024: 'strchr' : different types for formal and actual parameter 1
pyasn/pyasn_radix.c(668) : warning C4047: '=' : 'int' differs in levels of indirection from 'char *'
pyasn/pyasn_radix.c(668) : warning C4047: '==' : 'int' differs in levels of indirection from 'void *'
pyasn/pyasn_radix.c(668) : error C2065: 'p2' : undeclared identifier
pyasn/pyasn_radix.c(668) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(668) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
pyasn/pyasn_radix.c(668) : warning C4024: 'strchr' : different types for formal and actual parameter 1
pyasn/pyasn_radix.c(668) : warning C4047: '=' : 'int' differs in levels of indirection from 'char *'
pyasn/pyasn_radix.c(668) : warning C4047: '==' : 'int' differs in levels of indirection from 'void *'
pyasn/pyasn_radix.c(668) : error C2065: 'p2' : undeclared identifier
pyasn/pyasn_radix.c(668) : error C2065: 'p1' : undeclared identifier
pyasn/pyasn_radix.c(671) : error C2065: 'p1' : undeclared identifier
pyasn/pyasn_radix.c(671) : error C2100: illegal indirection
pyasn/pyasn_radix.c(671) : error C2065: 'p2' : undeclared identifier
pyasn/pyasn_radix.c(671) : error C2100: illegal indirection
pyasn/pyasn_radix.c(671) : error C2106: '=' : left operand must be l-value
pyasn/pyasn_radix.c(671) : error C2106: '=' : left operand must be l-value
pyasn/pyasn_radix.c(673) : error C2065: 'p1' : undeclared identifier
pyasn/pyasn_radix.c(673) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
pyasn/pyasn_radix.c(673) : warning C4024: 'atol' : different types for formal and actual parameter 1
pyasn/pyasn_radix.c(673) : error C2065: 'p2' : undeclared identifier
pyasn/pyasn_radix.c(673) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
pyasn/pyasn_radix.c(673) : warning C4024: 'atoi' : different types for formal and actual parameter 1
pyasn/pyasn_radix.c(673) : error C2065: 'buf' : undeclared identifier
pyasn/pyasn_radix.c(673) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
pyasn/pyasn_radix.c(673) : warning C4024: 'add_pyobject_to_radix_tree' : different types for formal and actual parameter 4
pyasn/pyasn_radix.c(713) : warning C4267: 'function' : conversion from 'size_t' to 'long', possible loss of data
error: command 'C:\\Users\\john\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe' failed with exit status 2

I am using the latest anaconda distrib and I believe I have installed all the prerequisites... Do you have any ideas?

Thanks!!

To deploy software more securely than via pip, we need packages for debian/ubuntu of used third-party libraries.

Would this be possible?

In commit 65b9576, open(..., encoding=...) was used which would not work with python < 3. Instead as suggested codecs.open() should be used, which would be consistent with python 2.7 and 3.

Suggestion for a new feature: a new feature can be added to pyasn package that returns the size of advertised ip space for each ASN ( sum of ips in the smallest prefixes of an ASN)

For a particular IP, an AS is returned that is not recognized by RIPE:

 >>> db = pyasn.pyasn('/data/data/db.rviews/ipasn_20141007.dat')
 >>> db.lookup('192.121.53.101')
(4261181720, '192.121.53.0/24')

versus:

$ whois -h whois.cymru.com " -v 192.121.53.101 2014-10-07 06:00:00 GMT"
AS      | IP               | BGP Prefix          | CC | Registry | Allocated  | Info                    | AS Name
43650   | 192.121.53.101   | 192.121.53.0/24     | SE | ripencc  | 2000-03-17 | 2014-10-07 06:00:00 GMT | SPOTIFY Spotify Technology SARL,BG

The error is in the IPASN.DAT file -- either in the conversion or the routeviews data itself; something to look into:

$ grep 192.121.53. /data/data/db.rviews/ipasn_20141007.dat
192.121.53.0/24    4261181720

It would be great to use pyasn utils, convert and download as callables. Is that a feature desired by more people?

Hello,
looks like a great tool. I was wondering if it shows the IP owner relationships (downstream/upstream provider relationships)? If anyone has an idea on how to map those relationships I would really appreciate it.
Thanks a lot

pyasn can take as input BGP route dumps from different sources. Understanding the differences between these sources can help select the default source.

• The download script currently defaults to RouteViews v4 archive.
• RouteViews also has a v4+v6 archive that might be a better default, given the increased adoption of IPv6. Interestingly enough, these dumps are typically smaller than the v4 only dump, despite having more prefixes.
• Some users have advocated using the RIPE RIS archives (http://data.ris.ripe.net/rrc12/) which are also v4+v6.

Understanding the differences between these MRT route dumps would be useful -- in terms of accuracy and completeness. For this purpose:

• A detailed-dump/debug functionality could be added to pyasn_util_convert -- to trace why the v4 only file is larger.
• Alternatively, a new script can be written to compares two generated IPASNDATs, and list IP ranges that would resolve to different ASNs.

Hello, I'm curious if there is value in adding a second .lookup signature to support integer IPs. I'm using pyasn in a Spark job, looking up a large number of IPs in a job (1B+), where passing 32bit ints around the cluster will be very beneficial instead of string IPs.

I could convert in Python before calling .lookup() but doing the int->str conversion in the underlying C would obviously be faster.

Any interest in this? If so, I'd be willing to work on a PR.

e.g., db.get_as_prefixes('1128') will return None, while db.get_as_prefixes(1128) gives the correct output. The desired behavior would be for the function to cast the input into an int. In some cases its possible; in others, it fails, and the callee has amore meaningful response

C:\Users\abcd\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.
0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\abcd\tracert\pyasn-1.6.0b1\pyasn -Ic:\Python27\include -Ic:\Python27\PC /Tcpyasn/pyasn_radix.c /Fobuild\temp.win32-2.7\Release\pyasn
/pyasn_radix.obj
pyasn_radix.c
pyasn/pyasn_radix.c(662) : error C2143: syntax error : missing ';' before 'type'

When I moved the corresponding declaration from line 662 to as the first statement within the block i.e. 656, it got compiled successfully.
char buf[512], *p1, *p2;

I realize the package needs the visual studio C++ building tools.

I - like many others - cant install it on my computer at work. I have checked on the pre-compiled binaries here https://www.lfd.uci.edu/~gohlke/pythonlibs/#wordcloud but unfortunately pyasn is not there.

Could you perhaps please provide a wheel of the package for python 3. ?
Is this possible?

Thanks!

Thank you for providing this library!

I recently decided to use this module within a project of mine in order to enrich IP addresses that I extracted from logs. During setup, I noticed that the service would be best represented as an API, so I wrapped a small Flask application around the library. I also noticed that while there are utilities to manage downloading incremental updates, there's no easy way to keep a working database up-to-date without user interaction.

To address these challenges, I've wrapped PyASN into a service called NetInfo and opensourced it. I wasn't sure if you'd want this sort of code in the repository, but if there's interest, let me know and we can work together.

Project details:
https://github.com/9b/netinfo

Write-up:
https://medium.com/@9bplus/netinfo-free-enrichment-for-ip-addresses-ecbcd62e43b6

$ pip install --upgrade pyasn
Collecting pyasn
Using cached pyasn-1.6.0b1.tar.gz
Building wheels for collected packages: pyasn
Running setup.py bdist_wheel for pyasn ... error
Complete output from command /home/ant/SopelPython3/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-35r70frq/pyasn/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmphmvt73kqpip-wheel- --python-tag cp34:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.4
creating build/lib.linux-x86_64-3.4/pyasn
copying pyasn/mrtx.py -> build/lib.linux-x86_64-3.4/pyasn
copying pyasn/init.py -> build/lib.linux-x86_64-3.4/pyasn
copying pyasn/_version.py -> build/lib.linux-x86_64-3.4/pyasn
running build_ext
building 'pyasn.pyasn_radix' extension
creating build/temp.linux-x86_64-3.4
creating build/temp.linux-x86_64-3.4/pyasn
creating build/temp.linux-x86_64-3.4/pyasn/_radix
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/pip-build-35r70frq/pyasn/pyasn -I/usr/include/python3.4m -I/home/ant/SopelPython3/include/python3.4m -c pyasn/pyasn_radix.c -o build/temp.linux-x86_64-3.4/pyasn/pyasn_radix.o
pyasn/pyasn_radix.c:39:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Failed building wheel for pyasn
Running setup.py clean for pyasn
Failed to build pyasn
Installing collected packages: pyasn
Running setup.py install for pyasn ... error
Complete output from command /home/ant/SopelPython3/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-35r70frq/pyasn/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-xjvl65cu-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ant/SopelPython3/include/site/python3.4/pyasn:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.4
creating build/lib.linux-x86_64-3.4/pyasn
copying pyasn/mrtx.py -> build/lib.linux-x86_64-3.4/pyasn
copying pyasn/init.py -> build/lib.linux-x86_64-3.4/pyasn
copying pyasn/_version.py -> build/lib.linux-x86_64-3.4/pyasn
running build_ext
building 'pyasn.pyasn_radix' extension
creating build/temp.linux-x86_64-3.4
creating build/temp.linux-x86_64-3.4/pyasn
creating build/temp.linux-x86_64-3.4/pyasn/_radix
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/pip-build-35r70frq/pyasn/pyasn -I/usr/include/python3.4m -I/home/ant/SopelPython3/include/python3.4m -c pyasn/pyasn_radix.c -o build/temp.linux-x86_64-3.4/pyasn/pyasn_radix.o
pyasn/pyasn_radix.c:39:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------

Command "/home/ant/SopelPython3/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-35r70frq/pyasn/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-xjvl65cu-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ant/SopelPython3/include/site/python3.4/pyasn" failed with error code 1 in /tmp/pip-build-35r70frq/pyasn/

Thank you in advance. :)

Hello and thanks for this really promising package!

I just wonder if the project is still alive/maintained? Some of the links on the readme do not work anymore. Also, the historical dumps seem to stop in 2015. Could you please let me know?

Thanks!

For automated adding of pull-requests.
https://travis-ci.org/
Example: https://github.com/NullHypothesis/exitmap/blob/master/.travis.yml

There are no tags indicating where the software has been released as a specific version.

It seems 0c1ae71 is 1.5.0b6, also on pypi.
af9acfb is 1.5.0b5
2d82fcb seems to be 1.5.0b4
bdf743c or 75285dd is 1.5.0b3

Hello,

I downloaded a rib file with:
pyasn_util_download.py --latest

Then I tried to convert it into a .dat with:
pyasn_util_convert.py --single rib.20160411.1400 ipasn_20160411.dat

And then it crashed with:

MRT RIB log importer 1.5.0b7
Traceback (most recent call last):
File "/home/apelegrina/.virtualenvs/test/bin/pyasn_util_convert.py", line 53, in
dat = mrtx.parse_mrt_file(f, print_progress=print_progress)
File "/home/apelegrina/.virtualenvs/test/lib/python2.7/site-packages/pyasn/mrtx.py", line 64, in parse_mrt_file
mrt = MrtRecord.next_dump_table_record(mrt_file)
File "/home/apelegrina/.virtualenvs/test/lib/python2.7/site-packages/pyasn/mrtx.py", line 208, in next_dump_table_record
buf = f.read(header_len) # read table-header
IOError: invalid data stream

Am I missing something?

Thanks

Hi,

Is it safe to assume that the data files are under a license similar to the MIT? I would like to ship one of these files with my code and would like to be compliant with the license.

Thanks.

I am trying to convert some RIB data from RIPE Otemachi (http://data.ris.ripe.net/rrc06/).

I am getting this error message:
Traceback (most recent call last):
File "pyasn_util_convert.py", line 81, in
prefixes = mrtx.parse_mrt_file(f, print_progress=not args.no_progress) # also skip-on-error=T?
File "/usr/local/lib/python2.7/dist-packages/pyasn-1.5.0b8-py2.7-linux-x86_64.egg/pyasn/mrtx.py", line 89, in parse_mrt_file
origin = mrt.get_first_origin_as()
File "/usr/local/lib/python2.7/dist-packages/pyasn-1.5.0b8-py2.7-linux-x86_64.egg/pyasn/mrtx.py", line 314, in get_first_origin_as
return path.get_origin_as()
File "/usr/local/lib/python2.7/dist-packages/pyasn-1.5.0b8-py2.7-linux-x86_64.egg/pyasn/mrtx.py", line 562, in get_origin_as
assert self.pathsegs[0].seg_type == self.BgpPathSegment.AS_SEQUENCE
IndexError: list index out of range

Files that generate this error are:
bview.20030701.0000.gz
bview.20050701.0000.gz
bview.20060701.0000.gz
bview.20070101.0000.gz
bview.20070701.0759.gz
bview.20080101.0000.gz
bview.20080701.0759.gz
bview.20090101.0759.gz
bview.20090701.0759.gz
bview.20100101.0759.gz
bview.20100701.0759.gz
bview.20110101.0000.gz
bview.20110701.0000.gz
bview.20120201.0000.gz
bview.20120701.0000.gz
bview.20130101.0000.gz
bview.20130701.0000.gz
bview.20140101.0000.gz
bview.20140701.0000.gz
bview.20150101.0000.gz

It seems that today's data of http://data.ris.ripe.net/rrc12/latest-bview.gz (permalink http://data.ris.ripe.net/rrc12/2019.10/bview.20191003.0000.gz) can't be parsed anymore. It worked at least until yesterday

> pyasn_util_convert.py --single bview.20191003.0000.gz ipasn.dat
Parsing MRT/RIB archive ..  MrtTD2Record (PEER-INDEX-TABLE, collector 1347535000, 102 peers)
  MRT record 100000 @6s
  MRT record 200000 @10s
  MRT record 300000 @13s
  MRT record 400000 @16s
  MRT record 500000 @19s
  MRT record 600000 @22s
  Exception parsing prefix record 185.206.208.42/32
Traceback (most recent call last):
  File "/usr/bin/pyasn_util_convert.py", line 71, in <module>
    skip_record_on_error=args.skip_on_error)
  File "/usr/lib64/python3.6/site-packages/pyasn/mrtx.py", line 106, in parse_mrt_file
    origin = mrt.get_first_origin_as()
  File "/usr/lib64/python3.6/site-packages/pyasn/mrtx.py", line 311, in get_first_origin_as
    return path.get_origin_as()
  File "/usr/lib64/python3.6/site-packages/pyasn/mrtx.py", line 578, in get_origin_as
    assert origin  # eventually, should not be 0 (no asn 0), or None, or an empty set
AssertionError

Currently, the download from routeviews.org happens via FTP in the pyasn_util_download.py. However, in order to use HTTP proxies (mandatory in some requirements), HTTP is preferred or only possible.

On the Pypi readme, by mistake it says db.lookup_asn(). it should be:

asndb = pyasn.pyasn('ipasn_20140513.dat')
asndb.lookup('8.8.8.8')
# should return: (15169, '8.8.8.0/24')

RIPE data is gzip-compressed, thus direct support for it would be nice.

Usage example

The tools seems to report only 1 AS out of multiple that multihomed IPs have.

E.g:
In [4]: asndb.lookup('37.209.196.7')

Out[4]: (134390, '37.209.196.0/24')

While IP2ASN

whois -h whois.cymru.com " -v 37.209.196.7 2017-06-10 00:00:00 GMT" |cut -d'|' -f1,2,3

AS | IP | BGP Prefix
18210 | 37.209.196.7 | 37.209.196.0/24
134390 | 37.209.196.7 | 37.209.196.0/24
134391 | 37.209.196.7 | 37.209.196.0/24
134396 | 37.209.196.7 | 37.209.196.0/24`

I'm writing a log parser using Apache Pig, and I'd like to use Pyasn to extract ASN information from IP addresses.

Normally, to use a Python extension in Apache Pig, you write a Python UDF and then register it in Pig "as jython"

So, this is what I did:

1. I successfully installed pyasn (in a previous try via pip-install, currently I have installed it manually, but still it doesn't work)

2. Wrote a Python UDF as follows:

#!/usr/bin/python

import sys
sys.path.append('/usr/lib64/python2.6/site-packages/')
sys.path.append('/usr/lib64/python2.6/site-packages/pyasn-1.6.0b1-py2.6-linux-x86_64.egg/')
sys.path.append('/usr/lib/python2.6/site-packages/')

import pyasn

@outputSchema("asn:chararray")
def asnLookup(ip):
	asndb = pyasn.pyasn('asn.dat')
	asn = asndb.lookup(ip)

	return asn

@outputSchema("asn_prefix:chararray")
def asnGetAsPrefixes(nbr):
	asndb = pyasn.pyasn('asn.dat')
	asn_prefix = asndb.get_as_prefixes(nbr)

	return asn_prefix

3. But when I try to register the UDF in Pig, I get the following exception:

grunt> register 'hdfs:///user/xxxxxx/LIB/PYASN/python_pyasn.py' using jython as pythonPyasn;
...
2017-11-23 17:18:13,236 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1121: Python Error. Traceback (most recent call last):
  File "/tmp/pig6864734086775637011tmp/python_pyasn.py", line 8, in <module>
    import pyasn
  File "/usr/lib64/python2.6/site-packages/pyasn-1.6.0b1-py2.6-linux-x86_64.egg/pyasn/__init__.py", line 20
SyntaxError: future feature print_function is not defined

I believe this is an incompatibility with Jython, and I've been trying extensively to make it work using a few workarounds and searching the web for possible solutions, but no success.

Any ideas?

Thanks!

add a property that returns the version (from manifest)

Pyasn provides AS1103 for IP:192.87.158.242 with prefix 192.87.0.0/16 when using ipasn_20150825.dat. This is the AS for Surfnet. Upon contacting Surfnet, they stated that they are the ISP for this range but it belongs to a customer of theirs, WKAPNET with AS1142 being the correct answer. (They aggregate it to /16 to advertise in BGP).

Team Cymru (http://asn.cymru.com/cgi-bin/whois.cgi) correctly identifies the AS; However RIPE, domaintools, maxmind and CAIDA all give the same answer as pyasn. Perhaps Team-Cymru supplements this with WHOIS registry information.

Hi,
I've got a Question concerning the License of PyASN's ancestors:
Is there any code left from the mrtd-project which was licensed under 4-Clause BSD?

Because clause 3 would be interesting to manage:

• 3. All advertising materials mentioning features or use of
• this software must display the following acknowledgement:
  *
• This product includes software developed by the University of
• Michigan, Merit Network, Inc., and their contributors.

BR dmth

So, right now, in mrtx.parse_mrt_file(..), if a single record has an error, the whole file gets skipped. I guess it would help to modify the function such that a single line error does not skip the rest of the lines.

The issue can be re-created as

$ curl -O http://data.ris.ripe.net/rrc06/2014.01/bview.20140112.1600.gz
$ gunzip bview.20140112.1600.gz
$ bzip2 bview.20140112.1600
$ pyasn_util_convert.py --single bview.20140112.1600.bz2 bview.20140112.1600.db

Investigate using Silk and pysilk with the available binary IP-to-ASN mapping for the binary representation.
http://tools.netsa.cert.org/silk-ipset/docs.html
https://tools.netsa.cert.org/silk/pysilk.html
http://routeviews-mirror.cert.org/pmap/
Specifically, use this object type to manipulate the available historical files:
https://tools.netsa.cert.org/silk/pysilk.html#PrefixMap-Object

Will need to be careful to be backwards compatible and not introduce (too many) dependencies.

This was brought to my attention by Maciej. The new pyasn is unexceptdly 15 times slower than the previous version. One guess is that it has something to do with the C compiler switches (e.g., if one is built in debug mode) when installing via PIP versus the old manual build. To check.

Here are the comparisons:
$ ipython
import PyASN
%time db1 = PyASN.new('/data/data/db.rviews/ipasn_20140101.dat')
CPU times: user 0.24 s, sys: 0.02 s, total: 0.26 s.
Wall time: 0.26 s
%timeit -n 1000 db1.Lookup('8.8.8.8')
1000 loops, best of 3: 532 ns per loop

$ ipython
import pyasn
%time db2 = pyasn.pyasn('/data/data/db.rviews/ipasn_20140101.dat')
CPU times: user 0.66 s, sys: 3.10 s, total: 3.76 s
Wall time: 3.78 s
%timeit -n 1000 db2.lookup('8.8.8.8')
1000 loops, best of 3: 10.4 us per loop

Results were obtained on Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) and Python 3.4.1 |Anaconda 2.0.1 (64-bit)| (default, May 19 2014, 13:02:41), both running on Redhat.

In a random sample of 1000 IP addresses 280 IPs had no AS name associated with the AS number after using pyasn's number->name mapping feature.

You are currently using
http://www.cidr-report.org/as2.0/autnums.html
for AS number to name mapping. That file has currently ~10k entries.

http://thyme.apnic.net/current/data-used-autnums
provides similar data (i.e. no country name) but has more entries ( ~30k entries).

How about merging these data sets and use the APNIC data for AS numbers where cidr-report has no name?

A few examples for which apnic has ASnames and cidr-report hasn't:

43711
46844
29314
34224
48031

Parameters 2&3 are dates, not 1&2; so it should be:

try:
    dt = datetime.strptime(argv[2], '%Y-%m-%d').date()
    dt_end = datetime.strptime(argv[3], '%Y-%m-%d').date()
except ValueError:

Also: add a flush() after the prints for the nohup log

This issue is a follow up on adding IPv6 functionality. The module now supports IPv6 if the IPASN file given to it has IPv6 prefixes. The pyasn_util_convert can convert/create this if the given MRT-RIB has IPv6 prefixes. pyasn_util_download currently downloads routeview-1 archives which only have IPv4, and the BGP dumps with IPv4/IPv6 prefixes to be downloaded manually (see: www.routeviews.org). It might be good to add an option to add files routeviews-4. The prerequisite is to understand the differences between the routes advertised in these different dumps more clearly to make sure changing the input has no adverse effects.