Git Product home page Git Product logo

abuse_finder's Introduction

abuse_finder

Python library to help automatically find the most appropriate contact for abuse reports. Supports Python2.7 and Python3.6

Currently supports the following types of observables:

  • Hostnames
  • IP addresses
  • URLs
  • Email addresses

Installation

You can install the latest version using pip:

$ python -m pip install git+https://github.com/certsocietegenerale/abuse_finder.git

Or from the source code:

$ git clone https://github.com/certsocietegenerale/abuse_finder.git
$ cd abuse_finder
$ python setup.py install

Usage

This library provides a function for each kind of supported observable: domain_abuse, ip_abuse, url_abuse and email_abuse.

All functions are similar, they are taking a single argument, which is a string containing the observable's value, and return a dict containing the following elements:

  • value: value of the observable
  • names: a list of guesses for the names of the organizations that should be contacted to report abuse
  • abuse: a list of guesses of email addresses to contact to report abuse
  • raw: raw data containing information used to make theses guesses (whois / network whois)

Example

>>> from abuse_finder import domain_abuse
>>> domain_abuse('github.com')

{
  u'value': 'github.com',
  u'abuse': [u'[email protected]'],
  u'names': [u'MarkMonitor, Inc.'],
  u'raw': u"Domain Name: github.com\nRegistry Domain ID: 1264983250_DOMAIN_COM-VRSN\nRegistrar WHOIS Server: whois.markmonitor.com\nRegistrar URL: http://www.markmonitor.com\nUpdated Date: 2016-10-21T13:15:27-0700\nCreation Date: 2007-10-09T11:20:50-0700\nRegistrar Registration Expiration Date: 2020-10-09T11:20:50-0700\nRegistrar: MarkMonitor, Inc.\nRegistrar IANA ID: 292\nRegistrar Abuse Contact Email: [email protected]\nRegistrar Abuse Contact Phone: +1.2083895740\nDomain Status: clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)\nDomain Status: clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)\nDomain Status: clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)\nRegistry Registrant ID: \nRegistrant Name: GitHub Hostmaster\nRegistrant Organization: GitHub, Inc.\nRegistrant Street: 88 Colin P Kelly Jr St, \nRegistrant City: San Francisco\nRegistrant State/Province: CA\nRegistrant Postal Code: 94107\nRegistrant Country: US\nRegistrant Phone: +1.4157354488\nRegistrant Phone Ext: \nRegistrant Fax: \nRegistrant Fax Ext: \nRegistrant Email: [email protected]\nRegistry Admin ID: \nAdmin Name: GitHub Hostmaster\nAdmin Organization: GitHub, Inc.\nAdmin Street: 88 Colin P Kelly Jr St, \nAdmin City: San Francisco\nAdmin State/Province: CA\nAdmin Postal Code: 94107\nAdmin Country: US\nAdmin Phone: +1.4157354488\nAdmin Phone Ext: \nAdmin Fax: \nAdmin Fax Ext: \nAdmin Email: [email protected]\nRegistry Tech ID: \nTech Name: GitHub Hostmaster\nTech Organization: GitHub, Inc.\nTech Street: 88 Colin P Kelly Jr St, \nTech City: San Francisco\nTech State/Province: CA\nTech Postal Code: 94107\nTech Country: US\nTech Phone: +1.4157354488\nTech Phone Ext: \nTech Fax: \nTech Fax Ext: \nTech Email: [email protected]\nName Server: ns-421.awsdns-52.com.\nName Server: ns-1707.awsdns-21.co.uk.\nName Server: ns-1283.awsdns-32.org.\nName Server: ns-520.awsdns-01.net.\nDNSSEC: unsigned\nURL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/\n>>> Last update of WHOIS database: 2017-01-04T08:44:03-0800 <<<\n\nThe Data in MarkMonitor.com's WHOIS database is provided by MarkMonitor.com for\ninformation purposes, and to assist persons in obtaining information about or\nrelated to a domain name registration record.  MarkMonitor.com does not guarantee\nits accuracy.  By submitting a WHOIS query, you agree that you will use this Data\nonly for lawful purposes and that, under no circumstances will you use this Data to:\n (1) allow, enable, or otherwise support the transmission of mass unsolicited,\n     commercial advertising or solicitations via e-mail (spam); or\n (2) enable high volume, automated, electronic processes that apply to\n     MarkMonitor.com (or its systems).\nMarkMonitor.com reserves the right to modify these terms at any time.\nBy submitting this query, you agree to abide by this policy.\n\nMarkMonitor is the Global Leader in Online Brand Protection.\n\nMarkMonitor Domain Management(TM)\nMarkMonitor Brand Protection(TM)\nMarkMonitor AntiPiracy(TM)\nMarkMonitor AntiFraud(TM)\nProfessional and Managed Services\n\nVisit MarkMonitor at http://www.markmonitor.com\nContact us at +1.8007459229\nIn Europe, at +44.02032062220\n\nFor more information on Whois status codes, please visit\n https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en\n--\n\n\n\n   Domain Name: GITHUB.COM\n   Registrar: MARKMONITOR INC.\n   Sponsoring Registrar IANA ID: 292\n   Whois Server: whois.markmonitor.com\n   Referral URL: http://www.markmonitor.com\n   Name Server: NS-1283.AWSDNS-32.ORG\n   Name Server: NS-1707.AWSDNS-21.CO.UK\n   Name Server: NS-421.AWSDNS-52.COM\n   Name Server: NS-520.AWSDNS-01.NET\n   Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited\n   Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited\n   Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited\n   Updated Date: 21-oct-2016\n   Creation Date: 09-oct-2007\n   Expiration Date: 09-oct-2020",
}

Implementation

Hostnames

For hostnames, a WHOIS request will be performed and parsed using pythonwhois. The name of the registrar will be returned, as well as the email address referenced for abuse reports if any.

IP Addresses

For IP addresses, a network WHOIS is performed and parsed using ipwhois.

The names are determined by using the description field of the parsed nets. The smallest net comes first.

The email addresses are determined by counting the references to abuse on each line containing email addresses.

URLs

When using URLs:

  1. First, the URL is parsed in order to extract the registered domain, or an IP address
  2. In case of a domain, a DNS request is made in order to get IP addresses matching with the registered domain
  3. A lookup is made for every IP address, and the results are combined

Email addresses

When using email addresses:

  1. First, a domain lookup is made, but this time looking for an abuse contact provided by the registrant (looking for 'abuse' in provided email addresses).
  2. If no contact was found, do the same for every domain present in MX records.

abuse_finder's People

Contributors

augustin-fl avatar certsocietegenerale avatar gaelmuller avatar garanews avatar letmer00t avatar udgover avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

abuse_finder's Issues

Using abuse_finder behind proxy

Hi,
I'm having an issue where I'm not able to use abuse finder behind a proxy. Is there anyway to add proxy settings so that the abuse finder can reach the internet to pull down the data ?

Error when using Python 3.8

While trying to use this with the new version of TheHive/Cortex (using python 3.8, the default for Ubuntu 20), the analyzer Abuse_Finder_3_0 gives the traceback shown in the issue at:

TheHive-Project/Cortex-Analyzers#940

Also, someone else posted the issue (they are using python 3.7) at:

TheHive-Project/Cortex#352

I did not post either of those issues, however, our TheHive/Cortex displays the same traceback as at the first URL above, as we are using the same versions of the components (including Ubuntu 20).

Investigating further, I also see the same issue using the current version of the abuse_finder (directly with a small test_domain.py script I wrote, not through the analyzer), with python 3.8 on macOS:

Traceback (most recent call last): File "test_domain.py", line 1, in <module> from abuse_finder import domain_abuse File "/Users/myuser/projects/abuse_finder/abuse_finder/__init__.py", line 2, in <module> from .domain import domain_abuse File "/Users/myuser/projects/abuse_finder/abuse_finder/domain.py", line 3, in <module> from pythonwhois.net import get_whois_raw File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/__init__.py", line 1, in <module> from . import net, parse File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/parse.py", line 363, in <module> registrant_regexes = [preprocess_regex(regex) for regex in registrant_regexes] File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/parse.py", line 363, in <listcomp> registrant_regexes = [preprocess_regex(regex) for regex in registrant_regexes] File "/Users/myuser/Library/Python/3.8/lib/python/site-packages/pythonwhois/parse.py", line 205, in preprocess_regex regex = re.sub(r"\\s\*\(\?P<([^>]+)>\.\+\)", r"\s*(?P<\1>\S.*)", regex) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/re.py", line 210, in sub return _compile(pattern, flags).sub(repl, string, count) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/re.py", line 327, in _subx template = _compile_repl(template, pattern) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/re.py", line 318, in _compile_repl return sre_parse.parse_template(repl, pattern) File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/sre_parse.py", line 1042, in parse_template raise s.error('bad escape %s' % this, len(this)) re.error: bad escape \s at position 0

Looking into it a bit further, I see that the pythonwhois package (that you are using in abuse_finder) has not been maintained since 2014, and it appears that there were some changes to regex handling as of python 3.7. The pythonwhois package likely needs to be updated to handle the changes.

There is a newer package called python-whois, which was last updated in June 2020. Perhaps that is a candidate to use as a replacement.

Do you have any plans to update abuse_finder so that it works with python 3.7/3.8? We would like to use the Abuse_Finder_3_0 Cortex Analyzer with the new version of MISP/TheHive/Cortex.

Thank you.

Abuse finder return utf-8 decode error in domains

Abuse finder return message:
Could not get WHOIS for vivo.com.br ('utf8' codec can't decode byte 0xd4 in position 379: invalid continuation byte)
When domain is vivo.com.br for example, this error ocorring in many .br domains

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.