Git Product home page Git Product logo

glastopf's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

glastopf's Issues

Analysis framework for the centrally collected sensor data

Goal is to leverage collected information to improve the honeypots classification, attack surface generation and emulation capabilities.

We don't really analyze data on the fly right now. Having this would put us in a position from which we can change the honeypots behavior based on the attacks we collect.

This will help us to spot trends and automatically react on it.
This allows rapid modification of the honeypot emulation on a global scale based on an event collected by a single sensor.

Continuous integration

We could benefit from continuous integration, especially to test out the various supported database types in my "soon to come" pull request.
Lukas, if you enable the travis-ci hooks for the glastopf project i will try to set it up.

Unit testing: test_dork_links

Pretty close to closing the sqlalchemy deal, but battling with a few unit tests. Is the unit test below supposed to pass at the moment?
As far as i understand the test is supposed to query the dork database, but at the moment it is querying the events db.

https://github.com/glastopf/glastopf/blob/master/testing/test_dorks.py:

def test_dork_links(self):
        """Objective: Test if a random link from the dork page exists in the database.
        Input: A random link from a created dork page.
        Expected Results: The path of the link should be at least once in the db.
        Notes: Links have the parameters truncated, so multiple entries are likely."""
        dirname = 'modules/handlers/emulators/dork_list/pages/'
        sample_file = choice(gen_dork_list.get_old_dork_pages_list(dirname))
        print "Randomly selected dork page:", sample_file.rsplit('/', 1)[1]
        with open(sample_file, 'r') as sample_data:
            data = fromstring(sample_data)
        links = data.cssselect('a')
        test_link_path = choice(links).get('href')
        print "Randomly selected path:", test_link_path
        data = self.db.select_entry(test_link_path) #Problem here? Supposed to query dork.db instead?
        print "Done searching for the entry."
        self.assertTrue(len(data) > 0)
        print "The dork db returned:",
        print str(len(data)), "entries,",
        print "which equates our expectation."

OperationalError crashes Database logging.

This can be triggered sometimes by issuing requests when the Glastopf is being set up.
I am using "39ea1e6a3236e6e" version.

After this, database logging gets crashed, although the webserver runs OK. I used sqlite3 for this.

2013-01-24 05:23:40,685 (glastopf) 192.168.1.107 requested GET / on 192.168.1.109:8080
Exception in thread Thread-2:
Traceback (most recent call last):
 File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()  
File "/usr/lib/python2.7/threading.py", line 505, in run    self.__target(*self.__args, **self.__kwargs)  
File "/home/phani/glastopf/glastopf.py", line 202, in post_processer    self.maindb.insert(attack_event)  
File "/home/phani/glastopf/modules/reporting/main/log_sql.py", line 51, in insert    conn.execute(self.events_table.insert(entry))  
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1194, in execute    params)  
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1274, in _execute_clauseelement    return self.__execute_context(context)  
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1305, in __execute_context    context.parameters[0], context=context)  
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1404, in _cursor_execute    context)  
File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1397, in _cursor_execute    context)
 File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 299, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (OperationalError) database is locked u'INSERT INTO events (time, source, request_method, request_url, request_parameters, request_version, request_header, req
uest_body, pattern, filename, response) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('2013-01-24 05:23:35', '192.168.1.107:56271', 'GET', '/', '', 'HTTP/1.1', '{"Accept-Language":
 "en-US,en;q=0.8", "Accept-Encoding": .........

Use logging instead of print

Various print statements are spread around the code base. These statements ought to be converted to use the logging module instead.
Which loglevels goes to console could then be configured in glastopf.cfg/glastopf.py.

Extract dorks from mnemosyne webservice

This is the initial part of bootstrapping glastopf as discussed in #29. The initial work will include basic extraction of dorks from Mnemosyne.
Glastopf will still support the traditional way of generating and extracting dorks.

Kill the webserver with a few request

Request:
http://192.168.9.34/index.php?id=%27%20%3D%27 (resend request a couple of times, less then 5 should do in order to kill the webserver.

Error:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/init.py", line 48, in loop
else: l.start()
OSError: [Errno 9] Bad file descriptor: '<pyev.Io object at 0x266ae20> has been stopped'
oserror [Errno 9] Bad file descriptor: '<pyev.Io object at 0x266ae20> has been stopped' (9, 'Bad file descriptor')

More lightweight testing setup

At the moment some tests takes very long time to complete. I think that development in general would improve if we could run through all testcases quickly.
Is it really necessary that we use a input set of around 12k of entries?

Table events no column time

After merging with the sqlalchemy patch I got the following error:

File log_sql.py:

WARNING:root:Error caught while inserting 1 events into SQL, will retry in 1 seconds. ((OperationalError) table events has no column named time u'INSERT INTO events (time, source, request_method, request_url, request_parameters, request_version, request_header, request_body, pattern, filename, response) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('2012-12-08 10:32:57', '68.235.36.193:52957', 'GET', '/guestbook.pl', '', 'HTTP/1.0', [...]

Make profiler detect attack tools

Most basic attack and recon tools (sqlmap, havij, nikto, metasploit, etc) attacks in a pretty detectable way. (most of the times anyways)
Capability should be added to the profiler to guess attack tool.

Make Glastopf $exploit_tool_name exploitable.

Randomize the virtualdocs files

The files in the virtualdocs directory are static and very easy to fingerprint. Might be useful to create them randomized on first start-up of Glastopf.

The required version of distribute (>=0.6.35) is not available

Downloading/unpacking glastopf
Downloading Glastopf-3.0.0.tar.gz (531Kb): 531Kb downloaded
Running setup.py egg_info for package glastopf
The required version of distribute (>=0.6.35) is not available,
and can't be installed while this script is running. Please
install a more recent version first, using
'easy_install -U distribute'.
(Currently using distribute 0.6.24dev-r0 (/usr/lib/python2.7/dist-packages))
Complete output from command python setup.py egg_info:
The required version of distribute (>=0.6.35) is not available,

Permission error

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 121, in regular_generate_dork
    self.generate_dork_pages()
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 98, in generate_dork_pages
    with codecs.open("{0}/{1}".format(self.pages_path, page_md5), "w", "utf-8") as dork_file:
  File "/usr/lib/python2.7/codecs.py", line 881, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 13] Permission denied: '/opt/honeypot/data/dork_pages/942af692e6b84eae76b8f54b9da9219b'

dorks.db

Get this message when using the dorks.db:

Traceback (most recent call last):
File "/usr/local/openhoney/glastopf/modules/handlers/emulators/dork_list/dork_db.py", line 57, in trueInsert
self.cursor.execute(sql, (content, 1, datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")))
OperationalError: table intext has 5 columns but 4 values were supplied
2013-01-18 11:20:17,166 (modules.handlers.emulators.dork_list.dork_db) Error while inserting into dork_db: table intext has 5 columns but 4 values were supplied

Using the dorks.db from http://glastopf.org/dork.db.tar.

Test not passing: Emulator testing for PHP CGI remote code execution CVE-2012-1823

Test is not passing at the moment.

======================================================================
FAIL: Objective: Emulator testing for PHP CGI remote code execution CVE-2012-1823
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jkv/glas-dev/glastopf/testing/test_emulators.py", line 232, in test_phpcgi_rce_emulator
    self.assertTrue("""testing""" == self.event.response)
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
Successfully parsed with sandbox
Return value: 

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------

Error generating dork pages: got an unexpected keyword argument 'token_pattern'

Running : 'python webserver.py' gives me this error:

2013-02-22 08:14:11,674 (glastopf) Generating initial dork pages - this can take a while.
Traceback (most recent call last):
  File "webserver.py", line 111, in <module>
    a = WebSockListener(host, port)
  File "webserver.py", line 38, in __init__
    self.glastopf_honeypot = glastopf.GlastopfHoneypot()
  File "/home/ubuntu/glastopf/glastopf.py", line 85, in __init__
    self.dork_generator.regular_generate_dork(0)
  File "/home/ubuntu/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 116, in regular_generate_dork
    self.generate_dork_pages()
  File "/home/ubuntu/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 74, in generate_dork_pages
    clusters = self.clustere.cluster(inurl_list)
  File "/home/ubuntu/glastopf/modules/handlers/emulators/dork_list/cluster.py", line 31, in cluster
    vectorizer = CountVectorizer(preprocessor=self.preprocessor, token_pattern=self.pattern, min_df=self.min_df)
TypeError: __init__() got an unexpected keyword argument 'token_pattern'

AttributeError: 'NoneType' object has no attribute 'group'

Request:

http://192.168.9.34/index.php?id=%0D%0AH%

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/__init__.py", line 566, in _readable
    self._event('read', data)
  File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/util.py", line 16, in _event
    cb(*args)
  File "webserver.py", line 63, in read
    response = self.glastopf_honeypot.handle_request(d, self.addr, self.c)
  File "/opt/glastopf/glastopf.py", line 108, in handle_request
    attack_event.parsed_request = self.HTTP_parser.parse_request(raw_request)
  File "/opt/glastopf/modules/HTTP/util.py", line 91, in parse_request
    parsed_request.method = re_grp.group(1)
AttributeError: 'NoneType' object has no attribute 'group'

No such file or directory: '/tomcat/manager_status.html'

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/__init__.py", line 561, in _readable
    self._event('read', data)
  File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/util.py", line 16, in _event
    cb(*args)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/EGG-INFO/scripts/glastopf-runner.py", line 67, in read
    response = self.glastopf_honeypot.handle_request(d, self.addr, self.c)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/glastopf.py", line 264, in handle_request
    emulator.handle(attack_event)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/tomcat_status.py", line 27, in handle
    with open(tomcat_manager_path, 'r') as tomcat_manager_file:
IOError: [Errno 2] No such file or directory: '/tomcat/manager_status.html'

"Outsource" the PHP sandbox

The PHP sandbox is currently a for of https://github.com/glastopf/phpox which some features missing. We want to remove the PHP sandbox from Glastopf in the future and install it externally as a module or run it as a service.

We could also run a central sandbox which should reduce processing time significantly if your sensor has a decent connection as we don't have to run the sample twice and can send the sensor historic data. This can be easily achieved if the sandbox is deployed as a service.

sql test error

Seems like the SQL test makes the test process hung. Ctrl+C causes the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 295, in _exit_function
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/glastopf/modules/reporting/log_sql.py", line 89, in consumer
    time.sleep(self.wait_seconds)
KeyboardInterrupt
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 148, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 133, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 295, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 148, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 133, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

create setup.py

Install dependencies and various setup tasks. Maybe even install with virtualenv for required modules.

"Fingerprintability"

We should start to make the honeypot less obvious to detect. This will be part of the new attack surface creation. So with a more variable layout/design/etc. you will not be able to "see" the honeypot.

I'm not a big fan to ship a dork collection with the honeypot. We might want to ship some for bootstrapping and then remove them as soon as the honeypot is collecting attacks. But what Google has seen, Google will remember so this might cause issues.

Integrating additional dorks from a exploit database is great but could make us vulnerable for detection.

Profiler logging is broken

There are no supported loggers in the profiler. Besides traditional logging, it might be usefull to transmit this information on a hpfeeds channel.

rename aux to auxiliary

aux is a reserved directory name which cannot be used in windows. modules/reporting/aux must be reanmed to modules/reporting/auxiliary

Permission denied to bind to port 80

Haven't tried this on a clean machine but it seems like we are dropping privileges before binding the WSGI server to port 80. This causes a permission error...

Handle malformed HTTP requests

We don't handle malformed requests properly (basically the parser just fails and the exception is not handled). We should raise a proper exception and return an error code to the client.

See also #33

Failure during dork database bootstrapping

I've got a relatively fresh install of Ubuntu 12.04 and followed the glastopf Ubuntu setup guide. When I try to start up glastopf, I get an error right after the message about bootstrapping the dork database. Here is the output.

:/opt/glastopf$ sudo python webserver.py
2013-02-27 12:29:35,272 (root) Webserver running on: 0.0.0.0:8080 waiting for connections.
2013-02-27 12:29:35,273 (glastopf) Starting Glastopf (Git: 42529b4)
2013-02-27 12:29:35,274 (glastopf) Connecting to main database with: sqlite:///db/glastopf.db
2013-02-27 12:29:35,286 (modules.handlers.emulators.dork_list.dork_page_generator) Bootstrapping dork database.

Traceback (most recent call last):
  File "webserver.py", line 111, in <module>
    a = WebSockListener(host, port)
  File "webserver.py", line 38, in __init__
    self.glastopf_honeypot = glastopf.GlastopfHoneypot()
  File "/opt/glastopf/glastopf.py", line 78, in __init__
    self.dork_generator = self.setup_dork_generator(conf_parser)
  File "/opt/glastopf/glastopf.py", line 124, in setup_dork_generator
    mnem_service_instance=mnemosyne_service)
  File "/opt/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 53, in __init__
    self.bootstrap_dorkdb()
  File "/opt/glastopf/modules/handlers/emulators/dork_list/dork_page_generator.py", line 141, in bootstrap_dorkdb
    dorks = self.mnem_service.get_dorks()
  File "/opt/glastopf/modules/handlers/emulators/dork_list/mnem_service.py", line 36, in get_dorks
    response = sess.post(base_url + '/login', payload, timeout=timeout, verify=False)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 241, in post
    return self.request('post', url, data=data, **kwargs)
TypeError: request() got an unexpected keyword argument 'verify'

Path to application mapping

It would be useful to have a request path -> web application mapping at least for the major web applications. E.g. wp-content belongs to WordPress. This way we can customize the honeypots response (server a WordPress template instead of the default one).

Pymongo Import Error

Python version is 2.7.1 on an Ubuntu 11.04
pymongo version: 1.8.1

I get the following error:

Traceback (most recent call last):
File "webserver.py", line 11, in
import glastopf
File "/home/phani/glastopf/glastopf.py", line 38, in
from modules.handlers.emulators.dork_list import database_mongo
File "/home/phani/glastopf/modules/handlers/emulators/dork_list/database_mongo.py", line 19, in
from pymongo import MongoClient, uri_parser
ImportError: cannot import name MongoClient

No such file or directory: '/opt/honeypot/data/comments.txt'

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/__init__.py", line 561, in _readable
    self._event('read', data)
  File "/usr/local/lib/python2.7/dist-packages/evnet-1.0_5-py2.7.egg/evnet/util.py", line 16, in _event
    cb(*args)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/EGG-INFO/scripts/glastopf-runner.py", line 67, in read
    response = self.glastopf_honeypot.handle_request(d, self.addr, self.c)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/glastopf.py", line 264, in handle_request
    emulator.handle(attack_event)
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/handlers/emulators/comments.py", line 41, in handle
    with codecs.open(os.path.join(self.data_dir, 'comments.txt'), "r", "utf-8") as comments_txt:
  File "/usr/lib/python2.7/codecs.py", line 881, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: '/opt/honeypot/data/comments.txt'

RFI emulator throws error if inject file/site is unavailable

Minor issue .. but here goes ..

Request:
GET /cgi-bin/index.php?filepath=http://XXXXXXXXXX on 192.168.7.34

Response:
2013-01-19 23:20:39,413 (modules.handlers.emulators.rfi) Failed to fetch injected file, I/O error:

Traceback (most recent call last):
  File "/opt/glastopf/modules/handlers/emulators/rfi.py", line 56, in download_file
    injected_file = urllib2.urlopen(req, timeout=4).read()
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 401, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 419, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open
    raise URLError(err)
URLError: <urlopen error timed out>

No such file or directory: 'modules/handlers/emulators/dork_list/pages'

2013-03-01 19:50:08,293 (glastopf.modules.privileges) Could not change file owner: [Errno 2] No such file or directory: 'modules/handlers/emulators/dork_list/pages'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Glastopf-3.0.0-py2.7.egg/glastopf/modules/privileges.py", line 34, in drop
    run_uid, run_gid)
OSError: [Errno 2] No such file or directory: 'modules/handlers/emulators/dork_list/pages'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.