The csirtg-fm-v1 from csirtgadgets

problem: fetchers are not pluggable

make smrt rules able to point to a plugin (drop in name space or a script on the system) for fetching

problem: timeout issue

2018-09-28 14:40:12,780 - INFO - csirtg_fm.cli[111] - processing: /etc/cif/rules/default - None:iprep
2018-09-28 14:40:12,781 - DEBUG - csirtg_fm.clients.http[97] - /var/lib/cif/packetmail.net
2018-09-28 14:40:12,781 - DEBUG - csirtg_fm.clients.http[119] - /var/lib/cif/packetmail.net/iprep_mail.txt
2018-09-28 14:40:12,781 - DEBUG - csirtg_fm.clients.http[136] - CACHE /var/lib/cif/packetmail.net/iprep_mail.txt
2018-09-28 14:40:12,781 - DEBUG - csirtg_fm.clients.http[203] - cache size is 0, downloading...
2018-09-28 14:40:12,783 - DEBUG - urllib3.connectionpool[823] - Starting new HTTPS connection (1): packetmail.net:443
2018-09-28 14:40:13,003 - DEBUG - urllib3.connectionpool[393] - https://packetmail.net:443 "GET /iprep_mail.txt HTTP/1.1" 412 510
2018-09-28 14:40:13,005 - DEBUG - csirtg_fm.cli[98] - testing parser: /var/lib/cif/packetmail.net/iprep_mail.txt
2018-09-28 14:40:13,006 - DEBUG - csirtg_fm.cli[100] - detected parser: None
2018-09-28 14:40:13,006 - INFO - csirtg_fm.cli[111] - processing: /etc/cif/rules/default - None:iprep_mail
2018-09-28 14:40:13,007 - DEBUG - csirtg_fm.clients.http[97] - /var/lib/cif/packetmail.net
2018-09-28 14:40:13,008 - DEBUG - csirtg_fm.clients.http[119] - /var/lib/cif/packetmail.net/iprep_ramnode.txt
2018-09-28 14:40:13,009 - DEBUG - csirtg_fm.clients.http[136] - CACHE /var/lib/cif/packetmail.net/iprep_ramnode.txt
2018-09-28 14:40:13,010 - DEBUG - csirtg_fm.clients.http[203] - cache size is 0, downloading...
2018-09-28 14:40:13,012 - DEBUG - urllib3.connectionpool[823] - Starting new HTTPS connection (1): packetmail.net:443
Process Process-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 176, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd80ffad898>, 'Connection to packetmail.net timed out. (connect timeout=120)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='packetmail.net', port=443): Max retries exceeded with url: /iprep_ramnode.txt (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd80ffad898>, 'Connection to packetmail.net timed out. (connect timeout=120)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/csirtg_fm/cli.py", line 94, in _run_fm
    cli.fetch(fetch=fetch)
  File "/usr/local/lib/python3.6/site-packages/csirtg_fm/clients/http.py", line 204, in fetch
    self._cache_write(self.handle)
  File "/usr/local/lib/python3.6/site-packages/csirtg_fm/clients/http.py", line 181, in _cache_write
    resp = self._cache_refresh(s, auth)
  File "/usr/local/lib/python3.6/site-packages/csirtg_fm/clients/http.py", line 151, in _cache_refresh
    resp = s.get(self.remote, stream=True, auth=auth, timeout=self.timeout, verify=self.verify_ssl)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 501, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='packetmail.net', port=443): Max retries exceeded with url: /iprep_ramnode.txt (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd80ffad898>, 'Connection to packetmail.net timed out. (connect timeout=120)'))

problem: test data should be dynamically generated

https://github.com/joke2k/faker/tree/master/faker

problem: aggregation should be done with iterators

https://github.com/csirtgadgets/csirtg-smrt-py/blob/master/csirtg_smrt/parser/cef.py#L147

def aggregate(indicators):
    last_t = None
    bucket = {}
    for i in indicators:
        t = round_time(dt=datetime.now(), round=int(args.aggregate))
        if t != last_t:
            for i, cnt in bucket.values():
                i.count = cnt
                yield i
            bucket = {}
            last_t = t

        if i.indicator in bucket:
            bucket[i.indicator][1] += 1
            logger.info('skipping send {}'.format(i.indicator))
        else:
            bucket[i.indicator]=[i, 0]

    for i, cnt in bucket.values():
        i.count = cnt
        yield i

problem: smrt doesn't listen on different transports

syslog
zyre

postprocess data structs

csirtgadgets/csirtg-smrt-v1#2

problem: smrt doesn't keep track of feeds that go 'dead'

track 404s, 500's, etc..
send notification when feed goes into 'deadmans' mode
function to manip yml files to ".off"

problem: smrt.py is too fat

needs restructuring.

problem: missing --remember-first-pass flag

test with blocklist.de ssh feed, but effectively; should archive the first pass of data [don't send to the client], then start sending there-after.

good for feeds w/o timestamps. who knows how far back that historical data goes...

remember_first_pass
set a flag (/tmp?) after the first pass ? or set a var in the smrt db?
what if we're a first pass for a new feed? should we start keeping feed stats in the archive?

problem: fetchers cannot load parsers

right now, parsers load the fetchers and it should be the other way around.

content detection needs to influence what parser to use
parsers should be plugins that the fetcher (or other stream) loads
test with syslog/zyre stream

csirtgadgets / csirtg-fm-v1 Goto Github PK

csirtg-fm-v1's People

Contributors

Stargazers

Watchers

Forkers

csirtg-fm-v1's Issues

Recommend Projects

Recommend Topics

Recommend Org