stamparm / maltrail Goto Github PK
View Code? Open in Web Editor NEWMalicious traffic detection system
License: MIT License
Malicious traffic detection system
License: MIT License
Hi,
It would be great if malware feeds are being pushed from server to sensors instead of sensors are downloading lists for distributed architectures where only server can reach the internet for rule-updates etc.. (Like a distributed security onion installation etc)
Hello,
first congrats for your job.
I wrote you because i try to change the password in the maltrail.conf, and now i can't connect to the server...
here is the mailtrail.conf user's section
USERS
#admin:$ff0ae5570e1f39a8$10000$d42e622afe0b0ede53b64b97a59d65c221edbf9dde2f0e95:0:0.0.0.0/0 # changeme!
toto:$8c81af41b9a90583$10000$650a0db5a4df76fee8a2557c1388be3c2b0b16362c423680:0:0.0.0.0/0
i ran on windows and all was ok before i change the password.
Thanks for your help
root@xyz:/var/log/maltrail# ping -c 1 136.161.101.53
PING 136.161.101.53 (136.161.101.53) 56(84) bytes of data.
64 bytes from 136.161.101.53: icmp_seq=1 ttl=54 time=81.6 ms
--- 136.161.101.53 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 81.669/81.669/81.669/0.000 ms
root@xyz:/var/log/maltrail# cat /var/log/maltrail/$(date +"%Y-%m-%d").log
cat: /var/log/maltrail/2015-12-16.log: No such file or directory
Would be awesome to have the ability to, perhaps, highlight an IP address and say copy it...or perform additional tasks (look up in virustotal or urlquery or urlvoid). Thank you!
these are only my 2cents, unfortunately i usually use bitbucket for my "projects", so i show you what i have added to your very good project for meet my needed inside my honeynet sensor:
https://bitbucket.org/n3ophyte/maltrail_dev/commits/branch/develop-ua
i can recreate it in github and then make a pull-request to a development branch of your repo if you think that this can also by useful for you, obv iam not a developer so if you think that there will be a better way to deal with this implementation no problems :)
my next development will be to create a python instance and a caching mechanism for scanning all logfiles for create a single JSON document that contains all historical matches for further search and a better overview of aggregate data that every sensor are collecting, something like:
[IP/Domain] {
[first_seen]
[last_seen]
[hits]
[UA] { [1] => .. [2] => ... }
[Reliability] = from 10 to 0
}
Reliability can be from 10 to 0, where 10 is added when we first add this information to the JSON document and then every day or two (or whatever we think that could be a good security timeframe) we decrease that number if we dont see that object anymore in the sensor...
this could be integrated in the portal so we have a single method for searching all the events over the time without scanning billions of lines of files that can contains many time the same ip/domain/url
Would be great to double-click a flag and have it pop the two character country code into the filter field and filter by country. Thank you.
When I filter with the word "high" or when i click "high" in the severity column not only filter "high" because in the info column we have "high consonant no such domain name".
ping 202.99.224.68
not loging,why?
but nmap is loging.
1.➜ maltrail git:(master) ✗ sudo python sensor.py
[sudo] password for asrr:
Maltrail (sensor) #v0.8.288
[i] using configuration file '/home/asrr/tools/maltrail/maltrail.conf'
[i] loading trails file...
[i] 814,061 trails loaded
[i] using '/var/log/maltrail' for log storage
[i] opening interface 'eth0'
[i] setting filter '(tcp[13] == 2) or (tcp[13] & 8 != 0) or not tcp'
[i] creating 3 more processes (4 CPU cores detected)
[o] running...
2.➜ maltrail git:(master) ✗ ip route get 202.99.224.68
202.99.224.68 via 172.16.9.70 dev eth0 src 172.16.9.67
cache
3.➜ maltrail git:(master) ✗ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 3c:97:0e:e8:3b:30
inet addr:172.16.9.67 Bcast:172.16.9.71 Mask:255.255.255.248
inet6 addr: fe80::3e97:eff:fee8:3b30/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:21314 errors:0 dropped:1 overruns:0 frame:0
TX packets:20767 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17267930 (17.2 MB) TX bytes:2672368 (2.6 MB)
4.➜ maltrail git:(master) ✗ cat ~/.maltrail/trails.csv |grep 202.99.224.68
➜ maltrail git:(master) ✗
5.➜ maltrail git:(master) ✗ cat maltrail.conf | grep MONITOR_INTERFACE
MONITOR_INTERFACE eth0
➜ maltrail git:(master) ✗
6.cat 2015-12-26.log|grep 202.99.224
"2015-12-26 12:17:54.146010" weizi-linux 172.16.9.67 59850 202.99.224.68 53 UDP DNS bluereader.org malware otx.alienvault.comu
but not ping
First off THIS APP IS AWESOME. Just started with it today and it rocks. Second, any chance we can get an autorefresh workin on this? Thanks so much!
Even when using the default password, the authentication does not work properly. I've tried setting a new password for the admin account but the submitted password never seems to match so I cannot view the user interface.
My suspicion is that it has to do with the server.py running on windows for some reason. I've tried looking at the authentication code in httpd.py but the main thing is that somewhere along there, the Session value never gets set properly due to issues reading from the config file. The only difference I really have is with the log paths being windows based () instead of unix based (/). Any thoughts on what might be causing the authentication issue with the default setup?
Hi,
On FreeBSD, 10.2 (64bits), I have this issue:
[i] using configuration file '/usr/maltrail/maltrail.conf'
free: not found
[!] unable to determine total physical memory. Please use absolute value for 'CAPTURE_BUFFER'
Sincerely
So here's what I got from my maltrail.conf:
USER_WHITELIST 127.0.0.1,localhost,71.21.81.200
But I continue to see this IP address in the list. Sensor running in debug doesn't show anything. I'm also specifically looking at the last seen time. Thank you!
Hello stamparm,
i'm glad to discover this tool, could be useful to use it. That's why i've started to try it and at this time i'm not able to do what is it supposed to do.
First of all, i've nothing on the webserver, saying there's no event. But the calendar show event and the number increase.
I've started in debug, it's an ubuntu 14.04.
To begin, in documentation, you don't use sudo to start the webserv daemon, and i got a perm error to read logs. It could be awesome to dos something who will work at the begin without needs of chown or chmod.
And the problem i have when starting the webserver as sudo/root :
[i] starting HTTP server at 'http://0.0.0.0:8338/'
[o] running...
Traceback (most recent call last):
File "/home/admin/maltrail/core/httpd.py", line 514, in _counts
current = datetime.datetime.strptime(os.path.splitext(os.path.basename(filename))[0], DATE_FORMAT)
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data 'error' does not match format '%Y-%m-%d'
My sensor log file looks like this :
"2015-12-29 03:57:27.818642" maltrail 149.202.238.199 41049 92.x.x.x 8888 TCP IP 149.202.238.199 "bad reputation (malicious)" alienvault.com
"2015-12-29 03:59:53.915040" maltrail 92.x.x.x - 136.161.101.53 - ICMP IP 136.161.101.53 "conficker (malware)" (static)
"2015-12-29 04:03:56.389039" maltrail 92.x.x.x - 136.161.101.53 - ICMP IP 136.161.101.53 "conficker (malware)" (static)
I'm trying too to put a sensor on an other server, but actually it's not able to send logs to webserver. I'll tell you more in other ticket if i don't find how to fix this.
Thanks again for this good job.
Hey again. So I've been running the latest git pull for about 4 hours now. After fixing the Cisco VLAN issue (thanks much!) I am seeing src and dst IP's that look correct, but the ports do not. An example:
Src: 64.74.133.82
Src Port: 50887
Dst: x.x.x.x
Dst Port: 56863, 56902
Yet packet capturing during this time shows no hits on port 50887. Bro-ids does show 64.74.133.82, but Src port ranges from 33573-38544, with Dst port ranges of 33440-33444, this is a traceroute. Betting something isn't getting translated correctly. Thank you.
Hi, both of services shutdown 2-3 hours since start up.
I launch it so:
python sensor.py &
python server.py &
What I doing bad ?
Regards!
Maltrail is great!
However, it captures packets before host-based firewalls see those packets, resulting in a lot of false positives. That is, if your firewall is already blocking the packets from blacklisted IPs, your host is protected from that threat. While it is interesting and educational to know what is being directed towards your hosts, it would be more useful to know what got past the firewall.
I don't want to whitelist my blacklisted IP addresses, in case the firewall rules change (or the firewall is off) and the firewall inadvertently is allowing some blacklisted IP addresses through.
Ideally, maltrail would use my firewall rules/blacklists to prepend a comment character to the logged line. Commented log lines would normally be ignored by the client when the web display is generated, but could be re-instated in the display if desired to peek at activity behind the firewall.
Perhaps allowing plugins at critical points in processing would be a better idea and more general. We would then write the code to do the special filtering that we desire.
My sensor was running since 26 december and I saw that the process "sensor" was using 60% of my total ram (2GO).
I don't really know why it using so much ram because if I restart it, sensor now use only 13% of my ram.
root@unixfox:~# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 16893 0.0 39.9 1080768 837880 ? Ssl 2015 10:08 /usr/bin/python2 /root/maltrail/sensor.py
root 16896 0.9 21.9 994892 461316 ? Sl 2015 146:11 /usr/bin/python2 /root/maltrail/sensor.py
Hello.
I have problem with logging like this user on #14 . My server run on 0.0.0.0:8338 and sensors running too (opening interface 'any'). Server run without sudo and sensors with sudo command. But there is no log when ping some IP. I can login on my server panel successfully. First I try with screen on server but nothing happen then I install VNC and run on VNC and this not problem. My configuration is default.
Please help me, thank you.
How can I use SHA512 and not SHA256 password?
From http://vxvault.siri-urz.net/URL_List.php to http://vxvault.net/URL_List.php.
related to /trails/feeds/vxvault.py
If server is behind cloudflare it shows coudflare ip, not the real ip :) Pls fix it.
Would be awesome to have the Trail popup use virustotal to lookup the IP address. Thank you!
Good evening.
I write to ask if anyone knows how to make a confguration so you can connect to the server through a VPN.
In local network work correctly. But when trying to connect to the server through a VPN, which is not put in the IP address settings.
For your reply, thanks.
When I scan IPs range sensor shutdown and need scan again... Where is problem? Is there timeout for sensor or what?
it could be very cool if we can select more then one day of available data to see more complete picture
As @stonfute said in this #30 (comment).
It would be awesome if you can implement a function to ban an IP on the web interface with a button !
After starting sensor and server, on web interface I get 'The requested url /maltrail/html/whoami not found'
here the exception:
[i] updating trails...
[o] 'https://feodotracker.abuse.ch/blocklist/?download=domainblocklist'
[!] something went wrong during remote data retrieval ('https://feodotracker.abuse.ch/blocklist/?download=domainblocklist')
[o] 'https://www.badips.com/get/list/any/2?age=30d'
[o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/ri_web_proxies_30d.ipset'
[o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/proxylists_30d.ipset'
[o] 'http://www.botscout.com/last_caught_cache.htm'
[o] 'http://vxvault.siri-urz.net/URL_List.php'
[o] 'http://malwaredomains.lehigh.edu/files/domains.txt'
[o] 'https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/socks_proxy_30d.ipset'
[o] 'https://zeustracker.abuse.ch/blocklist.php?download=badips'
[o] 'https://openphish.com/feed.txt'
[o] 'https://myip.ms/files/blacklist/htaccess/latest_blacklist.txt'
[o] 'https://www.maxmind.com/en/proxy-detection-sample-list'
[o] 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist'
[o] 'http://malwareurls.joxeankoret.com/normal.txt'
[o] 'http://talosintel.com/feeds/ip-filter.blf'
[o] 'https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1'
[o] 'https://rules.emergingthreats.net/open/suricata/rules/emerging-dns.rules'
[o] 'http://www.nothink.org/blacklist/blacklist_malware_irc.txt'
[o] 'http://danger.rulez.sk/projects/bruteforceblocker/blist.php'
[o] 'http://malwared.malwaremustdie.org/rss.php'
[o] 'http://www.voipbl.org/update/'
[!] Unhandled exception occurred ('Python int too large to convert to C long')
[x] Please report the following details at 'https://github.com/stamparm/maltrail/issues':
---
'Traceback (most recent call last):
File "sensor.py", line 389, in <module>
main()
File "sensor.py", line 382, in main
init()
File "sensor.py", line 287, in init
update_timer()
File "sensor.py", line 275, in update_timer
_ = update(server=config.UPDATE_SERVER)
File "/opt/maltrail/maltrail/core/update.py", line 82, in update
results = function()
File "/opt/maltrail/maltrail/trails/feeds/voipbl.py", line 29, in fetch
for address in xrange(start_int, end_int + 1):
OverflowError: Python int too large to convert to C long
'
---
Debian 6
Python 2.7.3
Thanks for your amazing work!!
Ability to manage multiple sensors with the web interface.
Hi there I think is good idea to let us setup in the config file config more than one "LOG_SERVER" or one LOG_SERVER and save that in local :-)
I use ELK solution and to let me use that with web server side I modify the "core/log.py" and in the section "def log_event(event_tuple)" in the first "if":
if config.LOG_SERVER:
remote_host, remote_port = config.LOG_SERVER.split(':')
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto("%s %s" % (sec, event), (remote_host, int(remote_port)))
I add:
handle = get_event_log_handle(sec)
os.write(handle, event)
Then now I can send and save the logs :-)
What do you think about that improvement ?
Thanks a happy holidays
Support for showing which vhost is targeted would be really useful, and fairly easy to extract from the traffic.
Hi, I am trying to install a sensor on Centos 6? Is possible?
Thanks in advance!
When I want to block an attacker, I null route him with "route add" (http://www.cyberciti.biz/tips/how-do-i-drop-or-block-attackers-ip-with-null-routes.html) but even after he is still displayed on the web interface in the first position in the column "last seen".
Maybe you should add an option to hide automatically IP who has been blocked.
Trying to start the soft in an ubuntu 14.04 :
:~/maltrail$ sudo python sensor.py
Maltrail (sensor) #v0.8.187
[i] using configuration file '/home/user/maltrail/maltrail.conf'
[i] loading trails file...
[i] 921,479 trails loaded
[i] using '/var/log/maltrail' for log storage
[!] in case of any problems with packet capture on virtual interface 'any', please put all monitoring interfaces to promiscuous mode manually (e.g. 'sudo ifconfig eth0 promisc')
[i] opening interface 'any'
[i] setting filter '(tcp[13] == 2) or (tcp[13] & 8 != 0) or not tcp'
[i] creating 3 more processes (4 CPU cores detected)
[!] unhandled exception occurred ('[Errno 12] Cannot allocate memory')
'Traceback (most recent call last):
File "sensor.py", line 531, in
main()
File "sensor.py", line 524, in main
init()
File "sensor.py", line 443, in init
_init_multiprocessing()
File "sensor.py", line 470, in _init_multiprocessing
_buffer = mmap.mmap(-1, BUFFER_LENGTH) # http://www.alexonlinux.com/direct-io-in-python
error: [Errno 12] Cannot allocate memory
If a can adding another data for fix this bug...
core/Settings.py line 111
-- retval = psutil.phymem_usage().total (depreciated in my psutil package?)
++ retval = psutil.virtual_memory().total (works)
https://mail.python.org/pipermail/python-announce-list/2012-August/009575.html
Worked fine for a few hours, then I'm getting hundreds of javascript alerts
Uncaught TypeError: Cannot read property 'html' of undefined Script: http://xxxxxx.com:8338/js/main.js Line:1387
Would love to be able to filter by a port number. Would be nice also to have src/dst/port/protocol filtering like maybe something below:
USER_WHITELIST src_192.168.1.1_tcp_25
USER_WHITELIST src-192.168.1.1-udp-53
Thank you.
Hello,
could be useful to have a buton to click to go back to the main view, without the need to clear the filter.
Thank you !
So I have a machine that listens to several netblocks. I am wanting to listen to just the external traffic. This external traffic only has routable IP addresses, we don't see any internal only (10.0.0.0/8, 192.168.0.0/24 for example) traffic. This traffic is on eth2:
MONITOR_INTERFACE eth2
This interface is in promiscuous mode:
eth2 Link encap:Ethernet HWaddr 00:00:00:00:00:01
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
I do not see any events on this interface, even though I do see hits from say, bro-ids:
139.196.104.39 58928 x.x.x.x 443 - - - 139.196.104.39 Intel::ADDR Conn::IN_ORIG bro alienvault
If I set maltrail to any, I see hits, but only internal. Is there something I can do to troubleshoot this? Thaank you.
Would be great to be able to apply some basic regex or lucene (http://www.lucenetutorial.com/lucene-query-syntax.html) type filtering in the input. For example:
NOT mass scanner
malicious AND malware
Thank you!
I love the software well done, easy to use, but so far it's a passive tool.
Adding a button that would launch :
iptables -I INPUT -s xxx.xxx.xxx.xxx -j DROP (xxx beeing the offender IP) on each offender line.
Could even add it as an autobanning feature, with maybe a treshold like fail2ban,
That would certainly make my day :)
(could even make it an option to autoban the known mass scanners at first start)
(I'm not forgiving, my jail2ban jails ban the 1st time a week, in case of recidive a year, but fail2ban is not that easy to configure, especially for new jails, this, with banning features, could really be a better solution)
Hello!
Would be great if there was a Windows sensor for this.
Use-Case: In most setups there is a mix of both Windows and Linux machines. I'd like to monitor both.
My mobile device can't really load maltrail because it've to render the interface like a PC.
Is it possible to have responsive on the web interface or support mobile devices ?
Hi! Thanks for you work.
Trying to run the system on FreeBSD 10.0-RELEASE (Python 2.7.6). Default configs. Got this traceback:
root@iota:/tmp/3/maltrail-master # python2.7 sensor.py
Maltrail (sensor) #v0.8.359
[i] using configuration file '/tmp/3/maltrail-master/maltrail.conf'
[i] using '/var/log/maltrail' for log storage
[?] at least 512MB of free memory required
[i] loading trails file...
[i] 867,140 trails loaded
[x] virtual interface 'any' missing. Replacing it with all interface names
[i] opening interface 'em0'
[i] opening interface 'taurus0'
[i] opening interface 'libra0'
[i] opening interface 'em0.5'
[i] opening interface 'em0.100'
[i] opening interface 'em0.101'
[i] opening interface 'em0.102'
[i] opening interface 'em0.103'
...
[i] opening interface 'em0.600'
[i] opening interface 'lo0'
[i] setting filter 'ip and (not tcp or (tcp[13] == 2) or (tcp[13] & 8 != 0))'
[i] creating 1 more processes (2 CPU cores detected)
[?] please install 'schedtool' for better CPU scheduling
[o] running...
Process 0:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/local/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/tmp/3/maltrail-master/core/parallel.py", line 92, in worker
sec, usec = struct.unpack("=II", content[:8])
error: unpack requires a string argument of length 8
[x] Ctrl-C pressed
The server starts fine:
root@iota:/tmp/3/maltrail-master # python2.7 server.py
Maltrail (server) #v0.8.359
[i] using configuration file '/tmp/3/maltrail-master/maltrail.conf'
[i] starting HTTP server at 'http://0.0.0.0:8338/'
[o] running...
The web-interface is up and running. I can see some threats there. But data doesn't update. It seems it analyse the traffic once (when starting) and then stops.
Can this error "error: unpack requires a string argument of length 8" cause the problem? Tnx!
Browser keeps request http://serverip:8338/events?date=2015-12-16
until chrome tab crash
probably too many events? after i stop server.py, The dashboard shows some data
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.