yeti-platform / yeti Goto Github PK
View Code? Open in Web Editor NEWYour Everyday Threat Intelligence
Home Page: https://yeti-platform.io/
License: Apache License 2.0
Your Everyday Threat Intelligence
Home Page: https://yeti-platform.io/
License: Apache License 2.0
Add support for Metadefeder Cloud public APIs and Threat Intel Feeds.
Can you recommend on how to update the Yeti instance with the latest version without losing your data such as observables investigations, etc.?
Some times Im clumsy and ad a link between two nodes that should not be there. Then I get sad when I cant delete it. Solution for has been to delete on of the nodes, reload the webpage, add the node and re-add new links. If I dont reload the webpage, it will add the old links if you try to add the node again.
Question | Answer |
---|---|
Git commit | 0ece661 |
OS version | Ubuntu 16.04.2 |
Browser | Firefox 52.0.2 |
Some button or way to delete just the link, without doing the long hack of deleting the node and all its link.
No way to delete links :(
When tags are renamed from admin, changes are not applied to observables.
Hello,
After adding some users on the system, it is not possible to logout of yeti account. So it is not possible to disable it from the panel.
Best regards,
Running "curl https://raw.githubusercontent.com/yeti-platform/yeti/master/extras/bootstrap.sh | sudo /bin/bash" in terminal clone and downloads as expected until collecting "pythonwhois" from requirements.txt entry "git+git://github.com/joepie91/python-whois#egg=pythonwhois" (line 15)
Occurs when repo is cloned as well.
Question | Answer |
---|---|
Git commit | commit 37f154d |
OS version | Parrot Security Distribution (Built on Ubuntu) |
Browser | Firefox ESR 45.8.0 |
Expect complete installation.
Incomplete installation, no IP address is provided. Not clear if this is an error in requirements.txt or version.
Yeti is throwing an exception when we attempt to upload observables. We are having difficulties identifying the commonalities between the observables that fail and the observables that work. We believe it might be associated with uploading large numbers of duplicate observables.
Has anyone seen this exception before?
Apr 4 14:56:27 <hostname> uwsgi[13723]: [2017-04-04 14:56:27,095] ERROR in app: Exception on /observable/ [POST]
Apr 4 14:56:27 <hostname> uwsgi[13723]: Traceback (most recent call last):
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
Apr 4 14:56:27 <hostname> uwsgi[13723]: response = self.full_dispatch_request()
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
Apr 4 14:56:27 <hostname> uwsgi[13723]: rv = self.handle_user_exception(e)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
Apr 4 14:56:27 <hostname> uwsgi[13723]: reraise(exc_type, exc_value, tb)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
Apr 4 14:56:27 <hostname> uwsgi[13723]: rv = self.dispatch_request()
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
Apr 4 14:56:27 <hostname> uwsgi[13723]: return self.view_functions[rule.endpoint](**req.view_args)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "/usr/local/lib/python2.7/dist-packages/flask_classy.py", line 200, in proxy
Apr 4 14:56:27 <hostname> uwsgi[13723]: response = view(**request.view_args)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "./core/web/helpers.py", line 32, in inner
Apr 4 14:56:27 <hostname> uwsgi[13723]: return f(*args, **kwargs)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "./core/web/frontend/observables.py", line 87, in index
Apr 4 14:56:27 <hostname> uwsgi[13723]: o = Observable.add_text(txt)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "./core/observables/observable.py", line 105, in add_text
Apr 4 14:56:27 <hostname> uwsgi[13723]: o = Observable.guess_type(text).get_or_create(value=text)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "./core/database.py", line 297, in get_or_create
Apr 4 14:56:27 <hostname> uwsgi[13723]: return cls.objects.get(value=obj.value)
Apr 4 14:56:27 <hostname> uwsgi[13723]: File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 271, in get
Apr 4 14:56:27 <hostname> uwsgi[13723]: raise queryset._document.DoesNotExist(msg)
Apr 4 14:56:27 <hostname> uwsgi[13723]: DoesNotExist: Observable.Ip matching query does not exist.
Question | Answer |
---|---|
Git commit | 27ae09f |
OS version | Ubuntu 16.04 |
Browser | Chrome 56.0.2924.87 |
Observables should be inserted into Yeti
We are receiving an "500 Internal Server Error" instead
I get an "Internal Server Error" when I try to connect to http://localhost:5000
Question | Answer |
---|---|
Git commit | Type `$ git log |
OS version | macOS 10.10.5 |
Browser | Version 56.0.2924.87 (64-bit) |
Expect the web UI to start up.
Browser hangs for a few seconds and then throws an error.
"Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."
I've also installed threat_note which is similar platform to Yeti. When I have it running I also have to connect to http://localhost:5000 in the browser. When I run Yeti and try to open in the browser the error message above is displayed and the favicon on the browser tab is the threat_note favicon; however, I didn't have threat_note running at the time.
I'm wondering if these two platforms are colliding in some way even though I'm only running one at a time. I've attached the error message from the command line.
Thanks!
yeti_error.txt
Add the ability to tag email observables
Build equivalence dictionaries for tags, eg. if an observable is tagged c&c
automatically change that to c2
.
Be able to search for http://github.com
when entering hxxp://github[.].com
Add a censys.io one time analytic in the investigation view
Yeti generates a 502 error when I try to add an IDN in the database.
When having multiple investigations popping up from different analysts, it would be nice to know who created (started/working on/owns) which.
After creating a template, it does not show up in Exports tab in the drop down menu for templates in New export. Reloading the page populates the drop down menu.
Question | Answer |
---|---|
Git commit | 0ece661 |
OS version | Ubuntu 16.04.2 |
Browser | Firefox 52.0.2 |
New template should be there.
New template is not there.
All step was fine until i start python2 yeti.py (tried also ./yeti.py)
[+] Yeti started. Point browser to http://localhost:5000/
Any clue?
When i try to open page all the noise come up:
ERROR:core.web.webapp:Exception on / [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1473, in full_dispatch_request
rv = self.preprocess_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1666, in preprocess_request
rv = func()
File "/home/docplague/Scrivania/arsenale/yeti/core/web/webapp.py", line 53, in frontend_login_required
if not current_user.is_active and (request.endpoint and request.endpoint != 'frontend.static'):
File "/usr/lib/python2.7/dist-packages/werkzeug/local.py", line 343, in getattr
return getattr(self._get_current_object(), name)
File "/usr/lib/python2.7/dist-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 26, in
current_user = LocalProxy(lambda: _get_user())
File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 302, in _get_user
current_app.login_manager._load_user()
File "/usr/local/lib/python2.7/dist-packages/flask_login/login_manager.py", line 313, in _load_user
return self._load_from_request(request)
File "/usr/local/lib/python2.7/dist-packages/flask_login/login_manager.py", line 370, in _load_from_request
user = self.request_callback(request)
File "/home/docplague/Scrivania/arsenale/yeti/core/web/webapp.py", line 44, in api_auth
return User.objects.get(api_key=request.headers.get('X-Api-Key'))
File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/manager.py", line 37, in get
queryset = queryset_class(owner, owner._get_collection())
File "/usr/local/lib/python2.7/dist-packages/mongoengine/document.py", line 206, in _get_collection
cls.ensure_indexes()
File "/usr/local/lib/python2.7/dist-packages/mongoengine/document.py", line 836, in ensure_indexes
collection.create_index(fields, background=background, **opts)
File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 1529, in create_index
self.__create_index(keys, kwargs)
File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 1417, in __create_index
with self._socket_for_writes() as sock_info:
File "/usr/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 823, in _get_socket
server = self._get_topology().select_server(selector)
File "/usr/lib/python2.7/dist-packages/pymongo/topology.py", line 214, in select_server
address))
File "/usr/lib/python2.7/dist-packages/pymongo/topology.py", line 189, in select_servers
self._error_message(selector))
ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
Debian 8.5
Question | Answer |
---|---|
Git commit | Type `$ git log |
OS version | Ubuntu 16.04, Windows 10, macOS 10.12.3 |
Browser | Chrome 56.0.2924.87 |
[How are you expecting the application to behave?]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
eg. Searching for a URL also queries its domain, etc.
I run the quick install for Yeti and after all is back to command prompt, I then try to move to the next step.
I have tried the pip install -r requirements.txt (no such file or directory)
Also, I cannot find yeti.py anywhere on my system. I have done the grep and looked in every file and folder i can find. I rebuilt my Ubuntu using 16.04 LTS in a virtual environment.
I ran the command
"sudo apt-get install build-essential git python-dev mongodb redis-server libxml2-dev libxslt-dev zlib1g-dev python-virtualenv"
Then "sudo pip install -r requirements.txt" it tells me no such file or directory
I skip that and try to run the "./yeti.py" doesn't find anything. I need some help.
Want to get this up and going, however, doesn't seem as easy as I was hoping.
I'm a Linux novice, but though that the instructions were fairly simple.
I suppose they are when they work.
Just a quick fix in the install docs. The systemd scripts for "Feeds" and "Exports" have unbalanced quotes.
https://github.com/daniel-gallagher/yeti/commit/b6597582b8ed43d3ea344e0a7ec5afb2f7054e8d
Just need to delete a saved investigation as I was just testing.
Question | Answer |
---|---|
Git commit | commit 27ae09f |
OS version | Debian stretch |
Browser | Chrome 57.0.2987.133 (64-bit) |
To be able to remove the investication
There's no button or link or anything that allows removing an investigation.
It would be helpful to have some graphs representing metrics on the Observable/tags to give you an over of the data set over the last 7/30 days, I was thinking about the following:
I would assume it would entail some aggregations of mongo data and maybe something like pygal. Don't mind taking a crack at it if it seems like a good idea, but looking for some suggestions on what to do the charts in,
Another option would be to put together a Splunk app, but that would limit it to folks who had access to Splunk. I do, but not everyone does,
RUN systemctl daemon-reload
---> Running in 7a7a7661c963
Failed to connect to bus: No such file or directory
os:ubuntu 16.04
Additional feed plugins for openphish, phishtank, and vxvault. I'll try to post more as time allows and figured this would be the best way since I do not have push access to the repo.
zeus
Export
Be able to export investigations in a structural way (json etc).
The export should contain at least the data displayed in the graph, probably with options to include descriptions, tags (include/exclude tags?), observables, context, TLP (Traffic Light Protocol (if the concept is ever implemented), etc.
Import
Be able to import an investigation (previously exported from yeti). Maybe with options to add a tag or source (like a feed) etc. (and maybe import it with a TLP constraint to the user importing it? if the TLP concept is ever implemented).
Create a timeline from an investigation.
There are several links to https://raw.githubusercontent.com/yeti-platform/yeti/master/extras/bootstrap.sh
These should be updated to centos_bootstrap.sh and ubuntu_bootstrap.sh, respectively.
(also at http://yeti-platform.readthedocs.io/en/latest/installation.html)
[Can't start yeti service]
Linux kali 4.9.0-kali3-amd64 #1 SMP Debian 4.9.16-1kali1 (2017-03-24) x86_64 GNU/Linux
Question | Answer |
---|---|
Git commit | Type `$ git log |
OS version | Ubuntu 16.04, Windows 10, macOS 10.12.3 |
Browser | Chrome 56.0.2924.87 |
[How are you expecting the application to behave?]
Start on local port 5000
File "yeti.py", line 11
print "[+] Yeti started. Point browser to http://localhost:5000/"
Send files to Yeti so that it is stored in the database as an Observable (of type File), derive hashes automatically.
First, GREAT app...just what I've been looking for. Second, in reading the API docs, is there a way to submit a file via API? Thank you.
When launching ./yeti.py, I get:
ImportError: No module named flask_restful
The python librairy flask-restful should be added to requirements.txt.
Saved investigation graphs should retain their name
Question | Answer |
---|---|
Git commit | commit 27ae09f |
OS version | Debian stretch |
Browser | Chrome 57.0.2987.133 (64-bit) |
Graph should retain name.
Query search view is confusing. Redesign it to put important information first and ease navigation
From what I can tell, importing domain names when those names have an underscore character does not work. Here are some samples (replaced the periods to prevent people from clicking on potentially malicious domain links...):
ERROR:root:Invalid hostname: gb_validateapplecareassitancesecurelog_in(dot)verfiction1(dot)link
ERROR:root:Invalid hostname: gb_validateapplecareassitancesecurelog_in(dot)vrfied1(dot)link
ERROR:root:Invalid hostname: gb_validateapplecareassitancesecurelog_in(dot)vrify1(dot)link
ERROR:root:Invalid hostname: hack_king10(dot)net23(dot)net
My initial thought is this may be related to the tldexport package used to split up the hostname for validation but I haven't had time to fully run through it.
Question | Answer |
---|---|
Git commit | 37f154d |
OS version | Not relevant |
Browser | Chrome 56.0.2924.87 |
Import hostname observable with an underscore via a feed script.
No, don't die.
Death to domains with underscores.
Add the ability to add BTC addresses as observable
Add the ability to perform Whois analytics on hostnames not just domain names.
I am embarrassed to ask, but I cannot find the default creds. It's certainly possible I missed a config file, but can't seem to find it.
I looked in yeti/core/auth/local/user_management.py
and found:
def get_default_user():
try:
# Assume authentication is anonymous if only 1 user
if User.objects.count() == 1:
return User.objects.get(username="yeti")
return AnonymousUserMixin()
except DoesNotExist:
return create_user("yeti", "yeti")
...but yeti:yeti
didn't seem to work.
Question | Answer |
---|---|
Git commit | 3a3dfa5 |
OS version | CentOS Linux release 7.3.1611 (Core) |
Browser | Chrome 57.0.2987.110 (64-bit) |
Browse to http://server:5000
I attempted admin:admin
, admin:password
, yeti:yeti
, and every combination in between.
Log into the web app
Invalid credentials
Be able to import x509 certificates and run analytics on them (censys.io...)
It appears that the Palevo Tracker has been discontinued and may need to be removed from the public feeds.
Mixed Content: The page at 'https://hostname.com/investigation/graph/observable/577f910a4e563b05955cac9b' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://hostname.com/observable/577f9be24e563b1b5bf05876'. This request has been blocked; the content must be served over HTTPS.
Add a shodan.io one time analytic in the investigation view
Can anyone share the configuration to link Yeti to MISP platform?
Thanks in advance!
Write documentation for updating Yeti and making backups.
Support the Virustotal public and private API, similar to the enrichment from PassiveTotal.
Public API documentation can be found here:
https://www.virustotal.com/en/documentation/public-api/
Private API documentation can be found here:
https://www.virustotal.com/en/documentation/private-api/
I am unable to do export as export format drop down box is empty and hence unable to save the export.
OS: Ubuntu 16.04
Browser: firefox, chrome
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.