API re-factor step 2

Let's start the re-factor by removing unused code.

Fix test to reflect the recent change in the Sketch model.

The model definitions for the Sketch app needs clean up. Add good documentation and tests.

Add functionality to create a timeline from a json file with events.

It would be nice to manually be able to hide events in a view. Possible implementation could be a special label that is applied to the event that will prevent the UI to render it.

This is meant for manual hiding events on a small dataset, not bulk hiding.
This will not limit the amount of events sent to the client, only what is visually displayed.

An indicator that there are hidden events in the view need to be in place and a way to toggle visibility of them.

Stories

Imagine if you could mix your investigative narrative with the raw timeline data from timesketch. This is what Timesketch stories is all about. Giving the analyst the power to share their thought and hypothesis in a narrative way embedding timesketch timeline events.

Sub tasks

TBD

It would be extremely useful if, between two events generated from plaso let's say, one could manually add something that happened like "10:30am the user received a phone call from someone pretending to be XYZ" or whatever other type of event/note with a timestamp.
Thanks.

Refactor the project

This issue will track the overall progress of a big re-factor to the project. There will be a couple of sub issues tracking individual steps in the re-factor, but when each of these sub issues is done they will be merged into this re-factor branch. And when all is done and the master branch is reviewed the re-factor will me merged into master.

Some highlights

• Move from Django to Flask
• Move from Django ORM to SQLalchemy
• Move from Tastypie rest framework to flask-restfull
• New ACL system
• New annotation system
• Better test coverage

I'd like to have a saved view option, where I can highlight several events (e.g., all the events associated with a type 10 login) and annotate them "RDP Login", then have them collapse in the UI into a single line.

Honor the ?next= argument when login.

When pressing the "starred" link in explore view it is impossible to search for anything else. It seems that the view is stuck on only showing starred events.

When trying to add a timeline via add_timeline.py I am getting the following exception:
Traceback (most recent call last):
File "/var/www/timesketch/utils/add_timeline.py", line 113, in
sys.exit(main())
File "/var/www/timesketch/utils/add_timeline.py", line 92, in main
elasticsearch.put_mapping(args.index, 'plaso_event', mapping)
File "/usr/local/lib/python2.7/dist-packages/pyelasticsearch/client.py", line 96, in decorate
return func(_args, query_params=query_params, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/pyelasticsearch/client.py", line 659, in put_mapping
query_params=query_params)
File "/usr/local/lib/python2.7/dist-packages/pyelasticsearch/client.py", line 254, in send_request
self._raise_exception(resp, prepped_response)
File "/usr/local/lib/python2.7/dist-packages/pyelasticsearch/client.py", line 269, in _raise_exception
raise error_class(response.status_code, error_message)
pyelasticsearch.exceptions.ElasticHttpError: (400, u'MergeMappingException[Merge failed with failures {[Cannot update path in _timestamp value. Value is datetime path in merged mapping is missing]}]')

I recently upgraded our ES instance from 1.2 to 1.4 not sure if that is a culprit or not. Previous additions to timesketch worked fine prior to the ES upgrade. Any suggestions?

Can not run add_plaso_timeline.py from the utils directory. We need to fix this script for Django 1.7 compatibility, i.e. we need to do django.setup().

When we are at it, we should clean up the script so it takes flags.

• The default list of timelines should be sorted by last added, both in the initial view and in search results.

Experimental UI

Add events per day/hour heatmap to query results by using elasticsearch aggregation ability.

Something like this:

Hi,

Everytime I use psort.py -o timesketch I get the following error

plaso/output/interface.py, line 45:
ValueError: Unused keyword arguments: filehandle.

After going into the file and commenting out the kwargs check it can contact my elasticsearch server but it crashes on the following error (it cant set the index aswell as the computer name/hostname):

[INFO] Adding events to timesketch..
Traceback:
for flag description in self.SECURITY_VALUES:
typerror: 'int' object is not iterable.

Cleanup the HTML templates and CSS.

• Remove inline CSS
• Make HTML more readable

When running the initial manage.py migrate to create the database, it fails with:
ImportError: cannot import name 'api' from 'timesketch'

Use Angular templates in directives

If, for example, I've done a search for a known bad indicator (e.g., IP address), I'd like to be able to easily star all the results with a single click.

Today we are using a ManyToMany field for timelines in a sketch. This is unnecessary complexity and makes migrations hard. Let's change this to be a normal foreign key relationships on the SketchTimeline model instead and add the timelines attribute as a property on the Sketch model.

The model definitions for the Sketch app needs clean up. Add good documentation and tests.

The date string we pass to Date() is not valid in firefox. This makes dates not shown in comments.

Sometimes plaso duplicates events. It would be handy if timesketch didn't show duplicate events.

If a search yields > 500 results, only the first (oldest) 500 results are displayed in the explore view without an easy way to page through the next batch of results. The end goal can be achieved by taking the last date/time and using that as Start date in the Time range filter but ideally a way (button etc.) at the top/bottom of the events results would be more ideal.

We need a settings view to put sketch related settings.

• Move the ACL modal that uses the API to this view

When using postgresql some queries fails with DataError.

This is due to the way SQLalchemy constructs the query, and the way we pass in the searchindex name to the filter.

API re-factor step 1

In preparation for a big re-factor and cleanup of the API code, we need to add proper tests.
Make sure we have good coverage before closing this issue.

Please get Timesketch a favicon so it's easy to pick out amongst numerous tabs in the browser. :-)

Could you add something in the config so we can bind this to 0.0.0.0. Its running locally vm network.
Thanks!

In recent versions of Elasticsearch there was changes made that forces us to change the way we create out indices and mappings. The process of adding timelines to Timesketch is way to complicated at the moment and a first step in making this better is to create a psort output plugin for Timesketch.

In order to be as consistent as possible we need to clean up the codebase. This PR moves all strings to unicode and all SQLAlchemy database column type from String/Text -> Unicode/UnicodeText.

This project looks like it's in an early stage, but is very interesting. I would like to play around with it. Can you add a requirements file? Looks like Elasticsearch will be needed also. I'd be happy to be a guinea pig for any basic instructions to get it running and would gladly assist in refining those.

Refactor the JS frontend code

The current structure of the frontend JS code is hard to test and is somewhat confusing, e.g mixing server-side templates with client-side templates. In order to make this better we need to refactor the code.

Sub tasks

• #61 - Move REST API calls to services
• #66 - Move controllers to directives
• #67 - Utilize templateURL for directives

See this old PR and comments as starting points: #45

Move Javascript and HTML to the refactor branch

• Clean up HTML
• Clean up CSS
• Move Javascript over (as is for now, refactor later)
• Move over third_party libs

We want to be able to edit a saved view, i.e. change the query and filters.

Once one has a certain view events they feel is report worthy, it'd be convenient to have the ability to export those events out of timesketch in some fashion/format (e.g. - CSV). The main purpose for this in my eyes would be to ease the data ingestion process into various reports vs. copying/pasting certain details from the events properties ... especially if there is a large amount of events starred/commented/in a certain view.

The settings.py won't work without adding timesketch.apps.ui to the INSTALLED_APPS as it'll throw a ProgrammingError after adding a sketch

Move away from controllers to directives instead.

Move REST API calls to services.

Add some fanciness to the parsing of the input boxes so that multiple input format will work (just a date, date in american format etc)

To support common SSO solutions Timesketch should be able to setup a session when the REMOTE_USER environment variable is set.

When looking at results of a Saved Search, have a button/link that pops open a new tab that shows the system's timeline with X number of records on either side of the selected entry. For example, when looking across numerous systems for 'at' jobs, if I found a new one I was interested in, I might want to see that system's timeline immediately surrounding the 'at' job.

Move views to django class based views.

CSS incompatibility on firefox make the search input field not showing text.

This is a Plaso bug. Let's work around it until it is fixed upstreams.

This would be similar to the -A and -B flags for grep/ack. A user might have an indicator and want to match nearby events to identify related activity.

I manually adjust the time filter to and use a wildcard search to achieve this effect. Maybe I'm missing how to do this in the ES or Lucene DSL?

It might be interesting to keep an annotation database for each Sketch. For example, if I have an IP address (192.168.4.55), and I annotate it with its hostname (argv-workstation), you could highlight everytime the IP showed up in the sketch, and a tooltip/mouseover could show the annotated hostname. This would keep the analyst from having to X-reference notes constantly.