obsidianforensics / hindsight Goto Github PK

We tried to parse a Google Chrome Profile of a Mac Account but unfortunately it seems like the content of LevelDB Files is not parsed. The Chrome Version is 76. We investigated the LevelDB via a Hex Editor to verify the content. Howsoever, after using hindsight we can't find the specific entries. Is this maybe a result of a specific mac decryption?

Thanks a lot.

Lots of mobile, IoT, etc devices use Chromium-based components, but they aren't structured in the file system exactly like Chrome. Hindsight currently uses this directory structure to determine what files to parse. It would be cool to have a mode where Hindsight walks the file system and examines each file to see if it knows how to parse it, regardless of file name or logical location. I think having this in a separate mode makes sense, rather than building it into the default way it works (but that may change if it works well).

Hi,

I came across this issue when using the compiled (.exe) version of your work. Using hindsight_gui.exe, results in an internal server error, the return screen is as follows:

Error: 500 Internal Server Error

Sorry, the requested URL 'http://localhost:8080/run' caused an error:

Internal Server Error

Exception:

IndexError('list index out of range',)

Traceback:

Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 207, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
IndexError: list index out of range

The GUI CMD shell contains the following:

Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 207, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
IndexError: list index out of range
127.0.0.1 - - [04/Sep/2019 16:26:39] "POST /run HTTP/1.1" 500 1305
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 207, in run
File "pyhindsight\browsers\chrome.py", line 1757, in process
IndexError: list index out of range
127.0.0.1 - - [04/Sep/2019 16:28:32] "POST /run HTTP/1.1" 500 1305

The log file shows the following:

################################################################################

Hindsight v2.3.0 (https://github.com/obsidianforensics/hindsight)

################################################################################
2019-09-04 16:28:32.635 | D | Options: {'display_version': None, 'plugins_info': [{'file_name': 'chrome_extensions', 'friendly_name': u'Chrome Extension Names', 'parent_path': None, 'version': u'20150125', 'error': None, 'error_msg': None}, {'file_name': 'generic_timestamps', 'friendly_name': 'Generic Timestamp Decoder', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'google_analytics', 'friendly_name': 'Google Analytics Cookie Parser', 'parent_path': None, 'version': '20170130', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'google_searches', 'friendly_name': u'Google Searches', 'parent_path': None, 'version': u'20160912', 'error': None, 'error_msg': None}, {'file_name': 'load_balancer_cookies', 'friendly_name': u'Load Balancer Cookie Decoder', 'parent_path': None, 'version': u'20160621', 'error': None, 'error_msg': None}, {'file_name': 'quantcast_cookies', 'friendly_name': 'Quantcast Cookie Parser', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'query_string_parser', 'friendly_name': 'Query String Parser', 'parent_path': None, 'version': '20170225', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'time_discrepancy_finder', 'friendly_name': 'Time Discrepancy Finder', 'parent_path': None, 'version': '20170129', 'error': None, 'error_msg': None}], 'input_path': 'C:\Users\Jon Cooper\AppData\Local\Google\Chrome\User Data\Default', 'log_path': 'hindsight.log', 'hindsight_version': '2.3.0', 'available_decrypts': {'windows': 1, 'mac': 0, 'linux': 0}, 'timezone': <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>, 'browser_type': 'Chrome', 'preferences': [], 'artifacts_counts': {}, 'selected_output_format': 'xlsx', 'plugin_results': {}, 'version': [], 'artifacts_display': None, 'cache_path': '', 'available_input_types': ['Chrome', 'Brave'], 'profile_paths': ['C:\Users\Jon Cooper\Desktop\User Data\Default'], 'available_output_formats': ['sqlite', 'xlsx'], 'selected_decrypts': None, 'selected_plugins': ['chrome_extensions', 'generic_timestamps', 'google_analytics', 'google_searches', 'load_balancer_cookies', 'quantcast_cookies', 'query_string_parser', 'time_discrepancy_finder'], 'parsed_artifacts': [], 'output_name': 'Hindsight Report (2019-09-04T16-25-22)', 'plugin_descriptions': [{'file_name': 'chrome_extensions', 'friendly_name': u'Chrome Extension Names', 'parent_path': None, 'version': u'20150125', 'error': None, 'error_msg': None}, {'file_name': 'generic_timestamps', 'friendly_name': 'Generic Timestamp Decoder', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'google_analytics', 'friendly_name': 'Google Analytics Cookie Parser', 'parent_path': None, 'version': '20170130', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'google_searches', 'friendly_name': u'Google Searches', 'parent_path': None, 'version': u'20160912', 'error': None, 'error_msg': None}, {'file_name': 'load_balancer_cookies', 'friendly_name': u'Load Balancer Cookie Decoder', 'parent_path': None, 'version': u'20160621', 'error': None, 'error_msg': None}, {'file_name': 'quantcast_cookies', 'friendly_name': 'Quantcast Cookie Parser', 'parent_path': None, 'version': '20160907', 'error': None, 'error_msg': None}, {'file_name': 'query_string_parser', 'friendly_name': 'Query String Parser', 'parent_path': None, 'version': '20170225', 'error': 'other', 'error_msg': AttributeError("'NoneType' object has no attribute 'parsed_artifacts'",)}, {'file_name': 'time_discrepancy_finder', 'friendly_name': 'Time Discrepancy Finder', 'parent_path': None, 'version': '20170129', 'error': None, 'error_msg': None}]}
2019-09-04 16:28:32.635 | I | Starting analysis
2019-09-04 16:28:32.635 | I | - Found 1 browser profile(s): ['C:\Users\Jon Cooper\AppData\Local\Google\Chrome\User Data\Default']
2019-09-04 16:28:32.635 | D | Supported items: ['History', 'Archived History', 'Web Data', 'Cookies', 'Login Data', 'Extension Cookies', 'Local Storage', 'Extensions', 'File System', 'Bookmarks']
2019-09-04 16:28:32.670 | D | Analyzing 'History' structure
2019-09-04 16:28:32.670 | D | - Starting possible versions: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73]
2019-09-04 16:28:32.670 | D | - Finishing possible versions: [68, 69, 70, 71, 72, 73]
2019-09-04 16:28:32.671 | D | Analyzing 'Cookies' structure
2019-09-04 16:28:32.671 | D | - Starting possible versions: [68, 69, 70, 71, 72, 73]
2019-09-04 16:28:32.671 | D | - Finishing possible versions: []
2019-09-04 16:28:32.671 | D | Analyzing 'Web Data' structure
2019-09-04 16:28:32.671 | D | - Starting possible versions: []
2019-09-04 16:28:32.671 | D | - Finishing possible versions: []
2019-09-04 16:28:32.671 | D | Analyzing 'Login Data' structure
2019-09-04 16:28:32.671 | D | - Starting possible versions: []
2019-09-04 16:28:32.671 | D | - Finishing possible versions: []

This occurs when trying to analyze a history obtained from a forensic sample, as well as the locally installed Google browser files.

I should note, the python version, running in MacOS, decodes captured Windows Chrome history just fine (the same data). This seems to only be in the compiled versions for Windows.

Right now, the JSONL is designed to be imported into Timesketch, so items that aren't in the "Timeline" section aren't included.

FR: add the "Storage" items to the JSONL output.

You might want to add that NONE of this works if Chrome is your default (and only real in use) browser and you want to test it.....since you open a BROWSER to localhost:8080....then the app bombs because the sqlite db's in the profile are all in use.

I was wondering if you had any updated documentation that could be added. The most current documentation is for version 1.3.0 4 years ago, and I know there has been a lot of things added since then.

Thanks!

Congratulations 😃

when i try to check the path with special character have an error.

Make new class for ContentSettingItem

could be simple human error but I try to run the -i and -o option and I continue to get error line 224 except ImportError, e:

Describe the problem
hindisght.py isn't executable directly from pypi package. There are a couple small issues: The first one is that it doesn't have the executable bit set. After making it executable though, it still complains about the interpreter not existing, which looks like it comes from windows newlines in the file. Executing directly with python works just fine.

Screenshots or Console Output

$ virtualenv hindsighttest
$ . hindsighttest/bin/activate
$ pip install pyhindsight
$ which hindsight
$ chmod 755 hindsighttest/bin/hindsight.py
$ hindsight.py -h
/usr/bin/env: ‘python3\r’: No such file or directory

$ hexdump -C hindsighttest/bin/hindsight.py | head -n 2
00000000  23 21 2f 75 73 72 2f 62  69 6e 2f 65 6e 76 20 70  |#!/usr/bin/env p|
00000010  79 74 68 6f 6e 33 0d 0a  0d 0a 22 22 22 48 69 6e  |ython3...."""Hin|

Expected behavior
Hindsight being executable after installation.

To Reproduce
See console output.

hindsight.log Snippet
N/A

System Details

$ uname -a
Linux myhostname 5.7.17-1-amd64 #1 SMP Debian 5.7.17-1 (2020-09-16) x86_64 GNU/Linux

Additional context
N/A

Hello,

When I run hindsight I get the following error:

$ ./hindsight.py -i ... -o output -f xlsx -l log.txt
Traceback (most recent call last):
  File "./hindsight.py", line 16, in <module>
    from pyhindsight.analysis import AnalysisSession
  File "/Users/user/Downloads/hindsight-master/pyhindsight/analysis.py", line 12, in <module>
    from pyhindsight.browsers.chrome import Chrome
  File "/Users/user/Downloads/hindsight-master/pyhindsight/browsers/chrome.py", line 22, in <module>
    import keyring
  File "/Users/user/.pyenv/versions/2.7.16/lib/python2.7/site-packages/keyring-19.2.0-py2.7.egg/keyring/__init__.py", line 1, in <module>
    from .core import (
  File "/Users/user/.pyenv/versions/2.7.16/lib/python2.7/site-packages/keyring-19.2.0-py2.7.egg/keyring/core.py", line 10, in <module>
    from . import backend
  File "/Users/user/.pyenv/versions/2.7.16/lib/python2.7/site-packages/keyring-19.2.0-py2.7.egg/keyring/backend.py", line 38
    class KeyringBackend(metaclass=KeyringBackendMeta):
                                  ^
SyntaxError: invalid syntax

The problem comes from the version of keyring which is installed, i.e. 19.2.0. I think they have dropped support for Python 2 in version 19 (here). I've changed setup.py to install version 18.0.0 and that solved the issue.

    install_requires=[
        'keyring==18.0.0',

I use macOS 10.14.6 and Python 2.7.16.

Bottle v0.12.13 server starting up (using WSGIRefServer())...
Listening on http://localhost:8080/
Hit Ctrl-C to quit.

No handlers could be found for logger "pyhindsight.analysis"
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET / HTTP/1.1" 200 18564
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/styles.css HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/github.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/slack.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/twitter.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/h.png HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/fonts/OpenSans-Regular.woff2 HTTP/1.1" 304 0
127.0.0.1 - - [14/Jun/2018 16:31:13] "GET /static/fonts/OpenSans-Light.woff2 HTTP/1.1" 304 0
                     Detected Chrome version:            [     66 ]
                                 URL records:            [   6815 ]
                            Download records:            [    162 ]
                               Cache records:            [  12119 ]
                           GPU Cache records:            [    490 ]
                         Media Cache records:            [    478 ]
                   Application Cache records:            [      0 ]
                              Cookie records:            [   5561 ]
                            Autofill records:            [   6366 ]
                            Bookmark records:            [      3 ]
                       Local Storage records:            [   2240 ]
                                  Extensions:            [     11 ]
                    Extension Cookie records:            [      0 ]
                          Login Data records:            [     51 ]
Traceback (most recent call last):
  File "/home/user/.local/lib/python2.7/site-packages/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/home/user/.local/lib/python2.7/site-packages/bottle.py", line 1740, in wrapper
    rv = callback(*a, **ka)
  File "hindsight_gui.py", line 163, in do_run
    analysis_session.run()
  File "/home/user/src/hindsight/pyhindsight/analysis.py", line 129, in run
    browser_analysis.process()
  File "/home/user/src/hindsight/pyhindsight/browsers/chrome.py", line 1713, in process
    self.get_preferences(self.profile_path, 'Preferences')
  File "/home/user/src/hindsight/pyhindsight/browsers/chrome.py", line 1240, in get_preferences
    pref_item = Chrome.PreferenceItem(url='', timestamp=to_datetime(timestamp[0], self.timezone),
TypeError: 'float' object has no attribute '__getitem__'
127.0.0.1 - - [14/Jun/2018 16:31:44] "POST /run HTTP/1.1" 500 1905

Any insight?

For example, given a starting point of

C:\Temp\tout

where there exists profiles like this:

C:\Temp\tout\C\Users\e\AppData\Local\Google\Chrome\User Data\Default

can you change hindsight to discover the profiles that exist, vs me having to manually specify and know what profiles exist?

we have a module for KAPE that uses hindsight but it requires me to run it against the second dir vs the first, so its not ideal.

would be nice to point hindsight to a dir and have it tell me whats there, maybe via a -r switch or something?

Hi Ryan. I'm working with data from a Windows machine, and I'm getting the following traceback error:

Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 438, in run
File "pyhindsight\browsers\chrome.py", line 1852, in process
File "pyhindsight\browsers\chrome.py", line 827, in get_extensions
WindowsError: [Error 267] The directory name is invalid: 'W:\Parent_directory\User Data\Default\Extensions\mallpejgeafdahhflmliiahjdpgbegpk/.'

Can you help with this error? Or if this error is not fatal to results-output, is there a way for Hindsight to exit despite the error, so I can get the parsed results? When I look at the CLI, I see that a bunch of records were parsed.

Ubuntu 15.10

./hindsight.py -i '../chromium' -o test

Couldn't import module 'xlsxwriter'; XLSX output disabled.

Couldn't import module 'win32crypt'; cookie decryption on Windows disabled.

Couldn't import module 'pytz'; all timestamps in XLSX output will be in examiner local time (CEST).

Hindsight v1.6.0
Start time: 2016-04-08 20:45:17.328
Input directory: ../chromium
Output name: test

Processing:
Detected Chrome version: [ 31-49 ]

Running plugins:
Google Searches (v20150728): - 0 searches parsed -
Generic Timestamp Decoder (v20140816): - 0 timestamps parsed -
Chrome Extension Names (v20150125): - 0 extension URLs parsed -
Time Discrepancy Finder (v20141214): - 0 differences parsed -
Query String Parser (v20150222): - 0 query strings parsed -
Google Analytics Cookie Parser (v20140813): - 0 cookies parsed -
Quantcast Cookie Parser (v20140813): - 0 cookies parsed -

Writing "test.sqlite"
Traceback (most recent call last):
File "./hindsight.py", line 2139, in
main()
File "./hindsight.py", line 2133, in main
write_sqlite(target_browser)
File "./hindsight.py", line 2041, in write_sqlite
for extension in browser.installed_extensions['data']:
TypeError: list indices must be integers, not str

Hindsight can't open SQLite dbs that are locked and in use. It should be feasible to make a temp copy of the locked database and use that instead. The same logic might apply to LevelDB directories, but I need to investigate more.

"shared_proto_db" under the profile account contains a LevelDB database containing downloaded URLs with a timestamp and a local filename. Very useful for tracking malware source

Primitive strings can be gathered from both .log and .ldb files like so
https://gist.github.com/Allong12/c164245ebabf98d134e73ee45cbee7b1

(created on behalf of @Allong12 - thanks for the idea!)

Describe the problem
Getting errors when trying to parse data

Screenshots or Console Output
Traceback (most recent call last):
File "hindsight.py", line 329, in
main()
File "hindsight.py", line 212, in main
run_status = analysis_session.run()
File "/home/alex/hindsight/pyhindsight/analysis.py", line 479, in run
browser_analysis.process()
File "/home/alex/hindsight/pyhindsight/browsers/chrome.py", line 2210, in process
self.get_site_characteristics(self.profile_path, 'Site Characteristics Database')
File "/home/alex/hindsight/pyhindsight/browsers/chrome.py", line 2000, in get_site_characteristics
items = utils.get_ldb_records(sc_root_path)
File "/home/alex/hindsight/pyhindsight/utils.py", line 159, in get_ldb_records
for record in db.iterate_records_raw():
File "/home/alex/hindsight/pyhindsight/lib/ccl_chrome_indexeddb/ccl_leveldb.py", line 572, in iterate_records_raw
yield from file_containing_records
File "/home/alex/hindsight/pyhindsight/lib/ccl_chrome_indexeddb/ccl_leveldb.py", line 334, in iter
for batch_offset, batch in self._get_batches():
File "/home/alex/hindsight/pyhindsight/lib/ccl_chrome_indexeddb/ccl_leveldb.py", line 329, in _get_batches
raise ValueError() # Cannot happen
ValueError

Expected behavior
Cannot happen

To Reproduce
Steps to reproduce the behavior:
Run script from the command line

hindsight.log Snippet
2021-02-20 13:57:48.552 | D | - Reading 6479 Local Storage raw LevelDB records; beginning parsing
2021-02-20 13:57:48.632 | I | - Parsed 5953 items from 0 files
2021-02-20 13:57:48.633 | I | Extensions:
2021-02-20 13:57:48.633 | I | - Reading from /home/XXXX/.config/google-chrome/Default/Extensions
2021-02-20 13:57:48.634 | D | - 11 files in Extensions directory: ['nmmhkkegccagdldgiimedpiccmgmieda', 'felcaaldnbdncclmgdcncolpebgiejap', 'blpcfgokakmgnkcojhhkbfbldkacnbeo', 'pkedcjkdefgpdelpbcmbmeomcjbeemfm', 'phfccdiigmfnmeejlcjpafpgmoppjpok', 'ghbmnnjooekpmoecnnnilnnbdlolhkhi', 'apdfllckaahabafndbhieahigkjlhalf', 'aohghmighlieiainnegkcijnfilokake', 'pjkljhegncpnkpknbcohdijeoejaedia', 'aapocclcgogkmnckokdopfmhonfmgoek', 'lmcianjdpdgjjpcnaaohnnepnbiagjii']
2021-02-20 13:57:48.634 | D | - 11 files in Extensions directory will be processed: ['nmmhkkegccagdldgiimedpiccmgmieda', 'felcaaldnbdncclmgdcncolpebgiejap', 'blpcfgokakmgnkcojhhkbfbldkacnbeo', 'pkedcjkdefgpdelpbcmbmeomcjbeemfm', 'phfccdiigmfnmeejlcjpafpgmoppjpok', 'ghbmnnjooekpmoecnnnilnnbdlolhkhi', 'apdfllckaahabafndbhieahigkjlhalf', 'aohghmighlieiainnegkcijnfilokake', 'pjkljhegncpnkpknbcohdijeoejaedia', 'aapocclcgogkmnckokdopfmhonfmgoek', 'lmcianjdpdgjjpcnaaohnnepnbiagjii']
2021-02-20 13:57:48.637 | I | - Parsed 11 items
2021-02-20 13:57:48.637 | I | Cookie items from Extension Cookies:
2021-02-20 13:57:48.637 | I | - Using SQL query for Cookie items for Chrome v66
2021-02-20 13:57:48.637 | I | - Reading from Extension Cookies in /home/XXXX/.config/google-chrome/Default
2021-02-20 13:57:48.638 | I | - Parsed 0 items
2021-02-20 13:57:48.638 | I | Login items from Login Data:
2021-02-20 13:57:48.638 | I | - Using SQL query for Login items for Chrome v78
2021-02-20 13:57:48.638 | I | - Reading from Login Data in /home/XXXX/.config/google-chrome/Default
2021-02-20 13:57:48.640 | I | - Using SQL query for Login Stat items for Chrome v48
2021-02-20 13:57:48.640 | I | - Reading from Login Data in /home/XXXX/.config/google-chrome/Default
2021-02-20 13:57:48.640 | I | - Parsed 44 items
2021-02-20 13:57:48.640 | I | Preferences:
2021-02-20 13:57:48.641 | I | - Reading from /home/XXXX/.config/google-chrome/Default/Preferences
2021-02-20 13:57:48.645 | I | - Parsed 14 items
2021-02-20 13:57:48.760 | I | Site Characteristics:
2021-02-20 13:57:48.760 | I | - Reading from /home/XXXX/.config/google-chrome/Default/Site Characteristics Database
2021-02-20 13:57:48.761 | D | - 6 files in Site Characteristics directory: ['MANIFEST-000001', 'LOG', 'CURRENT', 'LOG.old', '000003.log', 'LOCK']

System Details

• Analysis System OS (your forensic workstation): "Ubuntu 20.04.2 LTS"
• Method of Running Hindsight: python 3.8.5
• Hindsight version: v2021.01.16
• Target System OS: "Ubuntu 20.04.2 LTS"
• Target Browser: Chrome
• Target Browser Version: Google Chrome 87.0.4280.88

Additional context
Add any other context about the problem here.

Traceback (most recent call last):
File "hindsight.py", line 1919, in
main()
File "hindsight.py", line 1913, in main
write_sqlite(target_browser)
File "hindsight.py", line 1731, in write_sqlite
item.interpretation))
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

Hi Ryan. While parsing an exported Chrome\default directory, I encountered an exception error "KeyError (MediaKey) with a traceback output of:

Traceback (most recent call last):
File "bottle.py", line 868, in _handle
File "bottle.py", line 1748, in wrapper
File "hindsight_gui.py", line 187, in display_results
File "bottle.py", line 3630, in template
File "bottle.py", line 3419, in render
File "bottle.py", line 3406, in execute
File "C:\Users\LMARTI1\AppData\Local\Temp_MEI287322\templates\results.tpl", line 30, in
% include('templates/parsed_artifacts.tpl')
File "bottle.py", line 3396, in _include
File "bottle.py", line 3406, in execute
File "C:\Users\LMARTI1\AppData\Local\Temp_MEI287322\templates\parsed_artifacts.tpl", line 29, in
{{artifacts_counts[artifact]}}
KeyError: 'Media History'

Just prior to this error, the following was showing in my terminal window:

Detected Chrome version: [ 84 ]
URL records: [ 25124 ]
Download records: [ 268 ]
Media History records: [ 0 ]
GPU Cache records: [ 0 ]
Cookie records: [ 5931 ]
Autofill records: [ 851 ]
Bookmark records: [ 24 ]
Local Storage records: [ 10420 ]
Extensions: [ 12 ]
Extension Cookie records: [ 8 ]
Login Data records: [ 160 ]
Preference Items: [ 131 ]
File System Items: [ 41 ]
Chrome Extension Names (v20150125): - 0 extension URLs parsed -
Generic Timestamp Decoder (v20160907): - 0 timestamps parsed -
Google Analytics Cookie Parser (v20170130): - 0 cookies parsed -
Google Searches (v20160912): - 2353 searches parsed -
Load Balancer Cookie Decoder (v20200213): - 0 cookies parsed -
Quantcast Cookie Parser (v20160907): - 0 cookies parsed -
Query String Parser (v20170225): - 8332 query strings parsed -
Time Discrepancy Finder (v20170129): - 0 differences parsed -
127.0.0.1 - - [20/Apr/2021 15:06:09] "POST /run HTTP/1.1" 303 0

Could you help me with this issue, please? Thanks.

Hello all,
So i'm haing a weird problem, when running the hindsight.py in my linux envoirment I will be able to use this tool normally, sadly I need this to run in windows. When I attempt to do so (through the command line) with the same script and python versions etc. I believe the reason is the "error" is the line shown below. If anyone can help me out that would be awesome.
No handlers could be found for logger "pyhindsight.analysis"

A different structure for Chrome caches has been slowly rolling out, and we should add support for parsing it. It's already in use in Windows Chrome "Code Cache" (https://dfir.blog/chrome-evolution/), macOS Chrome (since ~72 - #65), and Linux Chrome for a while.

References:

• libyal/dtformats#10
• https://chromium.googlesource.com/chromium/src/net/+/master/disk_cache/simple/simple_entry_format.h#18

Great software.

I needed to install the pytz pacakge, which is not mentioned in the documentation.

https://pypi.python.org/pypi/pytz/

Thanks.

Hi,
I get this error after a while that it's processing, any hint ?
Thanks
Lorenzo

Error: 500 Internal Server Error

Sorry, the requested URL 'http://localhost:8080/run' caused an error:

Internal Server Error

Exception:

UnboundLocalError("local variable 'entry' referenced before assignment",)

Traceback:

Traceback (most recent call last):
File "bottle.py", line 862, in _handle
File "bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 161, in do_run
File "pyhindsight\analysis.py", line 127, in run
File "pyhindsight\browsers\chrome.py", line 1380, in process
File "pyhindsight\browsers\chrome.py", line 1057, in get_cache
UnboundLocalError: local variable 'entry' referenced before assignment

Using version 2.0.0, both hindsight.exe and the Web front-end, hingsight_gui.exe, produces the same error message and the app fails.

Error produced by hindsight.exe

Input directory: c:\Users\cms\Documents\Cases\HindsightTest\Export\User Data\Default
Output name: HindsightTest.xlsx
Processing:
Detected Chrome version: [ 54-56 ]
URL records: [ 272 ]
Download records: [ 27 ]
Traceback (most recent call last):
File "hindsight.py", line 286, in
File "hindsight.py", line 172, in main
File "pyhindsight\analysis.py", line 135, in run
File "pyhindsight\browsers\chrome.py", line 1169, in process
File "pyhindsight\browsers\chrome.py", line 1019, in get_cache
File "pyhindsight\browsers\chrome.py", line 1801, in init
File "pyhindsight\browsers\chrome.py", line 1842, in keyToStr
File "pyhindsight\browsers\chrome.py", line 1642, in data
AttributeError: CacheAddress instance has no attribute 'blockNumber'
Failed to execute script hindsight

Error produced by hindsight_gui.exe

Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:
Internal Server Error
Exception:
AttributeError("CacheAddress instance has no attribute 'blockNumber'",)
Traceback:
Traceback (most recent call last):
File "bottle.py", line 862, in _handle
File "bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 161, in do_run
File "pyhindsight\analysis.py", line 135, in run
File "pyhindsight\browsers\chrome.py", line 1169, in process
File "pyhindsight\browsers\chrome.py", line 1019, in get_cache
File "pyhindsight\browsers\chrome.py", line 1801, in init
File "pyhindsight\browsers\chrome.py", line 1842, in keyToStr
File "pyhindsight\browsers\chrome.py", line 1642, in data
AttributeError: CacheAddress instance has no attribute 'blockNumber'

I was testing the app using several different Chrome v56 profiles and each profile was extracted in its entirety from a Windows 10 user profile. I am not running the app against a mount point, such as with FTK Imager mount or EnCase PDE. The hindsight log file is attached for your reference.

hindsight.log.txt

If you need additional information or other tests run, feel free to ping me.

Thanks.

/Chris

'unicodeescape' codec can't decode bytes in position 74-75: truncated \UXXXXXXXX escape

Hi, I want to extract my data from local files. However, an error occurs. May I know what's wrong? And What shall I do?

Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 194, in run
File "pyhindsight\analysis.py", line 157, in find_browser_profiles
File "pyhindsight\analysis.py", line 145, in find_browser_profiles
WindowsError: [Error 5] Access is denied: 'D:\System Volume Information/.'

Hi Ryan. I just tried to use Hindsight to parse Chrome (Android). I received the following error:

Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 163, in do_run
File "pyhindsight\analysis.py", line 129, in run
File "pyhindsight\browsers\chrome.py", line 1721, in process
File "pyhindsight\browsers\chrome.py", line 1312, in get_cache
UnboundLocalError: local variable 'cacheBlock' referenced before assignment

The full path I had to use was: [parent dir]\com.android.chrome\app_chrome\Default

I could not use: [parent dir]\com.android.chrome\app_chrome\ (this is what is suggested when I run from the GUI. When I use this path, I don't get any error whatsoever. I only get 0 results.

Am I doing something wrong? This isn't the first time I've receive this message. I received a similar message attempting to parse chrome from another Android.

Please let me know what I can do to help.

Thank you.

It would be really nice to see this project packaged and push to pip, would be easier to install and use.

Yosemite 10.10.3
Python 2.7.10 (installed via brew, also using a virtualenv for hindsight)

Running hindsight gives an ascii error when writing the xlsx file. If I change the output format to json, all is well.

python hindsight.py -i '/Users/infosec/Library/Application Support/Google/Chrome/Default' -o test2
Couldn't import module 'win32crypt'; cookie decryption on Windows disabled.

Hindsight v1.4.5
Start time: 2015-06-19 08:03:31.687
Input directory: /Users/infosec/Library/Application Support/Google/Chrome/Default
Output name: test2

Processing:
Detected Chrome version: [ 42 ]
URL records: [ 21298 ]
Download records: [ 441 ]
Cookie records: [ 4974 ]
Autofill records: [ 1554 ]
Bookmark records: [ 1132 ]
Local Storage records: [ 24290 ]
Extensions: [ 14 ]
Login Data records: [ 11 ]
Preference Items: [ 20 ]

Running plugins:
Chrome Extension Names (v20150125): - 8 extension URLs parsed -
Generic Timestamp Decoder (v20140816): - 99 timestamps parsed -
Google Analytics Cookie Parser (v20140813): - 154 cookies parsed -
Google Searches (v20150222): - 1423 searches parsed -
Quantcast Cookie Parser (v20140813): - 14 cookies parsed -
Query String Parser (v20150222): - 5282 query strings parsed -
Time Discrepancy Finder (v20141214): - 73 differences parsed -

Writing "test2.xlsx"
Traceback (most recent call last):
File "hindsight.py", line 1920, in
main()
File "hindsight.py", line 1901, in main
write_excel(target_browser)
File "hindsight.py", line 1652, in write_excel
workbook.close()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 295, in close
self._store_workbook()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 518, in _store_workbook
xml_files = packager._create_package()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/packager.py", line 140, in _create_package
self._write_shared_strings_file()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/packager.py", line 280, in _write_shared_strings_file
sst._assemble_xml_file()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/sharedstrings.py", line 53, in _assemble_xml_file
self._write_sst_strings()
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/sharedstrings.py", line 83, in _write_sst_strings
self._write_si(string)
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/sharedstrings.py", line 110, in _write_si
self._xml_si_element(string, attributes)
File "/Users/Infosec/envs/hindsight/lib/python2.7/site-packages/xlsxwriter/xmlwriter.py", line 122, in _xml_si_element
self.fh.write("""<t%s>%s""" % (attr, string))
File "/Users/Infosec/envs/hindsight/lib/python2.7/codecs.py", line 694, in write
return self.writer.write(data)
File "/Users/Infosec/envs/hindsight/lib/python2.7/codecs.py", line 357, in write
data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 64: ordinal not in range(128)

Recent versions of FTK Imager list deleted files in the directory structure of logically mounted images. These "non-existent" files cause many programs to fail after handles are requested. Hindsight gives a "WindowsError: [Error 2] The system cannot find the file specified: " error (in my case the error occurred in the get_local_storage function).

The current workaround is to either use a different mounting tool or export the files out of the image.

When Hindsight scans a directory with subdirectories inaccessible due to NTFS permissions, it aborts with an error. There is no easy way to circumvent this on my side, as the partition is mounted as read only. Perhaps it would be better to log an error, but continue processing other subdirectories?

This is related to the improvement made in #24 (suggested by @EricZimmerman) and #34.

Screenshots or Console Output

[9868] Failed to execute script hindsight
Traceback (most recent call last):
  File "pyhindsight\analysis.py", line 365, in search_subdirs
PermissionError: [WinError 5] Access is denied: 'M:\\Users\\All Users\\Application Data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hindsight.py", line 327, in <module>
  File "hindsight.py", line 203, in main
  File "pyhindsight\analysis.py", line 393, in find_browser_profiles
  File "pyhindsight\analysis.py", line 376, in search_subdirs
  File "pyhindsight\analysis.py", line 376, in search_subdirs
  File "pyhindsight\analysis.py", line 368, in search_subdirs
TypeError: unsupported format string passed to PermissionError.__format__

To Reproduce
Set --input to a directory that contains subdirectories that cannot be accessed due to NTFS permissions. In the example above, hindsight was started with --input M:\Users.

System Details

• Windows 10
• Method of Running Hindsight: hindsight.exe (both with elevated privileges and as a regular user)
• Hindsight version: 20200607

Would it be possible to have Hindsight recursively search a given path and parse each profile? For example
ideally something like this would work so it would grab each Windows profile and then also if there are multiple Chrome user profiles it would parse them too

hindsight.exe -i C:\ -o test_run
hindsight.exe -i C:\Users -o test_run

Hello,

First of all, thank you for your App, it seems pretty great.
I've followed your installation instruction from the PDF and just adapted them to my computer running Mac OS X and it was pretty straight forward using MacPort. I've been able to run Hindsight and it seems to work. Unfortunately, no data is retrieved. I thought that it was a compatibility problem with Chrome 39 but you saying that your app is up-to-date. Could you please let me know what I am doing wrong ?

computerName:hindsight-master userName$ /Users/userName/Downloads/hindsight-master/hindsight.py -i /Users/userName/Library/Application\ Support/Google/Chrome
Couldn't import module 'win32crypt'; cookie decryption on Windows disabled.

Hindsight v1.4.0
Start time: 2014-12-20 10:30:41.690
Input directory: /Users/userName/Library/Application Support/Google/Chrome
Output name: Hindsight Internet History Analysis (2014-12-20T10-30-41)

Processing:
Detected Chrome version: [ 1-39 ]

Running plugins:
Chrome Extension Names (v20140813): - 0 extension URLs parsed -
Generic Timestamp Decoder (v20140816): - 0 timestamps parsed -
Google Analytics Cookie Parser (v20140813): - 0 cookies parsed -
Google Searches (v20141103): - 0 searches parsed -
Quantcast Cookie Parser (v20140813): - 0 cookies parsed -
Time Discrepancy Finder (v20141214): - 0 differences parsed -

Writing "Hindsight Internet History Analysis (2014-12-20T10-30-41).xlsx"

Finish time: 2014-12-20 10:30:41.714

I don't really get it since content should be here... I'm using it to try to see which informations are sent by Google Sync and see what is recoverable on a device that is actually not the one creating the history... But the history files should be here, as well as the extensions, and searches at list, right ?

Sorry for my bad english, thank you for your time !!

Hi. I am trying to parse data exported from a forensic image, to my case folder. I encounter the following traceback output:

Traceback (most recent call last):
File "bottle.py", line 862, in _handle
File "bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 161, in do_run
File "pyhindsight\analysis.py", line 127, in run
File "pyhindsight\browsers\chrome.py", line 1335, in process
IndexError: list index out of range

Exception is: IndexError('list index out of range',)

When I open chrome.py, and inspect line 1335, it appears there is a trigger for run failure, if a cache-related circumstance exists in the source. Am I on the right path with this interpretation? Am I doing something wrong when running Hindsight?

I ran both hindsight_gui.py and the command line version hindsight.py to test this issue, with identical results.

Running from git master at commit 13b7a64 (when run, it announces v20201120) in macOS Catalina under Python 3.9.0 with target browser macOS Catalina Chrome Version 87.0.4280.141, it found 0 cache records.

I specified Input Path: /Users/sidney/Library/Application Support/Google/Chrome/Default and Cache Path: /Users/sidney/Library/Caches/Google/Chrome/Default/Cache/.

I ran hindsight_gui.py and entered the above paths in the browser GUI window, then clicked Run.
For the command line form of the test, I used the command

python hindsight.py -i "/Users/sidney/Library/Application Support/Google/Chrome/Default" -o ~/tmp/hindsight -b Chrome -l hindsight.log -c "/Users/sidney/Library/Caches/Google/Chrome/Default/Cache/" --temp_dir hindsight_tmp

Here are the first lines of Parsed Artifacts in the Results page in the GUI version

Detected Chrome version: | 87 | 
URL records: | 21068 | 
Download records: | 958 | 
Cache records: | 0 | 

The command line version output the same numbers.

I expected to get more than 0 cache records.

When I looked in hindsight.log I found (snipping out timestamps for brevity)

| I | Cache items from /Users/sidney/Library/Caches/Google/Chrome/Default/Cache/:
| E |  - Failed to parse index block file

Steps to reproduce:

1. In macOS Catalina (didn't try elsewhere) with Chrome version 87, run Chrome so there should be something in its cache.
2. Either run the hindsight.py command line as shown above, or follow the rest of these steps for the GUI version
3. Start up hindsght_gui from the command line using python 3.9.0
4. In a browser, go to http://localhost:8080/ to access the hindsight GUI
5. Enter /Users/YOURUSERNAME/Library/Application Support/Google/Chrome/Default in the Input Path box
6. Enter /Users/YOURUSERNAME/Library/Caches/Google/Chrome/Default/Cache in the Cache Path box
7. Click on the Run button

Expected results: more than 0 cache records found
Actual results: 0 cache records found, and hindsight.log shows an error Failed to parse index block file

I'm receiving an error "Could not lock file" in two scenarios:

1. Running FTK Imager v 4.1.1 and attempting to parse the Chrome artifacts on several different images.
   Images are mounted in FTK as File System/Read Only.

2. Drive mounted with Arsenal Image Mounter. The same image that failed to parse with FTK works with Arsenal, BUT if I mount the Volume Shadow copies and try to scan one of these, I get the same error.

       Start time: 2018-10-03 17:24:51.216
  Input directory: i:\[root]\users\Administrator\AppData\Local\Google\Chrome\User Data\Default
      Output name: c:\cases\Desktop-FTK\Administrator\chrome.xlsx

 Processing:
                     Detected Chrome version:            [     66 ]            
                                 URL records:            [     71 ]            
                            Download records:            [      4 ]            
                               Cache records:            [   1671 ]            
                           GPU Cache records:            [      0 ]            
                         Media Cache records:            [      0 ]            
                              Cookie records:            [    414 ]            
                            Autofill records:            [      7 ]            
hindsight.exe : [6176] Failed to execute script hindsight
At line:1 char:1
+ & "C:\utilities\Hindsight\hindsight.exe" -i  "i:\[root]\users\Adminis ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ([6176] Failed t...cript hindsight:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Traceback (most recent call last):
  File "hindsight.py", line 293, in <module>
  File "hindsight.py", line 180, in main
  File "pyhindsight\analysis.py", line 129, in run
  File "pyhindsight\browsers\chrome.py", line 1757, in process
  File "pyhindsight\browsers\chrome.py", line 710, in get_local_storage
  File "pyhindsight\browsers\chrome.py", line 1432, in get_prefixed_leveldb_pairs
leveldb.LevelDBError: IO error: i:\[root]\users\Administrator\AppData\Local\Google\Chrome\User Data\Default\Local Storage\leveldb\LOCK: Could not lock file.

Same error using the GUI

Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:

Internal Server Error
Exception:
LevelDBError('IO error: I:\\[root]\\Users\\mpowers\\AppData\\Local\\Google\\Chrome\\User Data\\Default\\Local Storage\\leveldb\\LOCK: Could not lock file.',)
Traceback:
Traceback (most recent call last):
  File "site-packages\bottle.py", line 862, in _handle
  File "site-packages\bottle.py", line 1732, in wrapper
  File "hindsight_gui.py", line 163, in do_run
  File "pyhindsight\analysis.py", line 129, in run
  File "pyhindsight\browsers\chrome.py", line 1757, in process
  File "pyhindsight\browsers\chrome.py", line 710, in get_local_storage
  File "pyhindsight\browsers\chrome.py", line 1432, in get_prefixed_leveldb_pairs
LevelDBError: IO error: I:\[root]\Users\mpowers\AppData\Local\Google\Chrome\User Data\Default\Local Storage\leveldb\LOCK: Could not lock file.

Every time i try to provide the Default path of my Chrome, it results in server error (500) stating that the file doesn't exists. Although it does.

The end for Python 2.7 is in sight - I need to migrate Hindsight to Python 3.

Describe the problem
When executing hindsight.py from a different directory from where hindsight.py is installed, output files get saved to different directories depending on what output type you have. JSONL files get saved to the CWD, while XSLX files get saved to the same directory where hindsight.py is. When running out of a virtualenv this is a little confusing because they end up in the virtualenv/bin directory.

Screenshots or Console Output
N/A

Expected behavior
Output files in both cases saved to the current working directory.

To Reproduce
Create virtualenv, activate virtualenv, pip install pyhindsight, and from a directory other than the virtualenv/bin directory execute hindsight with an output file type of xslx.

hindsight.log Snippet
N/A

System Details

$ uname -a
Linux hostname 5.7.17-1-amd64 #1 SMP Debian 5.7.17-1 (2020-09-16) x86_64 GNU/Linux

Additional context
N/A

Dear sirs:
I would like to know what has happened when I tried to started Hinsight, since I followed all steps. It's my very first time using this programme.

Error: 500 Internal Server Error
Sorry, the requested URL 'http://localhost:8080/run' caused an error:

Internal Server Error
Exception:
AttributeError("'NoneType' object has no attribute 'selected_plugins'",)
Traceback:
Traceback (most recent call last):
File "site-packages\bottle.py", line 862, in _handle
File "site-packages\bottle.py", line 1732, in wrapper
File "hindsight_gui.py", line 130, in do_run
AttributeError: 'NoneType' object has no attribute 'selected_plugins'

Best regards from Spain
Carlos M.S.

Hello,

I got few errors on windows and linux about python modules i already installed all missed modules except "win32crypt"

I a got this error
"Couldn't import module 'win32crypt'; cookie decryption on Windows disabled."

Couldn't find a way to install this module to get cookies decrypted.

If you specify an output folder path that doesn't exist, it fails at the end of the run. Should either create the folder for you or at least check for existence before running.