I was getting the OOM thing in Kali so I'm trying in Cygwin.

After installing all the required packages, running ./synclibs.sh, ./autogen.sh and ./configure, I get the following when I run make:

libtool: link: gcc -g -O2 -Wall -o .libs/esedbexport.exe esedbexport.o esedboutput.o exchange.o export.o export_handle.o log_handle.o webcache.o windows_search.o windows_security.o  ../libcsystem/.libs/libcsystem.a ../libfmapi/.libs/libfmapi.a ../libfvalue/.libs/libfvalue.a ../libfwnt/.libs/libfwnt.a ../libfguid/.libs/libfguid.a ../libfdatetime/.libs/libfdatetime.a ../libcfile/.libs/libcfile.a ../libcpath/.libs/libcpath.a ../libuna/.libs/libuna.a ../libcsplit/.libs/libcsplit.a ../libesedb/.libs/libesedb.dll.a ../libcnotify/.libs/libcnotify.a ../libclocale/.libs/libclocale.a ../libcerror/.libs/libcerror.a ../libcstring/.libs/libcstring.a -lintl -L/usr/local/lib
../libcnotify/.libs/libcnotify.a(libcnotify_verbose.o): In function `libcnotify_verbose_set':
/home/ttalap/libesedb/libcnotify/libcnotify_verbose.c:36: multiple definition of `libcnotify_verbose_set'
../libesedb/.libs/libesedb.dll.a(d000265.o):(.text+0x0): first defined here
../libclocale/.libs/libclocale.a(libclocale_codepage.o): In function `libclocale_codepage_copy_from_string':
/home/ttalap/libesedb/libclocale/libclocale_codepage.c:130: multiple definition of `libclocale_codepage_copy_from_string'
../libesedb/.libs/libesedb.dll.a(d000250.o):(.text+0x0): first defined here
collect2: error: ld returned 1 exit status
Makefile:725: recipe for target 'esedbexport.exe' failed
make[1]: *** [esedbexport.exe] Error 1
make[1]: Leaving directory '/home/ttalap/libesedb/esedbtools'
Makefile:768: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

• add option to read backup catalog instead of catalog

I tried both Sections in https://github.com/libyal/libesedb/wiki/Building:
"Using Minimalist GNU for Windows (MinGW)" and "Using MSYS-MinGW"
(Yes, I use the msys shell and not the native Windows shell)

The problem is
" ./configure . No such file or directory"

make reports that no makefile is present, even though makefile.am is in the directory.

I get the following when attempting to export all tables from a windows 10 windows.edb file.
Using "libesedb-utils 20170121-4" from Ubuntu Boinic.

Any suggestions ?
----------------cmd line output ---------------
esedbexport -m all -v Windows.edb
esedbexport 20170121

Opening file.
Exporting table 1 (MSysObjects) out of 41.
Exporting index 2 (Name).
Exporting index 3 (RootObjects).
Exporting table 2 (MSysObjectsShadow) out of 41.
Exporting table 3 (MSysObjids) out of 41.
Exporting table 4 (MSysLocales) out of 41.
Exporting table 5 (CatalogManager_Properties) out of 41.
Exporting table 6 (CatalogStorageManager) out of 41.
Exporting table 7 (SystemIndex_Gthr) out of 41.
Exporting index 2 (indexDocID).
Exporting index 3 (indexRecovery).
Exporting table 8 (SystemIndex_GthrPth) out of 41.
Exporting table 9 (SystemIndex_GthrAppOwner) out of 41.
Exporting table 10 (SystemIndex_1_Properties) out of 41.
Exporting table 11 (SystemIndex_1) out of 41.
Exporting table 12 (SystemIndex_PropertyStore) out of 41.
Unable to export file.
libuna_unicode_character_copy_to_utf8: UTF-8 string too small.
libuna_utf8_string_with_index_copy_from_byte_stream: unable to copy Unicode character to UTF-8.
libfvalue_string_copy_to_utf8_string_with_index: unable to copy byte stream to UTF-8 string.
libfvalue_value_copy_to_utf8_string_with_index: unable to copy instance to UTF-8 string.
libfvalue_value_copy_to_utf8_string: unable to copy value: 0 to UTF-8 string.
libesedb_record_value_get_utf8_string: unable to copy value to UTF-8 string.
libesedb_long_value_get_utf8_string: unable to retrieve UTF-8 string from record value.
export_handle_export_long_record_value: unable to retrieve value string: 305.
export_handle_export_record_value: unable to export long record value: 305.
export_handle_export_record: unable to export record value: 305.
export_handle_export_table: unable to export record.
export_handle_export_file: unable to export table: 11.

The same edb file

The results are different.
ESEDatabaseView download : http://www.nirsoft.net/utils/ese_database_view.html

Thank you.

I built the libraries via msvcpp solution, but when I call any function (even libesedb_file_initialize(..)) I receive LNK2019 error. All the libs are linked to the project.

Hi, Joachim

First of all, thanks for open sourcing this wonderful ESE parser! I am currently building a WebCache parser using your ESE module, everything looks good except occasionally the module returns a record count of 0 for tables with only 1 record. I have attached a sample WebCahce file below where Container_30 and LeakFiles both return a count of 0, but using tools like ESEDatabaseViewer reveal they actually contain 1 record.

WebCacheV01_RecordCountIssue.zip

Thanks,
Calvin

root@Kali-LABS:~/esa# esedbexport
esedbexport 20141129

Over my last 3 pentests, I have had ntds.dit files of 7.6GB, 5.3GB and 400MB.

The 400MB extracts super fast, no noticeable difference from last release. However the larger ones, if I let them run would take weeks to extract. Using the last release of libesedb it take a few hours (10 or so for the larger ones)

Not sure what has changed, but I hope you can take another look at the extraction technique.

Also, multi threading would be dope :-)

Thanks in advance.

Dear,

I use the release "libesedb-experimental-20170121" and in particular, I'm trying to use the python library with Python 3.6.1 on Windows 10.

The python library seems empty (functions like file() are absent).

Test:

import sys
sys.path.append("C:\\Tools\\libesedb-20170121\\")
import pyesedb

dir(pyesedb)

Result: ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

Hey,

First off, great work in keeping this updated!

My issue is with the extraction speed of a 1.3 GB NTDS.dit file. I know you say this is not intended for large +1 GB databases, but the speed at which it extracts is DOG SLOW....it would takes weeks on end for this to extract the datatable.

When using the original 2012 alpha version , the extraction is SUPER FAST, however, it crashes half way through; consistently.

When trying to use the updated version (git), it doesn't crash but the extraction is so slow it prevents any type of conclusion.

Is there something I need to be doing differently to extract the tables faster?

I am on a CentOS 6 distro with 8 CPU and 16 GB RAM.

Thanks!

Hi
First I want to thanks for this great project
At this time i use this version : 20180716 and it is very slow and I think it tool more than a month to extract this huge database
I know impacket so please do not suggest this ! and also I read other issue but I did not get any useful response
So how should I speed up this extraction?

I can not export ntds.dit of windows server 2016

Unable to open: ../ntds.dit.
libesedb_catalog_definition_read: unsupported last fixed size data type: 12.
libesedb_catalog_read: unable to read catalog definition.
libesedb_file_open_read: unable to read catalog.
libesedb_file_open_file_io_handle: unable to read from file handle.
libesedb_file_open: unable to open file: ../ntds.dit.
info_handle_open: unable to open input file.

Hi, having an issue with your latest version - 20160128 from a gitpull.

When i use this command to extract:
esedbexport -l /tmp/esedbexport.log -t /tmp/ntds.dit.export <ntds.dit file>

After a while it gets killed on datatable 5 of 13.

I have discovered it is eating up all the memory and then it just dies.
I had 2GB in my VM Kali.
I then increased the VM to 12GB. and after a while it completed without errors. but i was monitoring the memory on the VM, it went to almost 100% just before a 140MB ntds.dit file finished. For larger files i guess you would need more memory to finish the job.

Is it correct in saying that i shouldve used this alpha version instead?
http://pkgs.fedoraproject.org/repo/pkgs/libesedb/libesedb-alpha-20120102.tar.gz/198a30c98ca1b3cb46d10a12bef8deaf/

Do you have any other latest alpha versions? because in your downloads section I only see 2 experimental ones.

Thank you for such amazing work btw.!

While reading this database, several records looks like corrupted.

Sample codes:

import pyesedb

eseFile = pyesedb.open("WebCacheV01.dat")
table0 = eseFile.get_table_by_name("Container_13")

for x in range(677, 702):
    record = table0.get_record(x);
    entryId = record.get_value_data_as_integer(0);
    url = record.get_value_data_as_string(17)
    print x,entryId,url

Sample File: https://drive.google.com/file/d/0ByRQ6t_DrBb6QkJfSVlRTnY1NUU/view?usp=sharing

Has anyone seen these errors? It only fails on the datatable - other tables will export successfully. I've also tried with the latest version and get the same errors. I'm not sure what's causing this and when ran in gdb there isn't a stacktrace. This is being ran on a CentOS.

root@myhost# esedbexport -T datatable ntds.dit
esedbexport 20120102

Opening file.
Exporting table 5 (datatable).
Unable to export file.
libesedb_page_tree_read_page: unsupported page flags: 0x9b42a802.
libesedb_page_tree_read_sub_nodes: unable to read page: 151006 at offset: 1237032960.
libfdata_tree_read_sub_nodes: unable to read sub nodes at offset: 1237032960.
libfdata_tree_node_read_leaf_node_values: unable to read sub nodes.
libfdata_tree_node_get_number_of_leaf_nodes: unable to retrieve node value.
libfdata_tree_node_read_leaf_node_values: unable to retrieve number of leaf nodes from sub node: 0.
libfdata_tree_node_get_number_of_leaf_nodes: unable to retrieve node value.
libfdata_tree_node_read_leaf_node_values: unable to retrieve number of leaf nodes from sub node: 0.
libfdata_tree_node_get_number_of_leaf_nodes: unable to retrieve node value.
libfdata_tree_get_number_of_leaf_nodes: unable to retrieve number of leaf nodes from root node.
libesedb_table_get_number_of_records: unable to retrieve number of leaf nodes from table values tree.
export_handle_export_table: unable to retrieve number of records.

On Debian's armel, armhf, i386, hppa, powerpc, x32 autobuilders, the tests fail with

esedb_test_data_segment.c:135 result (1) != -1
Unable to run test: libesedb_data_segment_initialize
Testing: data_segment  (FAIL)

Hello. I don't search esedbdumphash in last version libesedb. Old version can't dump big file 500Mb
error:
Opening file.
Dumping from table 4 (datatable).
Error
Unable to export file.
libfvalue_value_copy_to_32bit: entry data size out of bounds.
libesedb_record_get_value_32bit: unable to copy value to 32-bit value.
export_handle_export_record_is_needed_for_hash: unable to retrieve value.
export_handle_export_table: unable to export record.
export_handle_export_file: unable to export table: 3.

Hello there,
I've been experimenting with the lib with both 2014 and 2015 versions and different Exchange .edb files. It seems that about all exchange 2013 files give the same error on the same data type and more or less all on the same tables.

Here's a dump with HAVE_DEBUG_OUTPUT switched on only where the error is thrown:

esedbexport 20150522

Opening file.
Exporting table 29 (Folder_100).
libesedb_data_definition_read_record: tagged data type offset data size : 20
libesedb_data_definition_read_record: tagged data type offset data:
00000000: 01 01 30 80 04 01 31 80  05 01 32 80 15 01 40 80   ..0...1. ..2...@.
00000010: 18 01 79 80 01 16 83 81                            ..y.....

libesedb_data_definition_read_record: (256) tagged data type identifier : 256
libesedb_data_definition_read_record: (256) tagged data type offset             : 0x8018 (32792)
Unable to export file.
libesedb_data_definition_read_record: invalid tagged data type offset value out of bounds.
libesedb_record_initialize: unable to read data definition record.
libesedb_table_get_record: unable to create record.
export_handle_export_table: unable to retrieve record: 0.
export_handle_export_file: unable to export table: 28.

I can provide you test files if it may prove useful for debugging

At the moment there is no way to quickly scan all the records in a table. The following methods work very slow on large EDB files (10+gb):

libesedb_table_get_number_of_records
libesedb_table_get_record

It would be really great to have a function that would just return the next record without scanning the entire table tree, something like that:
libesedb_table_get_next_record
or via callback function
libesedb_table_iterate_records( int (*func) (libesedb_record_t *record )

Thanks!

Hi Joachim,

I was wondering if libesedb can handle/open databases marked as dirt and if not how to handle errors in Python? Is there a complete list of errors pyesedb.open() can raise?

Regards,
Jerome

When I execute a command: / root / Desktop / libesedb-20120102 / esedbtools / esedbexport /root/Desktop/ntds.dit

It returns the error message：
esedbexport 20120102

Opening file.
Exporting table 1 (MSysObjects) out of 11.
Exporting table 2 (MSysObjectsShadow) out of 11.
Exporting table 3 (MSysUnicodeFixupVer2) out of 11.
Exporting table 4 (datatable) out of 11.
Unable to export file.
libuna_unicode_character_copy_from_utf8: invalid 1st UTF-8 character byte: 0x80.
libuna_utf8_string_size_from_utf8_stream: unable to copy Unicode character from UTF-8 stream.
libfvalue_value_get_utf8_string_size: unable to determine UTF-8 string size of UTF-8 stream for codepage 1200.
libesedb_record_get_value_utf8_string_size: unable retrieve UTF-8 string size.
export_handle_export_record_value: unable to retrieve size of value string: 25 (271).
export_handle_export_record: unable to export record value: 25.
export_handle_export_table: unable to export record.
export_handle_export_file: unable to export table: 3.

version:20140803
run:esedbexport -t /root/Desktop/ntds.dit /root/Desktop/ntds.dit
return:
Opening file.
Exporting table 1 (MSysObjects) out of 11.
Exporting table 2 (MSysObjectsShadow) out of 11.
Exporting table 3 (MSysUnicodeFixupVer2) out of 11.
Exporting table 4 (datatable) out of 11.
kiiled

Binary columns can use compression.
Unfortunately, in Python bindings there is only a method to read the column data as raw (get_value_data), which does not decompress the data.
There are different methods to read different column types, but not as binary data with decompression.
For example, get_value_data_as_string reads the data, decompresses it if necessary, and converts to a string, but it cannot be used to read a binary column, even if we know that the content is a string.
I noticed that libesedb_record.c has the method libesedb_record_get_value_binary_data, which does exactly what I think it is missing in pyesedb_record.
I'm trying to read emails from an edb file from Exchange Server and I'm stuck with compressed binary contents that I am not able to decompress.

https://travis-ci.org/libyal/libesedb/jobs/274889738#L9532

line 33855: AX_LIBFMAPI_CHECK_LOCAL: command not found

hello does libesedb supported exchange 2013 file format

I have downloaded the zip file which contains all the source code. When I open the solution file in visual studio 2008, and start building the solution, I get lot of errors. Am I missing something ?

I'm not sure this is an issue and worth seeing)) Exporting table by name, that doesnt exists, produces successful message and no output (as expected :D)

How to reproduce:

esedbexport.exe -T lalalainvalidname WebCacheV01.dat

Actual behavior:
Output is:

esedbexport 20150409
Opening file.
Export completed.

Expected behavior:
Smth that tells that this table doesnt exists in defined database.

native body in every message table is a large binary data..any way i can get the actual body of the email from that.

Hi,

I didn't use the library but looked at the documentation and found a mistake.

At the database header at offset 104 it suppose to be the database ID (4 bytes) and then the log signature (28 bytes).

I'm not sure but I compared the header with the dump of eseutil.exe, and it makes sense.

https://github.com/libyal/libesedb/blob/master/libesedb/libesedb_record.c#L1638

1.the libesedb_page_read_values function in libesedb_page.c in libesedb allow remote attackers to cause a denial of service(invalid memory read and application crash) via a crafted esedb file.

esedbexport -m all libesedb_page_read_values
==9809==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d000008450 at pc 0x0000005fc2e3 bp 0x7ffcdcf21330 sp 0x7ffcdcf21328
READ of size 1 at 0x62d000008450 thread T0
    #0 0x5fc2e2 in libesedb_page_read_values /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:1248:29
    #1 0x5fa6b3 in libesedb_page_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:779:7
    #2 0x5ee496 in libesedb_io_handle_read_page /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_io_handle.c:264:6
    #3 0x650539 in libfdata_vector_get_element_value_by_index /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1613:7
    #4 0x650eaa in libfdata_vector_get_element_value_at_offset /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1749:6

0x62d000008450 is located 80 bytes to the right of 32768-byte region [0x62d000000400,0x62d000008400)
allocated by thread T0 here:
    #0 0x4f0a68 in malloc (/home/xxx/Desktop/afl-of-things/libesedb/esedbtools/esedbexport+0x4f0a68)
    #1 0x5f9730 in libesedb_page_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:373:27
    #2 0x5ee496 in libesedb_io_handle_read_page /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_io_handle.c:264:6
    #3 0x650539 in libfdata_vector_get_element_value_by_index /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1613:7
    #4 0x650eaa in libfdata_vector_get_element_value_at_offset /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1749:6

POC:libesedb_page_read_values
This vulnerability has been assigned as CVE-2018-15158.

2.the libesedb_page_read_tags function in libesedb_page.c in libesedb allow remote attackers to cause a denial of service(invalid memory read and application crash) via a crafted esedb file.

esedbexport -m all libesedb_page_read_tags
==9812==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d0000003ff at pc 0x0000005fb33c bp 0x7ffd89f8c8d0 sp 0x7ffd89f8c8c8
READ of size 1 at 0x62d0000003ff thread T0
    #0 0x5fb33b in libesedb_page_read_tags /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:948:3
    #1 0x5fa63d in libesedb_page_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:760:7
    #2 0x5ee496 in libesedb_io_handle_read_page /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_io_handle.c:264:6
    #3 0x650539 in libfdata_vector_get_element_value_by_index /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1613:7
    #4 0x650eaa in libfdata_vector_get_element_value_at_offset /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1749:6

0x62d0000003ff is located 1 bytes to the left of 32768-byte region [0x62d000000400,0x62d000008400)
allocated by thread T0 here:
    #0 0x4f0a68 in malloc (/home/xxx/Desktop/afl-of-things/libesedb/esedbtools/esedbexport+0x4f0a68)
    #1 0x5f9730 in libesedb_page_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:373:27
    #2 0x5ee496 in libesedb_io_handle_read_page /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_io_handle.c:264:6
    #3 0x650539 in libfdata_vector_get_element_value_by_index /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1613:7

POC:libesedb_page_read_tags
This vulnerability has been assigned as CVE-2018-15159.

3.the libesedb_catalog_definition_read function in libesedb_catalog_definition.c in libesedb allow remote attackers to cause a denial of service(invalid memory read and application crash) via a crafted esedb file.

esedbexport -m all libesedb_catalog_definition_read
==9815==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d000012400 at pc 0x0000004ef8ed bp 0x7ffc8d44a8c0 sp 0x7ffc8d44a070
READ of size 32512 at 0x62d000012400 thread T0
    #0 0x4ef8ec in __asan_memcpy (/home/xxx/Desktop/afl-of-things/libesedb/esedbtools/esedbexport+0x4ef8ec)
    #1 0x66b011 in libesedb_catalog_definition_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_catalog_definition.c
    #2 0x66900a in libesedb_catalog_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_catalog.c:1037:7
    #3 0x5e7ff8 in libesedb_file_open_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_file.c:1331:7
    #4 0x5e62ae in libesedb_file_open_file_io_handle /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_file.c:633:6

0x62d000012400 is located 0 bytes to the right of 32768-byte region [0x62d00000a400,0x62d000012400)
allocated by thread T0 here:
    #0 0x4f0a68 in malloc (/home/xxx/Desktop/afl-of-things/libesedb/esedbtools/esedbexport+0x4f0a68)
    #1 0x5f9730 in libesedb_page_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:373:27
    #2 0x5ee496 in libesedb_io_handle_read_page /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_io_handle.c:264:6
    #3 0x650539 in libfdata_vector_get_element_value_by_index /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1613:7
    #4 0x650eaa in libfdata_vector_get_element_value_at_offset /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1749:6

POC:libesedb_catalog_definition_read
This vulnerability has been assigned as CVE-2018-15160.

4.the libesedb_key_append_data function in libesedb_key.c in libesedb allow remote attackers to cause a denial of service(invalid memory read and application crash) via a crafted esedb file.

esedbexport -m all libesedb_key_append_data
==9818==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62d000008400 at pc 0x0000004ef8ed bp 0x7ffcdb636260 sp 0x7ffcdb635a10
READ of size 8191 at 0x62d000008400 thread T0
    #0 0x4ef8ec in __asan_memcpy (/home/xxx/Desktop/afl-of-things/libesedb/esedbtools/esedbexport+0x4ef8ec)
    #1 0x5ef039 in libesedb_key_append_data /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_key.c:307:7
    #2 0x5ff086 in libesedb_page_tree_read_page /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page_tree.c:1439:7
    #3 0x5ffe4d in libesedb_page_tree_read_node /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page_tree.c:1722:6
    #4 0x62c855 in libfdata_btree_read_node /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_btree.c:1287:7

0x62d000008400 is located 0 bytes to the right of 32768-byte region [0x62d000000400,0x62d000008400)
allocated by thread T0 here:
    #0 0x4f0a68 in malloc (/home/xxx/Desktop/afl-of-things/libesedb/esedbtools/esedbexport+0x4f0a68)
    #1 0x5f9730 in libesedb_page_read /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_page.c:373:27
    #2 0x5ee496 in libesedb_io_handle_read_page /home/xxx/Desktop/afl-of-things/libesedb/libesedb/libesedb_io_handle.c:264:6
    #3 0x650539 in libfdata_vector_get_element_value_by_index /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1613:7
    #4 0x650eaa in libfdata_vector_get_element_value_at_offset /home/xxx/Desktop/afl-of-things/libesedb/libfdata/libfdata_vector.c:1749:6

POC:libesedb_key_append_data
This vulnerability has been assigned as CVE-2018-15161.

pocs.zip

Error parsing Container_16 table. Attached python sample of error case:

lib returns 0 as a count of records for definite table.
ESEDatabaseView tool parses this table fine, displaying 357 records inside.

Sample database: https://onedrive.live.com/?cid=83cbbef6b31667e8&id=83cbbef6b31667e8%218243&ithint=file,dat&authkey=!AA3esiSKO8xXMK8

Exporting table 1 (MSysObjects) out of 11.
Exporting table 2 (MSysObjectsShadow) out of 11.
Exporting table 3 (MSysUnicodeFixupVer2) out of 11.
Exporting table 4 (datatable) out of 11.
Killed.

My ntds.dit file is approximately 443mb , is there a reason why it is getting killed. My Kali linux was using 3gm of ram and an i7 3610qm

How can i get long string value?

by using esedbexpert, it can not get very long string vlaue.

it just show me ' ' or 'L' .

this should show me value like under.

Offset 0 1 2 3 4 5 6 7 8 9 A B C D E F

00DC8EF0 00 00 08 20 00 00 00 04 00 00 00 00 56 00 69 00 V i
00DC8F00 73 00 69 00 74 00 65 00 64 00 3A 00 20 00 74 00 s i t e d : t
00DC8F10 69 00 6D 00 6A 00 6A 00 68 00 40 00 68 00 74 00 i m j j h @ h t
00DC8F20 74 00 70 00 3A 00 2F 00 2F 00 67 00 6F 00 6F 00 t p : / / g o o
00DC8F30 67 00 6C 00 65 00 61 00 64 00 73 00 2E 00 67 00 g l e a d s . g
00DC8F40 2E 00 64 00 6F 00 75 00 62 00 6C 00 65 00 63 00 . d o u b l e c
00DC8F50 6C 00 69 00 63 00 6B 00 2E 00 6E 00 65 00 74 00 l i c k . n e t
00DC8F60 2F 00 70 00 61 00 67 00 65 00 61 00 64 00 2F 00 / p a g e a d /
00DC8F70 61 00 64 00 73 00 3F 00 61 00 62 00 6C 00 3D 00 a d s ? a b l =
00DC8F80 43 00 53 00 26 00 63 00 6C 00 69 00 65 00 6E 00 C S & c l i e n
00DC8F90 74 00 3D 00 63 00 61 00 2D 00 70 00 75 00 62 00 t = c a - p u b
00DC8FA0 2D 00 38 00 36 00 31 00 33 00 36 00 31 00 36 00 - 8 6 1 3 6 1 6
00DC8FB0 38 00 37 00 32 00 37 00 34 00 30 00 35 00 34 00 8 7 2 7 4 0 5 4
00DC8FC0 30 00 26 00 6F 00 75 00 74 00 70 00 75 00 74 00 0 & o u t p u t
00DC8FD0 3D 00 68 00 74 00 6D 00 6C 00 26 00 68 00 3D 00 = h t m l & h =
00DC8FE0 35 00 30 00 26 00 73 00 6C 00 6F 00 74 00 6E 00 5 0 & s l o t n
00DC8FF0 61 00 6D 00 65 00 3D 00 38 00 32 00 34 00 37 00 a m e = 8 2 4 7
00DC9000 34 00 34 00 34 00 37 00 31 00 34 00 26 00 61 00 4 4 4 7 1 4 & a
00DC9010 64 00 6B 00 3D 00 32 00 35 00 37 00 36 00 38 00 d k = 2 5 7 6 8
00DC9020 30 00 35 00 32 00 31 00 36 00 26 00 77 00 3D 00 0 5 2 1 6 & w =
00DC9030 33 00 32 00 30 00 26 00 6C 00 6D 00 74 00 3D 00 3 2 0 & l m t =
00DC9040 31 00 34 00 30 00 33 00 34 00 38 00 31 00 38 00 1 4 0 3 4 8 1 8
00DC9050 37 00 39 00 26 00 65 00 61 00 3D 00 30 00 26 00 7 9 & e a = 0 &
00DC9060 66 00 6C 00 61 00 73 00 68 00 3D 00 31 00 33 00 f l a s h = 1 3
00DC9070 2E 00 30 00 2E 00 30 00 2E 00 32 00 31 00 34 00 . 0 . 0 . 2 1 4
00DC9080 26 00 75 00 72 00 6C 00 3D 00 68 00 74 00 74 00 & u r l = h t t
00DC9090 70 00 25 00 33 00 41 00 25 00 32 00 46 00 25 00 p % 3 A % 2 F %
00DC90A0 32 00 46 00 77 00 77 00 77 00 2E 00 62 00 61 00 2 F w w w . b a
00DC90B0 6E 00 64 00 69 00 73 00 6F 00 66 00 74 00 2E 00 n d i s o f t .
00DC90C0 63 00 6F 00 2E 00 6B 00 72 00 25 00 32 00 46 00 c o . k r % 2 F
00DC90D0 62 00 61 00 6E 00 64 00 69 00 7A 00 69 00 70 00 b a n d i z i p
00DC90E0 25 00 32 00 46 00 75 00 70 00 64 00 61 00 74 00 % 2 F u p d a t
00DC90F0 65 00 69 00 6E 00 66 00 6F 00 25 00 32 00 46 00 e i n f o % 2 F
00DC9100 69 00 6E 00 64 00 65 00 78 00 2E 00 70 00 68 00 i n d e x . p h
00DC9110 70 00 26 00 64 00 74 00 3D 00 31 00 34 00 30 00 p & d t = 1 4 0
00DC9120 33 00 34 00 38 00 31 00 38 00 37 00 37 00 35 00 3 4 8 1 8 7 7 5
00DC9130 38 00 37 00 26 00 73 00 68 00 76 00 3D 00 72 00 8 7 & s h v = r
00DC9140 32 00 30 00 31 00 34 00 30 00 36 00 31 00 37 00 2 0 1 4 0 6 1 7
00DC9150 26 00 63 00 62 00 76 00 3D 00 72 00 32 00 30 00 & c b v = r 2 0
00DC9160 31 00 34 00 30 00 34 00 31 00 37 00 26 00 73 00 1 4 0 4 1 7 & s
00DC9170 61 00 6C 00 64 00 72 00 3D 00 73 00 62 00 26 00 a l d r = s b &
00DC9180 63 00 6F 00 72 00 72 00 65 00 6C 00 61 00 74 00 c o r r e l a t
00DC9190 6F 00 72 00 3D 00 31 00 34 00 30 00 33 00 34 00 o r = 1 4 0 3 4
00DC91A0 38 00 31 00 38 00 37 00 38 00 34 00 33 00 37 00 8 1 8 7 8 4 3 7
00DC91B0 26 00 66 00 72 00 6D 00 3D 00 32 00 30 00 26 00 & f r m = 2 0 &
00DC91C0 67 00 61 00 5F 00 76 00 69 00 64 00 3D 00 33 00 g a _ v i d = 3
00DC91D0 32 00 31 00 36 00 32 00 39 00 35 00 38 00 37 00 2 1 6 2 9 5 8 7
00DC91E0 2E 00 31 00 34 00 30 00 33 00 34 00 38 00 31 00 . 1 4 0 3 4 8 1
00DC91F0 38 00 38 00 30 00 26 00 67 00 61 00 5F 00 73 00 8 8 0 & g a _ s
00DC9200 69 00 64 00 3D 00 31 00 34 00 30 00 33 00 34 00 i d = 1 4 0 3 4
00DC9210 38 00 31 00 38 00 38 00 30 00 26 00 67 00 61 00 8 1 8 8 0 & g a
00DC9220 5F 00 68 00 69 00 64 00 3D 00 32 00 30 00 38 00 _ h i d = 2 0 8
00DC9230 37 00 34 00 36 00 33 00 34 00 31 00 38 00 26 00 7 4 6 3 4 1 8 &
00DC9240 67 00 61 00 5F 00 66 00 63 00 3D 00 30 00 26 00 g a _ f c = 0 &
00DC9250 75 00 5F 00 74 00 7A 00 3D 00 35 00 34 00 30 00 u _ t z = 5 4 0
00DC9260 26 00 75 00 5F 00 68 00 69 00 73 00 3D 00 30 00 & u _ h i s = 0
00DC9270 26 00 75 00 5F 00 6A 00 61 00 76 00 61 00 3D 00 & u _ j a v a =
00DC9280 31 00 26 00 75 00 5F 00 68 00 3D 00 31 00 30 00 1 & u _ h = 1 0
00DC9290 38 00 30 00 26 00 75 00 5F 00 77 00 3D 00 31 00 8 0 & u _ w = 1
00DC92A0 39 00 32 00 30 00 26 00 75 00 5F 00 61 00 68 00 9 2 0 & u _ a h
00DC92B0 3D 00 31 00 30 00 34 00 30 00 26 00 75 00 5F 00 = 1 0 4 0 & u _
00DC92C0 61 00 77 00 3D 00 31 00 39 00 32 00 30 00 26 00 a w = 1 9 2 0 &
00DC92D0 75 00 5F 00 63 00 64 00 3D 00 33 00 32 00 26 00 u _ c d = 3 2 &
00DC92E0 75 00 5F 00 6E 00 70 00 6C 00 75 00 67 00 3D 00 u _ n p l u g =
00DC92F0 30 00 26 00 75 00 5F 00 6E 00 6D 00 69 00 6D 00 0 & u _ n m i m
00DC9300 65 00 3D 00 30 00 26 00 64 00 66 00 66 00 3D 00 e = 0 & d f f =
00DC9310 25 00 45 00 42 00 25 00 41 00 37 00 25 00 39 00 % E B % A 7 % 9
00DC9320 31 00 25 00 45 00 43 00 25 00 39 00 44 00 25 00 1 % E C % 9 D %
00DC9330 38 00 30 00 25 00 32 00 30 00 25 00 45 00 41 00 8 0 % 2 0 % E A
00DC9340 25 00 42 00 33 00 25 00 41 00 30 00 25 00 45 00 % B 3 % A 0 % E
00DC9350 42 00 25 00 39 00 34 00 25 00 39 00 35 00 26 00 B % 9 4 % 9 5 &
00DC9360 64 00 66 00 73 00 3D 00 31 00 33 00 26 00 61 00 d f s = 1 3 & a
00DC9370 64 00 78 00 3D 00 31 00 30 00 26 00 61 00 64 00 d x = 1 0 & a d
00DC9380 79 00 3D 00 35 00 30 00 26 00 62 00 69 00 77 00 y = 5 0 & b i w
00DC9390 3D 00 35 00 30 00 31 00 26 00 62 00 69 00 68 00 = 5 0 1 & b i h
00DC93A0 3D 00 33 00 31 00 38 00 26 00 65 00 69 00 64 00 = 3 1 8 & e i d
00DC93B0 3D 00 33 00 31 00 37 00 31 00 35 00 30 00 33 00 = 3 1 7 1 5 0 3
00DC93C0 30 00 34 00 26 00 6F 00 69 00 64 00 3D 00 33 00 0 4 & o i d = 3
00DC93D0 26 00 72 00 78 00 3D 00 30 00 26 00 65 00 61 00 & r x = 0 & e a
00DC93E0 65 00 3D 00 34 00 26 00 64 00 6F 00 63 00 6D 00 e = 4 & d o c m
00DC93F0 3D 00 37 00 26 00 76 00 69 00 73 00 3D 00 30 00 = 7 & v i s = 0
00DC9400 26 00 70 00 70 00 6A 00 6C 00 3D 00 75 00 26 00 & p p j l = u &
00DC9410 66 00 75 00 3D 00 30 00 26 00 69 00 66 00 69 00 f u = 0 & i f i
00DC9420 3D 00 31 00 26 00 64 00 74 00 64 00 3D 00 33 00 = 1 & d t d = 3
00DC9430 37 00 38 00 31 00 00 00 04 20 00 00 00 04 01 00 7 8 1

but, that value only i can see like this. Watch Name
Value => #4#0
also it sat to me string size is only 2.
and column type is LIBESEDB_COLUMN_TYPE_LARGE_TEXT.
how can i get this kind of value by unsing libesedb ??

While trying to build libesedb, the Debian/s390x autobuilder failed in the libesedb_checksum_calculate_little_endian_xor32 test:

esedb_test_checksum.c:1237 checksum_value (1617174027) != 669295665
Unable to run test: libesedb_checksum_calculate_little_endian_xor32
Testing: checksum  (FAIL)

I have found some spots in libesedb_checksum_calculate_little_endian_xor32 that look like they could use some le32toh or be32toh, but I haven't figured out what exactly is going wrong.

Can this lib be used to parse MS Exchange database? if yes, what's the supported version? Exchange 2000, 2003, 2007, 2013?

Sample db file

How to reproduce:

import pyesedb

db = pyesedb.open(...)
table0 = db.get_table_by_name("Container_21")

num = table0.get_number_of_records()
print table0.get_name(), " records: ", num

The result ( on table "Container_21") is:

libesedb_table_get_number_of_records: unable to retrieve number of leaf values from table values tree.values: unsupported page tags value size value out of bounds.

esedbinfo output - dont see any troubles with Container_21:

Table: 34 Container_21 (58)
Number of columns: 25
Column Identifier Name Type
1 1 EntryId Integer 64-bit signed
2 2 ContainerId Integer 64-bit signed
3 3 CacheId Integer 64-bit signed
4 4 UrlHash Integer 64-bit signed
5 5 SecureDirectory Integer 32-bit unsigned
6 6 FileSize Integer 64-bit signed
7 7 Type Integer 32-bit unsigned
8 8 Flags Integer 32-bit unsigned
9 9 AccessCount Integer 32-bit unsigned
10 10 SyncTime Integer 64-bit signed
11 11 CreationTime Integer 64-bit signed
12 12 ExpiryTime Integer 64-bit signed
13 13 ModifiedTime Integer 64-bit signed
14 14 AccessedTime Integer 64-bit signed
15 15 PostCheckTime Integer 64-bit signed
16 16 SyncCount Integer 32-bit unsigned
17 17 ExemptionDelta Integer 32-bit unsigned
18 256 Url Large text
19 257 Filename Large text
20 258 FileExtension Large text
21 259 RequestHeaders Large binary data
22 260 ResponseHeaders Large binary data
23 261 RedirectUrl Large text
24 262 Group Large binary data
25 263 ExtraData Large binary data

Number of indexes: 1
Index: 1 HashEntryIdIndex (58)

Index: 1 HashEntryIdIndex (58)

I've cloned latest but getting the following.

$ ./autogen.sh
libtoolize: putting auxiliary files in .'. libtoolize: copying file ./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, m4'. libtoolize: copying file m4/libtool.m4'
libtoolize: copying file m4/ltoptions.m4' libtoolize: copying file m4/ltsugar.m4'
libtoolize: copying file m4/ltversion.m4' libtoolize: copying file m4/lt~obsolete.m4'
configure.ac:35: warning: The 'AM_PROG_MKDIR_P' macro is deprecated, and its use is discouraged.
configure.ac:35: You should use the Autoconf-provided 'AC_PROG_MKDIR_P' macro instead,
configure.ac:35: and use '$(MKDIR_P)' instead of '$(mkdir_p)'in your Makefile.am files.
configure.ac:205: error: required file 'libcerror/Makefile.in' not found
configure.ac:206: error: required file 'libcthreads/Makefile.in' not found
configure.ac:207: error: required file 'libcdata/Makefile.in' not found
configure.ac:208: error: required file 'libclocale/Makefile.in' not found
configure.ac:209: error: required file 'libcnotify/Makefile.in' not found
configure.ac:210: error: required file 'libcsplit/Makefile.in' not found
configure.ac:211: error: required file 'libuna/Makefile.in' not found
configure.ac:212: error: required file 'libcfile/Makefile.in' not found
configure.ac:213: error: required file 'libcpath/Makefile.in' not found
configure.ac:214: error: required file 'libbfio/Makefile.in' not found
configure.ac:215: error: required file 'libfcache/Makefile.in' not found
configure.ac:216: error: required file 'libfdata/Makefile.in' not found
configure.ac:217: error: required file 'libfdatetime/Makefile.in' not found
configure.ac:218: error: required file 'libfguid/Makefile.in' not found
configure.ac:219: error: required file 'libfvalue/Makefile.in' not found
configure.ac:220: error: required file 'libfwnt/Makefile.in' not found
configure.ac:225: error: required file 'libfmapi/Makefile.in' not found
configure.ac:226: error: required file 'libmapidb/Makefile.in' not found
Makefile.am:3: error: required directory ./libcerror does not exist
Makefile.am:3: error: required directory ./libcthreads does not exist
Makefile.am:3: error: required directory ./libcdata does not exist
Makefile.am:3: error: required directory ./libclocale does not exist
Makefile.am:3: error: required directory ./libcnotify does not exist
Makefile.am:3: error: required directory ./libcsplit does not exist
Makefile.am:3: error: required directory ./libuna does not exist
Makefile.am:3: error: required directory ./libcfile does not exist
Makefile.am:3: error: required directory ./libcpath does not exist
Makefile.am:3: error: required directory ./libbfio does not exist
Makefile.am:3: error: required directory ./libfcache does not exist
Makefile.am:3: error: required directory ./libfdata does not exist
Makefile.am:3: error: required directory ./libfdatetime does not exist
Makefile.am:3: error: required directory ./libfguid does not exist
Makefile.am:3: error: required directory ./libfvalue does not exist
Makefile.am:3: error: required directory ./libfwnt does not exist
Makefile.am:3: error: required directory ./libfmapi does not exist
Makefile.am:3: error: required directory ./libmapidb does not exist
autoreconf: automake failed with exit status: 1

$ automake --version
automake (GNU automake) 1.14.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl-2.0.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Tom Tromey [email protected]
and Alexandre Duret-Lutz [email protected].

$ autoconf --version
autoconf (GNU Autoconf) 2.69
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+/Autoconf: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html, http://gnu.org/licenses/exceptions.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by David J. MacKenzie and Akim Demaille.

I installed the libesedb 20170121 following the install document in the archive. After the install, when I am trying to run the 'esedbexport -m tables /ntds.dit' I am receiving the below error. (I can't get a screenshot since this is running on a standalone system that I cannot connect to my production network) Was this an incorrect install or am I likely missing something or not seeing an error during the install?

esedbexport: error while loading shared libraries: libesedb.so.1: cannot open shared object file: No such file or directory

When trying to parse the response header of a download record (Container 29, column 21), it appears that the returned value is incorrect (b'\x01\x00\x00\x00' instead of a large blob that can be seen using NirSoft ESEDatabaseview - https://www.nirsoft.net/utils/ese_database_view.html).

e.g.

tbl = db.get_table_by_name('Container_29')
lst = [raw_history_record.get_value_data(21) for raw_history_record in tbl.records]
lst
[b'\x01\x00\x00\x00']

WebCacheV01.zip

Trying to parse a Windows.edb file from Windows 10 1803 file results in the following errors:

$ ./esedbexport -t EXPORTS Windows.edb
esedbexport 20200102

Opening file.
Database type: Windows Search.
Exporting table 1 (MSysObjects) out of 29.
Exporting table 2 (MSysObjectsShadow) out of 29.
Exporting table 3 (MSysObjids) out of 29.
Exporting table 4 (MSysLocales) out of 29.
Exporting table 5 (CatalogManager_Properties) out of 29.
Exporting table 6 (CatalogStorageManager) out of 29.
Exporting table 7 (SystemIndex_Gthr) out of 29.
Exporting table 8 (SystemIndex_GthrPth) out of 29.
Exporting table 9 (SystemIndex_GthrAppOwner) out of 29.
Exporting table 10 (SystemIndex_1_Properties) out of 29.
Exporting table 11 (SystemIndex_1) out of 29.
Exporting table 12 (SystemIndex_PropertyStore) out of 29.
Unable to export file.
export_handle_export_basic_record_value: missing value string.
export_handle_export_record_value: unable to export basic record value: 305.
export_handle_export_record: unable to export record value: 305.
export_handle_export_table: unable to export record.
export_handle_export_file: unable to export table: 11.

The errors are the same when running the last two experimental releases (libesedb-20181229 and libesedb-20191220).

I've recompiled 20200102 with --enable-verbose-output --enable-debug-output and re-ran just exporting the problematic table (SystemIndex_PropertyStore) and used the -v switch for debug output.

./esedbexport -T SystemIndex_PropertyStore -t EXPORTS -v Windows.edb > SystemIndex_PropertyStore.debug 2>&1

It's been running 20+ minutes and the file where I'm capturing the debug output is approaching 1GB which doesn't seem right, but maybe it is? The Windows.edb file is 224MB.

The tools are being run on macOS 10.14.6.

Is there something else I should be trying? Or something I'm missing?

Thanks!

This with the default NTDS.dit after elevating the Windows Server 2016 server to AD controller.

I'm using vssadmin to create a shadow copy then verifying with DIT snapshot viewer. I had to run Esentutl on the file before it would load in the DIT snapshot viewer.

$ /usr/local/bin/esedbexport NTDS.dit
esedbexport 20171022

Opening file.
Unable to open: NTDS.dit.
libesedb_catalog_definition_read: unsupported last fixed size data type: 12.
libesedb_catalog_read: unable to read catalog definition.
libesedb_file_open_read: unable to read catalog.
libesedb_file_open_file_io_handle: unable to read from file handle.
libesedb_file_open: unable to open file: NTDS.dit.
export_handle_open: unable to open input file.

I am unable to build using Yosemite, I have followed the build instructions but when I come to make I get the following error:

libcfile/libcfile_support.c:742:2: error: Missing file remove function
# error Missing file remove function

 ^
1 error generated.

Also fix Python bindings

Hi,

I'm using the python module in order to browse a NTDS file. I noticed that the multi_value class is included in the module but I'm struggling to get an object of that type from a record. Is there a way that I can :

• Get the multi_value object from a record (when is_multi_value() is true )
  OR
• interpret something form get_value_data in order to recover an array of my data and their type
  OR
• something else that I missed ..

Thank you in advance for the reply. Anyway I have to thank you too for the good work 👍

Hi, I apologize if this is a dumb question. Trying to compile today on Kali Linux 2017.3 64 bit, at the end of the make process I am getting the following error:
"
In file included from libesedb_data_definition.c:39:0:
libesedb_libfvalue.h:36:10: Fatal Error: libfvalue_split_string.h: No such file or directory
#include <libfvalue_split_string.h>
compilation terminated

Any ideas where I can get this file? Thanks very much.

Hello! How can I get large record values? If I call get-record_value..() for the long binary (>1 Mb), it will return me only a little part.

Thanks for the effort in developing the library.

This issue is a duplicate of issue #2 and #40 except that we are offering a possible solution to improve the performance.

Problem:
Performance was truly abysmal on large files. Performance falls off a cliff once the file size reaches a certain point. We didn't measure the exact location of the 'cliff' but a 3GB file was acceptable, while a 6GB (400K records) file wasn't. Time to dump one table from the 6GB file was 3+ hours on high end hardware.

Solution:
Increased LIBESEDB_MAXIMUM_CACHE_ENTRIES_TABLE_VALUES from 32k to 128k.
The cache is implemented as a hash table, but when the hash table is starting to get full entries are ejected from the cache only to be re-read multiple times later on. Making the hash table larger fixes this by avoiding hash table collisions.

In cases where a table has a large number of columns, like in the Windows.edb file with 600 columns, performance was further improved by replacing column catalog linked list with array. e.g.

Replace,
libcdata_list_initialize(&( ( *table_definition )->column_catalog_definition_list )
with
libcdata_array_initialize(&( ( *table_definition )->column_catalog_definition_list )

These changes gave around a 40x speed improvement on large files. Scan time went from 3h to 3min. Of course the trade off is slightly increased RAM usage.