libyal / libevtx Goto Github PK

Library and tools to access the Windows XML Event Log (EVTX) format

License: GNU Lesser General Public License v3.0

PowerShell 1.59% Shell 4.13% C 74.86% Makefile 1.33% C++ 0.27% Python 1.18% M4 15.82% Roff 0.82%

libevtx's Introduction

The name libyal was initially a pun on the naming theme of the various library 
projects. Now it serves the purpose of providing an overview of the available 
projects in a single location and as a home for scripts to help maintain the 
projects.

For more information see:

* Project documentation: https://github.com/libyal/libyal/wiki/Home
* Overiew of available projects: https://github.com/libyal/libyal/wiki/Overview

libevtx's People

Contributors

Stargazers

Watchers

libevtx's Issues

pyevtx: unsupported security identifier contains more than 10 sub authoritites

I am getting the following error within strings inside the record class (record.strings).
object[1] failed - <type 'exceptions.IOError'>: pyevtx_record_get_string_by_index: unable to retrieve string: 1 size. libfwnt_security_identifier_copy_from_byte_stream: unsupported security identifier contains more than 10 sub authoritites.
This prevents record.xml_string from being generated it would appear.

Here is an example file, script, and output from evtxexport.exe:
EVTX file: https://www.dropbox.com/s/1j5e6qnrs45di1u/Archive-Security-2013-10-01-20-02-28-916.evtx?dl=0

Here is example script:

import sys
import pyevtx

#Filename
filename = sys.argv[1]

#Record Index
index = int(sys.argv[2])

#Open pyevtx file
evtxfile = pyevtx.file()
evtxfile.open(filename)

#Get record by index
record = evtxfile.get_record(index)

#print record id#
print 'EventRecordID: {}'.format(record.identifier)

#print xml string#
print 'XML String: {}'.format(record.xml_string)

#print strings#
i = 0
for rstring in record.strings:
    print 'string[{}]: {}'.format(i,rstring)
    i += 1

When passing these params to test script:
Archive-Security-2013-10-01-20-02-28-916.evtx 10613

I get the following Error:
EventRecordID: 1397257 Traceback (most recent call last): File "debug_evtx_record.py", l ine 21, in <module> print 'XML String: {}'.format(record.xml_string) IOError: pyevtx_record_get_xml_string: unable to retrieve XML string size.

If I look at this in a debugger I see that one of the record.strings has the error:
object[1] failed - <type 'exceptions.IOError'>: pyevtx_record_get_string_by_index: unable to retrieve string: 1 size. libfwnt_security_identifier_copy_from_byte_stream: unsupported security identifier contains more than 10 sub authoritites.

Here is debug vars:

However, when I run evtxexport.exe, I get all the xml, and strings. Here is the example of the same record:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="54849625-5478-4994-A5BA-3E3B0328C30D}"/>
    <EventID>4732</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>13826</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8020000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-01T18:51:20.218822900Z"/>
    <EventRecordID>1397257</EventRecordID>
    <Correlation/>
    <Execution ProcessID="728" ThreadID="12284"/>
    <Channel>Security</Channel>
    <Computer>Bifrost</Computer>
    <Security/>
  </System>
  <EventData>
    <Data Name="MemberName">-</Data>
    <Data Name="MemberSid">S-1-11-96-3623454863-58364-18864-2661722203-1597581903-3241140313-1528907555-2380831335-2281093177-363464117</Data>
    <Data Name="TargetUserName">HomeUsers</Data>
    <Data Name="TargetDomainName">Bifrost</Data>
    <Data Name="TargetSid">S-1-5-21-718126207-1171771683-1750804747-1002</Data>
    <Data Name="SubjectUserSid">S-1-5-18</Data>
    <Data Name="SubjectUserName">BIFROST$</Data>
    <Data Name="SubjectDomainName">ASGARD</Data>
    <Data Name="SubjectLogonId">0x00000000000003e7</Data>
    <Data Name="PrivilegeList">-</Data>
  </EventData>
</Event>

evtxexport: show both the composite and non-composite identifier

Please provide API functions to retrieve "string_identifiers_array" values, too

Currently the only way to get the string_identifiers_array data seems to be using the XML functions, and to parse the string IDs from the XML blob. That's not a good idea.

Please provide functions similar to libevtx_record_get_utf8_string.

Thank you!

libfwevt_xml_document_substitute_template_value: unable to retrieve template value: 4 from array.

Hello Joachim!

I get the following fatal error when exporting logs using evtxexport (b524d6b):

Unable to export file.
libcdata_array_get_entry_by_index: invalid entry index value out of bounds.
libfwevt_xml_document_substitute_template_value: unable to retrieve template value: 4 from array.
libfwevt_xml_document_read_optional_substitution: unable to substitute template value.
libfwevt_xml_document_read_attribute: unable to read optional substitution.
libfwevt_xml_document_read_element: unable to read attribute.
libfwevt_xml_document_read_element: unable to read element.
libfwevt_xml_document_read_element: unable to read element.
libfwevt_xml_document_read_template_instance: unable to read element.
libfwevt_xml_document_read_fragment: unable to read document template instance.
libfwevt_xml_document_substitute_template_value: unable to read fragment header.
libfwevt_xml_document_read_optional_substitution: unable to substitute template value.
libfwevt_xml_document_read_element: unable to read optional substitution.
libfwevt_xml_document_read_template_instance: unable to read element.
libfwevt_xml_document_read_fragment: unable to read document template instance.
libfwevt_xml_document_read_with_template_values: unable to read fragment header.
libfwevt_xml_document_read: unable to read XML document.
libevtx_record_values_read_xml_document: unable to read binary XML document.
libevtx_io_handle_read_chunk: unable to read record values XML document.
libfdata_list_get_element_value: unable to read element data at offset: 0x00004b30.
libfdata_list_get_element_value_by_index: unable to retrieve element value.
libevtx_file_get_record_by_index: unable to retrieve record values: 20.
export_handle_export_records: unable to retrieve record: 20.
export_handle_export_file: unable to export records.

I isolated the broken record in the attached broken.evtx.gz file. This file can be opened in Windows Event Viewer, it corresponds to "The VSS service is shutting down due to shutdown event from the Service Control Manager. %1". Yet, the record is 68KB ?!?

$ evtxinfo broken.evtx
evtxinfo 20190904

Windows Event Viewer Log (EVTX) information:
        Version                         : 3.1
        Number of records               : 1
        Number of recovered records     : 111

$ evtxexport broken.evtx
evtxexport 20190904

Unable to export file.
libcdata_array_get_entry_by_index: invalid entry index value out of bounds.
libfwevt_xml_document_substitute_template_value: unable to retrieve template value: 4 from array.
libfwevt_xml_document_read_optional_substitution: unable to substitute template value.
libfwevt_xml_document_read_attribute: unable to read optional substitution.
libfwevt_xml_document_read_element: unable to read attribute.
libfwevt_xml_document_read_element: unable to read element.
libfwevt_xml_document_read_element: unable to read element.
libfwevt_xml_document_read_template_instance: unable to read element.
libfwevt_xml_document_read_fragment: unable to read document template instance.
libfwevt_xml_document_substitute_template_value: unable to read fragment header.
libfwevt_xml_document_read_optional_substitution: unable to substitute template value.
libfwevt_xml_document_read_element: unable to read optional substitution.
libfwevt_xml_document_read_template_instance: unable to read element.
libfwevt_xml_document_read_fragment: unable to read document template instance.
libfwevt_xml_document_read_with_template_values: unable to read fragment header.
libfwevt_xml_document_read: unable to read XML document.
libevtx_record_values_read_xml_document: unable to read binary XML document.
libevtx_io_handle_read_chunk: unable to read record values XML document.
libfdata_list_get_element_value: unable to read element data at offset: 0x00001200.
libfdata_list_get_element_value_by_index: unable to retrieve element value.
libevtx_file_get_record_by_index: unable to retrieve record values: 0.
export_handle_export_records: unable to retrieve record: 0.
export_handle_export_file: unable to export records.

If I use evtx_structure.py, I get the following:

$ evtx_structure.py broken.evtx
File header
  magic: ElfFile�
  oldest_chunk: 0x0
  current_chunk_number: 0x0
  next_record_number: 0x2
  header_size: 0x80
  minor_version: 0x1
  major_version: 0x3
  header_chunk_size: 0x1000
  chunk_count: 0x1
  flags: 0x0
  checksum: 0xd0ff1810
  verify: True
  dirty: False
  full: False
  Chunk
    offset: 0x1000
    magic: ElfChnk�
    file_first_record_number: 0x1
    file_last_record_number: 0x1
    log_first_record_number: 0x1
    log_last_record_number: 0x1
    header_size: 0x80
    last_record_offset: 0x200
    next_record_offset: 0x6d8
    data_checksum: 0x779c967b
    header_checksum: 0x1b3405e2
    verify: True
    templates: 1
    Record
      offset: 0x1200
      magic: 0x2a2a
      size: 0x4d8
      number: 0x1
      timestamp: 2018-07-23 09:26:38.304127
      verify: True
      RootNode(offset=0x18)
        StreamStartNode(offset=0x18)
        TemplateInstanceNode(offset=0x1c, resident=True, length=0x345)
          TemplateNode(offset=0x26)
            StreamStartNode(offset=0x3e)
            OpenStartElementNode(offset=0x42) --> Event
              AttributeNode(offset=0x65) --> xmlns
                ValueNode(offset=0x7e)
                  WstringTypeNode(offset=0x80) --> http://schemas.microsoft.com/win/2004/08/events/event
              CloseStartElementNode(offset=0xec)
              OpenStartElementNode(offset=0xed) --> System
                CloseStartElementNode(offset=0x10e)
                OpenStartElementNode(offset=0x10f) --> Provider
                  AttributeNode(offset=0x138) --> Name
                    ValueNode(offset=0x14f)
                      WstringTypeNode(offset=0x151) --> VSS
                  CloseEmptyElementNode(offset=0x159)
                OpenStartElementNode(offset=0x15a) --> EventID
                  AttributeNode(offset=0x181) --> Qualifiers
                    ConditionalSubstitutionNode(offset=0x1a4)
                  CloseStartElementNode(offset=0x1a8)
                  ConditionalSubstitutionNode(offset=0x1a9)
                  CloseElementNode(offset=0x1ad)
                OpenStartElementNode(offset=0x1ae) --> Level
                  CloseStartElementNode(offset=0x1cd)
                  ConditionalSubstitutionNode(offset=0x1ce)
                  CloseElementNode(offset=0x1d2)
                OpenStartElementNode(offset=0x1d3) --> Task
                  CloseStartElementNode(offset=0x1f0)
                  ConditionalSubstitutionNode(offset=0x1f1)
                  CloseElementNode(offset=0x1f5)
                OpenStartElementNode(offset=0x1f6) --> Keywords
                  CloseStartElementNode(offset=0x21b)
                  ConditionalSubstitutionNode(offset=0x21c)
                  CloseElementNode(offset=0x220)
                OpenStartElementNode(offset=0x221) --> TimeCreated
                  AttributeNode(offset=0x250) --> SystemTime
                    ConditionalSubstitutionNode(offset=0x273)
                  CloseEmptyElementNode(offset=0x277)
                OpenStartElementNode(offset=0x278) --> EventRecordID
                  CloseStartElementNode(offset=0x2a7)
                  ConditionalSubstitutionNode(offset=0x2a8)
                  CloseElementNode(offset=0x2ac)
                OpenStartElementNode(offset=0x2ad) --> Channel
                  CloseStartElementNode(offset=0x2d0)
                  ValueNode(offset=0x2d1)
                    WstringTypeNode(offset=0x2d3) --> Application
                  CloseElementNode(offset=0x2eb)
                OpenStartElementNode(offset=0x2ec) --> Computer
                  CloseStartElementNode(offset=0x311)
                  ValueNode(offset=0x312)
                    WstringTypeNode(offset=0x314) --> XXXX
                  CloseElementNode(offset=0x332)
                OpenStartElementNode(offset=0x333) --> Security
                  AttributeNode(offset=0x35c) --> UserID
                    ConditionalSubstitutionNode(offset=0x377)
                  CloseEmptyElementNode(offset=0x37b)
                CloseElementNode(offset=0x37c)
              ConditionalSubstitutionNode(offset=0x37d)
              CloseElementNode(offset=0x381)
            EndOfStreamNode(offset=0x382)
      Substitutions(offset=0x383)
        UnsignedByteTypeNode(offset=0x3d7) --> 4
        UnsignedByteTypeNode(offset=0x3d8) --> 0
        UnsignedWordTypeNode(offset=0x3d9) --> 0
        UnsignedWordTypeNode(offset=0x3db) --> 8225
        UnsignedWordTypeNode(offset=0x3dd) --> 0
        Hex64TypeNode(offset=0x3df) --> 0x0080000000000000
        FiletimeTypeNode(offset=0x3e7) --> 2018-07-23 09:26:38.272814
        NullTypeNode(offset=0x3ef)
        UnsignedDwordTypeNode(offset=0x3ef) --> 0
        UnsignedDwordTypeNode(offset=0x3f3) --> 0
        UnsignedQwordTypeNode(offset=0x3f7) --> 1812
        UnsignedByteTypeNode(offset=0x3ff) --> 0
        NullTypeNode(offset=0x400)
        NullTypeNode(offset=0x400)
        NullTypeNode(offset=0x400)
        NullTypeNode(offset=0x400)
        NullTypeNode(offset=0x400)
        NullTypeNode(offset=0x400)
        NullTypeNode(offset=0x400)
        BXmlTypeNode(offset=0x400) --> 
          RootNode(offset=0x400)
            StreamStartNode(offset=0x400)
            TemplateInstanceNode(offset=0x404, resident=False)
          Substitutions(offset=0x40e)
            WstringArrayTypeNode(offset=0x41e) --> <string></string>

            UnsignedDwordTypeNode(offset=0x420) --> 168
            BinaryTypeNode(offset=0x424) --> LSBDb2RlOiAgQ09SU1ZDQzAwMDAwNzU3LSBDYWxsOiAgQ09SU1ZDQzAwMDAwNzQxLSBQSUQ6ICAwMDAwMTIwMC0gVElEOiAgMDAwMDEyMTItIENNRDogIEM6XFdJTkRPV1Ncc3lzdGVtMzJcdnNzdmMuZXhlICAgLSBVc2VyOiBOYW1lOiBOVCBBVVRIT1JJVFlcU1lTVEVNLCBTSUQ6Uy0xLTUtMTgg

I wish I could help you more!

Windows Build Issues

Hi,

I'm trying to get a build compiled on Windows 7 x64 w/ VS 2013 but it seems to be bombing out when building libcthreads throwing...

1>------ Build started: Project: libcthreads, Configuration: Release Win32 ------
1>  libcthreads_thread_pool.c
1>..\..\libcthreads\libcthreads_thread_pool.c(44): fatal error C1189: #error :  TP_POOL support not implemented yet
========== Build: 0 succeeded, 1 failed, 10 up-to-date, 0 skipped ==========

I noticed it said multi-threading support is planned? Is there a way to disable the Thread Pool?

Unable to make due to changes in libfvalue

Hi,

When I download the latest source distribution and run

./synclibs.sh
./autogen.sh
./configure
make

I get the following error:

In file included from libfwevt_xml_tag.h:31:0,
from libfwevt_xml_document.h:32,
from libfwevt_template.c:35:
libfwevt_libfvalue.h:35:36: fatal error: libfvalue_split_string.h: No such file or directory
compilation terminated.

It looks like this is caused by this commit to libfvalue. Is the API the same in libfvalue (other than being split into two files)? Can I just update the include statements to the two files or are there other changes that need to be made? Thanks for creating such a useful library!

CMake

Hi,

First thanks a lot for libyal.
I would like to ask you if you are interesting in a PR that add support for CMake?

Thanks

resource_file_get_provider: invalid resource file (runing on linux)

I have been poking at the code, trying to parse various EVTX logs, using latest github releases.
Works very good for the 4 standard logs, compiles as static, GREAT!

The one thing that stumbled me is using resource providers. I couldn't figure a way to point evtxexport to the right path of the (dll in this case) provider.

I am trying to parse "Microsoft-Windows-Application-Experience/Program-Telemetry.evtx" from a mounted image of Win2008(ja) server, mounted under mnt/host/C

/tmp/evtxexport -c windows-932 -r "mnt/host/C/Windows/System32/config" -p "mnt/host/C/Windows/System32" mnt/host/C/Windows/System32/winevt/Logs/Microsoft-Windows-Application-Experience%4Program-Telemetry.evtx

The (head) of the result being:

evtxexport 20160107

Event number                    : 1
Written time                    : Aug 11, 2013 13:12:06.2323313900 UTC
Event level                     : Information (4)
User security identifier        : S-1-5-18
Computer name                   : MASKED
Provider identifier             : {EEF54E71-0661-422D-9A98-82FD4940B820}
Source name                     : Microsoft-Windows-Application-Experience
Event identifier                : 0x000002bd (701)
Resource filename               : %SystemRoot%\system32\aeevts.dll
Unable to export record: 0.

What can be a proper way to handle the %SystemRoot% and/or the whole path to the resource, in this case mnt/host/C/Windows/System32/aeevts.dll ? Is this supposed to be working in Windows (sorry, no testing win environment)?

I guess one way is to add option to replace envvars like %SystemRoot%, or just take the filename and append to -p resource_files_path provided path.

unable to build with visual studio

A few errors happens using visual studio when make libwrc.
Maybe, several files are skipped in project.
libwrc_resource_values.c
libwrc_wevt_channel.c ......

have evtexport handle CRLF platform dependent in embedded in strings

String values in evtx files sometimes contain embedded newlines, which are CRLF because they're written on Windows. In the attached exmaple, the value starting at offset 0x19472 is one such:

Application.evtx.gz

C:\Windows\System32\LogFiles\Scm\SCM.EVM
C:\Windows\servicing\Sessions\Sessions.xml
C:\Windows\System32\LogFiles\Scm\SCM.EVM
C:\Windows\Logs\CBS\FilterList.log
C:\Windows\Temp\WER6E75.tmp.WERInternalMetadata.xml
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\Critical_10.0.10586.0_1_58b6cec169647e71609bf1745452c849866c6e89_00000000_cab_12bc6e95\memory.hdmp
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\Critical_10.0.10586.0_1_58b6cec169647e71609bf1745452c849866c6e89_00000000_cab_12bc6e95\minidump.mdmp

evtxexport writes these string values to stdout witout altering them. Because stdout is a text stream, it translates \n to the platform-appropriate line ending. On Windows the \r\n which ends each line in the above string has its \n translated to \r\n, while on Unix the \n remains an \n. The result is that in evtxexport's output, string values with embedded line endings have \r\r\n in them on Windows and \r\n in them on Unix---neither of which is a platform-appropriate line ending.

The correct thing to do is to translate the \r\n in these strings to \n before writing them to stdout, as then stdout will produce the platform-appropriate line ending. (Note that switching stdout to binary mode would not fix the problem, as in that case the line endings would remain \r\n on Unix.)

possibility of recovering records that were effectively "cleared."?

From: #2 by @ShaneKent

Hello! Firstly, thank you for all of the hard work that you've put into this library. My team and I have found an immense amount of usefulness in what you've created.

That being said, do you have any information on the possibility of recovering records that were effectively "cleared." We have an event log file that was cleared via the Windows EventViewer and we're trying to see if there is a way to recover these deleted records.

Any suggestions or ideas? We're not even sure that it's possible...

Link to Token types does not work on Github?

In the libevtx/documentation/Windows XML Event Log (EVTX).asciidoc file, the link of Token types is dead.

Replace #token_type to #token-types.

make sure libfvalue was built with libfdatetime and libfwnt support

CI test are failing with shared libfvalue
https://api.travis-ci.org/v3/job/547556325/log.txt

make sure libfvalue was built with libfdatetime and libfwnt support

Get complete message

Hi Joachim:
Thanks for your work, the library is very useful to me. But I encounter a problem now, I want to get a complete message from evtx files, which API should I use , can you give me an example? Thank you! Now I can get some strings by using libevtx_record_get_utf8_string.

Should the checksum of chunk be 64-bit?

In the file documentation/Windows XML Event Log (EVTX).asciidoc in Chunk header section says:

| Offset | Size | Value | Description
...
|124 | 4 |   | Checksum CRC32 of the first 120 bytes and bytes 128 to 512 of the chunk.

But the value is a Int64, need 8 bytes: from 124 to 132, by example: '\x8c\x95\xaf\xac\x00\x00\x00\x00'` is 2897188236, four bytes is int32 only:

>>> struct.unpack('<i', b'x8c\x95\xaf\xac')
(-1397779060,)
>>> struct.unpack('<q', b'\x8c\x95\xaf\xac\x00\x00\x00\x00')
(2897188236,)

In my file the CRC32 of bytes 0 to 120 and 128 to 512 is 2897188236, the checksum is correct when use int64.

The same problem happens with Event records checksum, is a int64, need 8 bytes, from 52 to 60. In my file the value is 8"8\xd7\x00\x00\x00\x00: 3610780216, same value using a int32 with 4 bytes is -684187080.

Issues using libevtx as a library

Hi,

I am able to compile (from the git repo) and use evtxtools as intended. However, I would like to use libevtx as a library to parse and convert evtx files in my own code. I'm attempting to use parts of evtxtools/evtxtools.c. However, when I try to compile (I have included all the appropriate dependency directories) I am getting many errors such as:

/home/user/Documents/libevtx/evtxtools/evtxtools_libcerror.h:45:23: fatal error: libcerror.h: No such file or directory

When I run a find command looking for "libcerror.h" it does not find anything. However, it appears that there is a libcerror_error.h file. If I modify the include a similar error pops up for a different file. I have downloaded all the dependencies using synclibs.sh. Do you have any suggestions? Thanks!

Unescaped ampersand character in EventXML attribute value output?

Hi, I was trying to parse a file and run into an invalid character

The sample was taken from a Microsoft-Windows-DriverFrameworks-UserMode%4Operational.evtx

The XML looks more or less like this:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
  </System>
  <UserData>
    <UMDFHostDeviceArrivalBegin instance="WPDBUSENUMROOT\UMB\2&37C117B&0&STORAGE#VOLUME#_??_USBSTOR#DISK" lifetime="{ABABAB-ABAB-ABAB-ABABA-ABABABAB}" xmlns:auto-ns2="http://schemas.microsoft.com/win/2004/08/events" xmlns="http://www.microsoft.com/DriverFrameworks/UserMode/Event"/>
  </UserData>
</Event>

The & ampersand should be escaped with &

libevtx_file_get_recovered_record fail

Hi,

As I try to use the libevtx to extract some log information, I successfully get the number of recovered_record but calling libevtx_file_get_recovered_record may fail.
For now, I have three files and only one give correct information on this call, others return error :
libevtx_file_get_recovered_record: unable to retrieve record values: 7.
Can you help me with this and say to me what may be the issue?

Also, I wonder what is the difference between record and recovered_record.

Best regards,
Pierrick

libfwevt_xml_document_read_element: invalid template value size value out of bounds.

I am trying to export evtx information from the log files, but I get below error. EVTX files are opening without any problem in Windows EventViewer. evtxinfo is working fine too, and displays number of records correctly.

Error I get is below.

Unable to export file.
libfwevt_xml_document_read_element: invalid template value size value out of bounds.

invalid XML file output

Thanks for writing this library, i think it will be very useful to me.
A minor nit however regarding evtxexport.
When you ask for xml output (-f xml) the output is not valid xml because

The first line contains the program name - this line should either be omitted or surrounded with XML comment ()
there is no 'root' tag surrounding the individual tags, say <Events>[<Event>]*</Events>
Wrt 'recovered' entries - the xml was just not valid e.g., missing end tag ()

changes for 1 requires moving the call to evtxoutput_version_fprint after argument processing so the export_format is known:
`if( evtxexport_export_handle->export_format == EXPORT_FORMAT_XML )
{
fprintf(stdout, "<!-- \n");
}

evtxoutput_version_fprint(
 stdout,
 program );

if( evtxexport_export_handle->export_format == EXPORT_FORMAT_XML )
{
fprintf(stdout, "\n-->\n");
}`

changes for 2: are just a couple of additional fprintf s in export_handle.c

Unable to export file

hi Joachim,

I encounter the above mentioned error by trying to export the contents of the evtx file Microsoft-Windows-Ntfs%4Operational.evtx with evtxexport :

Unable to export file.
libfwevt_xml_document_substitute_template_value: invalid template value size value out of bounds.
libfwevt_xml_document_read_normal_substitution: unable to substitute template value.
libfwevt_xml_document_read_element: unable to read normal substitution.
libfwevt_xml_document_read_element: unable to read element.
libfwevt_xml_document_read_template_instance: unable to read element.
libfwevt_xml_document_substitute_template_value: unable to read document template instance.
libfwevt_xml_document_read_optional_substitution: unable to substitute template value.
libfwevt_xml_document_read_element: unable to read optional substitution.
libfwevt_xml_document_read_template_instance: unable to read element.
libfwevt_xml_document_read_fragment: unable to read document template instance.
libfwevt_xml_document_read_with_template_values: unable to read fragment header.
libfwevt_xml_document_read: unable to read XML document.
libevtx_record_values_read_xml_document: unable to read binary XML document.
libevtx_io_handle_read_chunk: unable to read record values XML document.
libfdata_list_get_element_value: unable to read element data at offset: 0x00005138.
libfdata_list_get_element_value_by_index: unable to retrieve element value.
libevtx_file_get_record: unable to retrieve record values: 13.
export_handle_export_records: unable to retrieve record: 13.
export_handle_export_file: unable to export records.

my python script based on libevtx with pyevtx also fails with same error :

Traceback (most recent call last):
  File "/usr/local/bin/evtx2tsv", line 110, in <module>
    dump_records(evtx.records)
  File "/usr/local/bin/evtx2tsv", line 97, in dump_records
    for record in records:
OSError: pyevtx_file_get_record_by_index: unable to retrieve record: 13. libfwevt_xml_document_substitute_template_value: invalid template value size value out of bounds. libfwevt_xml_document_read_normal_substitution: unable to substitute template value. libfwevt_xml_document_read_element: unable to read normal substitution. libfwevt_xml_document_read_element: unable to read element. libfwevt_xml_document_read_template_instance: unable to read element. libfwevt_xml_document_substitute_template_value: unable to read document template instance. libfwevt_xml_document_read_optional_substitution: unable to substitute template value. libfwevt_xml_document_read_element: unable to read optional substitution. libfwevt_xml_document_read_template_instance: unable to read element. libfwevt_xml_document_read_fragment: unable to read document template instance. libfwevt_xml_document_read_with_template_values: unable to read fragment header. libfwevt_xml_document_read: unable to read XML document. libevtx_record_values_read_xml_document: unable to read binary XML document. libevtx_io_handle_read_chunk: unable to read record values XML document. libfdata_list_get_element_value: unable to read element data at offset: 0x00005138. libfdata_list_get_element_value_by_index: unable to retrieve element value. libevtx_file_get_record: unable to retrieve record values: 13.

the evtx file Microsoft-Windows-Ntfs%4Operational.evtx does not appear corrupted because it is correctly exported from Windows : if you wish, I can send you the exported and the original evtx file...

regards, lacsaP.

Invalid XML character

I haven't traced down the root cause of this, but i am getting a number of Events with ^C characters for the LockoutObservationWindow and/or the MinPasswordLength.
These are not valid XML characters and cause xml parsers to fail, event with an xml 1.1 parser these would have to at least be character entity references not simple bytes.
Not sure if this is some issue with the virtual machine setup. It is a windows 7 vmware-based vm.

Could someone give me a pointer in the code where such characters would be generated so i can prevent them from getting into the xml stream. Thanks,

an example:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}"/>
    <EventID>4739</EventID>
    <Version>0</Version>
    <Level>0</Level>
    <Task>13569</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8020000000000000</Keywords>
    <TimeCreated SystemTime="2016-10-25T16:36:27.531877000Z"/>
    <EventRecordID>5040</EventRecordID>
    <Correlation/>
    <Execution ProcessID="656" ThreadID="716"/>
    <Channel>Security</Channel>
    <Computer>WIN7VM.local</Computer>
    <Security/>
  </System>
  <EventData>
    <Data Name="DomainPolicyChanged">Lockout Policy</Data>
    <Data Name="DomainName">WIN7VM</Data>
    <Data Name="DomainSid">S-1-5-21-1424057123-4072980456-840877789</Data>
    <Data Name="SubjectUserSid">S-1-5-18</Data>
    <Data Name="SubjectUserName">WIN7VM$</Data>
    <Data Name="SubjectDomainName">DOMAIN</Data>
    <Data Name="SubjectLogonId">0x00000000000003e7</Data>
    <Data Name="PrivilegeList">-</Data>
    <Data Name="MinPasswordAge">-</Data>
    <Data Name="MaxPasswordAge">-</Data>
    <Data Name="ForceLogoff">-</Data>
    <Data Name="LockoutThreshold">5</Data>
    <Data Name="LockoutObservationWindow">�</Data>
    <Data Name="LockoutDuration"/>
    <Data Name="PasswordProperties"/>
    <Data Name="MinPasswordLength">�</Data>
    <Data Name="PasswordHistoryLength"/>
    <Data Name="MachineAccountQuota"/>
    <Data Name="MixedDomainMode">-</Data>
    <Data Name="DomainBehaviorVersion">-</Data>
    <Data Name="OemInformation">-</Data>
  </EventData>
</Event>

add (content) creation time to evtxexport output

Per https://blog.fox-it.com/2019/06/04/export-corrupts-windows-event-log-files/

add (content) creation time (when present) to evtxexport output (mainly as UX improvement)
describe corruption scenario in documentation

for your information: other uses of binary xml

over at williballenthin/python-evtx#42 there's an interesting request for a parser for the EVTX log entries queried via RPC. notably, this API returns a binary blob per event record. while the blob header is different from the EVTX record header, its obvious most of the binary XML format is shared. perhaps at your convenience, you might also be interested in reviewing this data.

for example, since records are transmitted independently, strings and templates seem to be placed in-line, rather than defined in a shared location. i think there are a few flags in the node headers which dictate this; however, i'm still exploring.

i'm not expecting anything from you here --- just passing along the pointer.

Please add data types to the information tables

In the file documentation/Windows XML Event Log (EVTX).asciidoc in the data information tables need the data types, by example if the value is plain bytes, dword, int32, int64, etc. By example, the record identifier is a int64 or dword?

Feature Request: "relaxed" mode parsing for use on files recovered from memory

Because of the strict file checks, it is not always possible to libevtx (and thus plaso etc.) to parse recovered EvtX files (i.e. via dumpfiles in volatility)

e.g. using https://github.com/williballenthin/EVTXtract as a baseline

Forensics:~> evtxtract artefacts/volatility/files/file.None.0xffffe001f66235d0.Microsoft-Windows-Sysmon%4Operational.evtx.dat >>/dev/null
INFO:root:recovered 174 complete records
INFO:root:recovered 0 incomplete records

Forensics:~> evtxexport artefacts/volatility/files/file.None.0xffffe001f66235d0.Microsoft-Windows-Sysmon%4Operational.evtx.dat
evtxexport 20170122

Unable to open: artefacts/volatility/files/file.None.0xffffe001f66235d0.Microsoft-Windows-Sysmon%4Operational.evtx.dat.
libevtx_io_handle_read_file_header: unsupported file signature.
libevtx_file_open_read: unable to read file header.
libevtx_file_open_file_io_handle: unable to read from file handle.
libevtx_file_open: unable to open file: bsidesau2018/artefacts/volatility/files/file.None.0xffffe001f66235d0.Microsoft-Windows-Sysmon%4Operational.evtx.dat.
export_handle_open_input: unable to open input file.

Should the number of chunks be 32-bit?

In the file documentation/Windows XML Event Log (EVTX).asciidoc in File header section says:

| Offset | Size | Value | Description
...
| 42 | 2 | | Number of chunks

But the value is a Int32, need 4 bytes: from 42 to 46, by example: \xfe\xff\x00\x00 are 65534 chunks, two bytes is int16 only:

>>> struct.unpack('<h', b'\xfe\xff')
(-2,)
>>> struct.unpack('<i', b'\xfe\xff\x00\x00')
(65534,)

And Unknown (Empty values) are from 46 to 76.

Additional python record properties

Would be it possible to add a method and descriptor to the pyevtx record for Channel?

data in records of pyevtx and <EventData> xml node

hi Joachim,
will data attribute of a pyevtx.record object be the same thing as <EventData> xml node ?
if not, how can I get just <EventData> node without playing xml on xml_string attribute ?
regards, lacsaP.

libyal / libevtx Goto Github PK

libevtx's Introduction

libevtx's People

Contributors

Stargazers

Watchers

Forkers

libevtx's Issues

Recommend Projects

Recommend Topics

Recommend Org