[File] [PATCH] Magdir/windows Microsoft Event Trace Logs *.ETL + EVTX
Jörg Jenderek
joerg.jen.der.ek at gmx.net
Mon Oct 3 00:23:20 UTC 2022
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello,
Some days ago i run the cleaning tool czkawka found on
https://qarmin.github.io/czkawka/. One menu item concerns bad
extensions. After running tool i looked in saved file list
results_bad_extensions.txt for bad extension examples.
One listed extension is ETL.
These file are Microsoft Event Trace Logs found on Windows systems at
different locations. When running file command version 5.43 on such
ETL examples and related files i get an output like:
NPETraceSession.etl: Apple HFS/HFS+ resource fork
OpenSSH%4Admin.evtx: MS Windows Vista Event Log,
1 chunks (no. 0 in use), empty, DIRTY
RestoreUI.3.etl: data
Wifi.etl: dBase III DBT, version number 0,
next free block index 81920, 1st item
"10586.494.amd64fre.th2_release_
WindowsBackup.4.etl: GeoSwath RDF
boot_BASE+CSWITCH_1.etl: dBase III DBT, version number 0,
next free block index 1048576, 1st item
"\377\377\377\377\377\377\377\377\377\377
dumpfile.evtx: MS Windows Vista Event Log,
7 chunks (no. 6 in use), next record no. 286
service.0.etl: data
Furthermore for ETL samples wrong mime type is shown with -i option.
With option --extension 3 byte sequence etl is not shown.
For comparison reason i run the file format identification utility
TrID ( See https://mark0.net/soft-trid-e.html). This describes many
examples (690 of 758) correctly as "Window tracing/diagnostic binary
log" by definition etl.trid.xml (See appended trid-etl-v.txt.gz).
For comparison reason i also run the file format identification
utility DROID ( See https://sourceforge.net/projects/droid/). This
does not recognize the ETL samples.
That these are Trace Logs files can be verified by commands like:
tracerpt.EXE AutoLogger-Diagtrack-Listener.etl -of EVTX
etl2xml --input AMSITrace.etl --output AMSITrace.xml
Luckily i found some unofficial file format information. So that
informations are now expressed by comment lines after EVTX
section inside Magdir/windows like:
# URL: http://fileformats.archiveteam.org/wiki/ETL
# Reference: http://mark0.net/download/triddefs_xml.7z
# defs/e/etl.trid.xml
# https://www.geoffchappell.com/studies/windows/
# km/ntoskrnl/api/etw/tracelog/trace_logfile_header.htm
Unfortunately ETL have no strong magic at the beginning. So i put
displaying part inside sub routine trace-etl, This looks like:
0 name trace-etl
>0 ubyte x Windows Event Trace Log
!:mime application/etl
!:ext etl
>0 search/0x2b4/sb :\0\x5c\0
>>&-2 lestring16 x "%s"
According to unofficial Software Analyst Geoff Chappell after the
trace log file header the names of the logger and the log file are
stored as null-terminated Unicode strings. By last line the original
log file name (like "c:\Windows\Logs\NetSetup\service.0.etl"
"C:\Windows\System32\LogFiles\WMI\Wifi.etl") is shown. The header and
the sizes are different from variant (32- or 64-bit) and also from
version to version. So i use a brute force way to find the beginning
by looking for DOS drive letter part ( string :\ encoded as UTF-16).
For example boot_BASE+CSWITCH_1.etl this does not work, because the
real log file name appears later (about 1441610h) and other ETL
names appear before (about 699080h).
According to website nirsoft.net it has an own mime type.
That is what i found on my systems after i have installed the Windows
Toolkit. Then ETL files are connected via registry as "wpa.etl_file"
with Windows Performance Analyzer wpa.exe.
But unfortunately it is not officially registered as IANA. Oh,
Microsoft now says they become friends of open source, wants to
improve user experience and protect the user by things like secure
boot. This is like illness. They try to care about the symptoms and
not the basic reasons. That is not surprising, when there exist many
places inside the windows system, which are not explained and
documented. So they must make high efforts to protect directories
by their virus defender software. This would not be necessary if
things are well documented. Remember, Microsoft is not a well fare
society. It is a company to make money and in the past the used
methods like mafia. So why they should have changed their behaviour
now. So you can not trust them like Putin. The gifts they offered
maybe under Tolkien motto "a system to find, bind and bring them
all in the darkness to becomes slaves". Even when Microsoft would
promise to be not evil. This does not help when an American
government like Trump arises again and forces Microsoft as American
company by law to give all data to him. The counter part in my eyes
is Linus Torvalds. He has never tried to make "big" business. From
the beginning he shared his ideas with all people. So become
prepared that you can switch to alternatives when things become worse
.
Translating the TrID definitions the test for detections starts like:
0 ubyte 0
>0 search/0x699086/b .\0e\0t\0l\0\0\0
>>0 use trace-etl
According to Geoff Chappell every ETL file begins with a
WMI_BUFFER_HEADER, a SYSTEM_TRACE_HEADER and a TRACE_LOGFILE_HEADER.
This seems to start with a nil byte, but i do not know if this is
always true. After the trace log file header the names of the logger
and the log file are stored as null-terminated Unicode strings. So
ETL files contain near the beginning the bytes sequence
.\0e\0\t\0l\0\0\0 triggered by used etl file name suffix encoded as
UTF-16. Normally this search range is only some hundreds bytes, but
for example boot_BASE+CSWITCH_1.etl i must raise this range up
699086h. Unfortunately this only works if in src/file.h
FILE_BYTES_MAX is raised above 699086h (6,59 MiB).
There exist a section for Vista Event Log (*.EVTX) inside
Magdir/windows which are related with ETL files. The mentioned URL
does not exist any more. So i replace it with archived version and
add also site with more informations. So this now becomes like:
# URL: https://github.com/libyal/libevtx/blob/main/
# documentation/
# Windows%20XML%20Event%20Log%20(EVTX).asciidoc
# Reference: https://web.archive.org/web/20110803085000/
# https://computer.forensikblog.de/en
# /2007/05/some_magic.html
# http://mark0.net/download/triddefs_xml.7z
# defs/e/evtx.trid.xml
The description happens by lines starting like:
0 string ElfFile\0 MS Windows Vista Event Log
>0x2a leshort x \b, %d chunks
This now becomes like:
0 string ElfFile\0 MS Windows
!:mime application/x-ms-evtx
!:ext evtx
>0x24 ulelong =0x00030001 Vista-8.1 Event Log
>0x24 ulelong !0x00030001 10-11 Event Log, version
>>0x26 uleshort x %u
>>0x24 uleshort x \b.%u
>0x2a leshort x \b, %d chunks
Instead of generic application/octet-stream mime type i show an user
defined one. According to the documentation for Vista and later the
version is 3.1 and for Windows 10 (2004) and later the version is
3.2. Because it is irritating when on newer Windows system files are
described as "Vista Event Log". So for the first case show phrase
"Vista-8.1" instead "Vista" and for newer Windows systems show phrase
"10-11" instead "Vista" and also the version number.
After applying the above mentioned modifications by patch
file-5.43-windows-etl.diff then most ETL samples are now described
correctly (depending on FILE_BYTES_MAX ins src/file.h). 17 examples
are not recognized. But these seem to be only two examples
(EtwRTEventlog-Security.etl EtwRTDiagLog.etl) in different
directories. In found these examples on Vista system. All these
examples start with string rlfs at the beginning and contains no
readable file name. So i do not know if this triggered by accident or
if this another "undocumented" feature of Microsoft. This with option
- -m Magdir/windows now looks like:
NPETraceSession.etl: Windows Event Trace Log
"C:\Dokumente und Einstellungen\user_1234\
Lokale Einstellungen\Anwendungsdaten\
NPE\NPETraceInProgress.etl"
OpenSSH%4Admin.evtx: MS Windows 10-11 Event Log, version 3.2,
1 chunks (no. 0 in use), empty, DIRTY
RestoreUI.3.etl: Windows Event Trace Log
"C:\WINDOWS\Logs\SystemRestore\
RestoreUI.0.etl"
Wifi.etl: Windows Event Trace Log
"C:\Windows\System32\LogFiles\WMI\Wifi.etl"
WindowsBackup.4.etl: Windows Event Trace Log
"C:\Windows\Logs\WindowsBackup\
WindowsBackup.0.1.etl"
boot_BASE+CSWITCH_1.etl: Windows Event Trace Log
dumpfile.evtx: MS Windows Vista-8.1 Event Log, version 3.1,
7 chunks (no. 6 in use), next record no. 286
service.0.etl: Windows Event Trace Log
"C:\Windows\Logs\NetSetup\service.0.etl"
I hope my diff file can be applied in future version of file
utility.
With best wishes,
Jörg Jenderek
- --
Jörg Jenderek
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iF0EARECAB0WIQS5/qNWKD4ASGOJGL+v8rHJQhrU1gUCYzorYAAKCRCv8rHJQhrU
1vYdAKCs0mgoxnFO8vFKxWkFvYC4q6KqjgCeJosNM8c8vwzmYynhW2n8C6o2qDg=
=hHdi
-----END PGP SIGNATURE-----
-------------- next part --------------
--- file-5.43/magic/Magdir/windows.old 2022-07-06 20:56:40.000000000 +0200
+++ file-5.43/magic/Magdir/windows 2022-10-03 01:41:15.247421100 +0200
@@ -117,6 +117,19 @@
# Summary: Vista Event Log
-# Extension: .evtx
# Created by: Andreas Schuster (https://computer.forensikblog.de/)
-# Reference (1): https://computer.forensikblog.de/en/2007/05/some_magic.html
-0 string ElfFile\0 MS Windows Vista Event Log
+# Update: Joerg Jenderek
+# URL: https://github.com/libyal/libevtx/blob/main/documentation/Windows%20XML%20Event%20Log%20(EVTX).asciidoc
+# Reference (1): https://web.archive.org/web/20110803085000/
+# https://computer.forensikblog.de/en/2007/05/some_magic.html
+# http://mark0.net/download/triddefs_xml.7z/defs/e/evtx.trid.xml
+# Note: called "Vista Event Log" by TrID and "Event Log" by Windows
+# verified partly by `wevtutil.exe gli /lf:true dumpfile.evtx`
+0 string ElfFile\0 MS Windows
+#!:mime application/octet-stream
+!:mime application/x-ms-evtx
+!:ext evtx
+# Major+Minor format version: 3.1~Vista and later 3.2~Windows 10 (2004) and later
+>0x24 ulelong =0x00030001 Vista-8.1 Event Log
+>0x24 ulelong !0x00030001 10-11 Event Log, version
+>>0x26 uleshort x %u
+>>0x24 uleshort x \b.%u
>0x2a leshort x \b, %d chunks
@@ -128,2 +141,28 @@
+# Summary: Windows Event Trace Log
+# From: Joerg Jenderek
+# URL: http://fileformats.archiveteam.org/wiki/ETL
+# Reference: http://mark0.net/download/triddefs_xml.7z/defs/e/etl.trid.xml
+# https://www.geoffchappell.com/studies/windows/km/ntoskrnl/api/etw/tracelog/trace_logfile_header.htm
+# Note: called "Window tracing/diagnostic binary log" by TrID
+# verified by `tracerpt.EXE Wifi.etl -of EVTX`
+# and by etl-parser `etl2xml --input AMSITrace.etl --output AMSITrace.xml`
+# Every ETL file begins with a WMI_BUFFER_HEADER, a SYSTEM_TRACE_HEADER and a TRACE_LOGFILE_HEADER
+0 ubyte 0
+# look for corresponding encoded as UTF-16 file name extension like in: boot_BASE+CSWITCH_1.etl
+>0 search/0x699087/b .\0e\0t\0l\0\0\0
+# GRR: line above only works if in ../../src/file.h FILE_BYTES_MAX is raised above 699086h (6,59 MiB)
+>>0 use trace-etl
+# display information of Windows Performance Analyzer Trace File (file name)
+0 name trace-etl
+>0 ubyte x Windows Event Trace Log
+#!:mime application/x-ms-etl
+# http://extension.nirsoft.net/etl
+!:mime application/etl
+!:ext etl
+# look for DOS drive letter part of log file name like: PhotosAppTracing_startedInBGMode.etl
+>0 search/0x2b4/sb :\0\x5c\0
+# like: "c:\Windows\Logs\NetSetup\service.0.etl" "C:\Windows\System32\LogFiles\WMI\Wifi.etl"
+>>&-2 lestring16 x "%s"
+
# Summary: Windows System Deployment Image
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-5.43-windows-etl.diff.sig
Type: application/octet-stream
Size: 1553 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20221003/cd85c124/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: trid-v-etl.txt.gz
Type: application/x-gzip
Size: 847 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20221003/cd85c124/attachment-0001.bin>
More information about the File
mailing list