n4r1b / ferrisetw Goto Github PK
View Code? Open in Web Editor NEWBasically a KrabsETW rip-off written in Rust
License: Other
Basically a KrabsETW rip-off written in Rust
License: Other
I believe this is used for enums.
This repository doesn't have a license file. I see that the Cargo.toml says "MIT or Apache 2.0".
Recommend adding LICENSE.md files to more clearly indicate the license.
ferrisetw has a thread safety issue(xiaopengli89/precord#9), I checked the code and the cause of the issue should be here:
https://github.com/n4r1b/ferrisetw/blob/master/src/trace.rs#L99
This would be quite useful.
See also: microsoft/krabsetw#160
Example code in doctests are not checked by the compiler.
We should enable them, e.g. by having cargo test --doc
in the GitHub Actions config.
Note that this requires fixing them, because none of them is currently passing (mainly because they are missing use ...
)
Info
ETW allows filters to be defined for a Provider in a session. MSDN - Defining Filters.
KrabsETW already provides a mechanism to do filtering either by events_id or by using more complicated predicates
TODO
Hi, let me first thank you for making this crate, it allowed me to do what I needed to do. And it has mostly "just worked", which is always a pleasure!
But I hit some limitations as well, for example with the variable length "Address" properties of some events from the Microsoft.Windows.Winsock.AFD provider, because their length is stored indirectly in another property. And this is just a TODO in the code so far...
Luckily for me, I found there was a fork of ferrisetw (jxy-s) that already had this feature implemented, together with some extra serialization support and whatnot. So I just changed my Cargo.toml to reference that repo/branch instead of the published crate and carried on. Great.
But now I'd like to go further with my project and referencing dependencies from random Git repos is not really feasible, so I have a basic question for you, @n4r1b: How much time do you have for this project, would you be willing to work on adding/integrating this feature, release a new version at some point etc.?
Of course, usually it would be rather the author of the fork going back to the upstream with a PR. So a similar question to @jxy-s here, would you be interested in making some PRs to get your improvements merged upstream maybe?
And I know everybody's time is limited, so I could help with rebasing/cleaning/testing/publishing PRs too. I'd just like to discuss first before stepping on anybody's toes 🙂
Thank you
That's what is suggested by Clippy
warning: usage of `Box<Arc<CallbackData>>`
--> src\trace.rs:173:20
|
173 | callback_data: Box<Arc<CallbackData>>,
| ^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::redundant_allocation)]` on by default
= note: `Arc<CallbackData>` is already on the heap, `Box<Arc<CallbackData>>` makes an extra allocation
= help: consider using just `Box<CallbackData>` or `Arc<CallbackData>`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_allocation
In our case, we do not only want the CallbackData
to be on the heap, but the ref counter as well.
https://github.com/rust-lang/rust/blob/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/sync.rs#L352 suggests the ref counters (both atomic::usize
s) really live on the heap, so this change should be sensible.
That's to avoid an allocation, on a struct that's only created a handful of times, this currently properly works that way, and it's a private implementation detail of the crate (so it can be changed at anytime without any trouble for the users), so I'm leaving this for later.
Parser::try_parse("ThisPropertyDoesNotExist")
returns the last property
I added support for these to etw-reader to handle the KernelTraceControl events: https://github.com/jrmuizel/etw-profiler/blob/main/etw-reader/src/custom_schemas.rs
This was done by creating an EventSchema
trait: https://github.com/jrmuizel/etw-profiler/blob/2dddbc2eb2c02e93b14f69aac60895fe7efe82ef/etw-reader/src/schema.rs#L86
Hello n4r1b,
Here are a few ideas that could improve the crate, but that would likely break the compatibility, so they would probably need bumping to a new major version.
(note: I have more or less planned to work on them, and I already have a few draft branches locally, I'll probably make MRs soon, but I'm currently focusing on non-breaking changes first)
Feel free to comment or discuss them, maybe there are better solutions to these issues I haven't thought about yet
// TODO: Add Filters option
). As a kind of side-effect in my draft branch, this slightly modified a public API, and thus should be left for a major bumpTraceFlags
instead of a u32
open
, start
and process
. One of them is probably not required.PropertyInfo
or Schema
. This issue comes from krabsetw names, but I feel they're unfortunate because they do not really convey their actual meaning (could be respectively PropertyWithBuffer
and EventAndSchema
for instance)Parser
contains the Schema
, which contains the EventRecord
, the Parser owns the Event data, and that feels weird. If I read correctly, this even prevents an ideal caching of such structs.Provider
(e.g. some methods should be mutually exclusive and called only once, such as by_guid
and by_name
)Provider
and Trace
have setters for fields that are pub
. We could probably have only one of themUserTrace
(or KernelTrace
). We cannot do this currently, as we feeding OpenTraceA
with pointers to UserTrace::data
that must not be moved during the lifetime of the trace.EventTraceLogfile
EventRecord
should be a read-only safe wrapper, instead of a simple newtyperecord.ExtendedDataCount
Ideas from #25 most likely require an API change, and hence a major version bump as well
A SID is a pretty common field within ETW events so providing a way to parse it into a String is a must.
See KrabsETW implementation KrabsETW - parser.hpp#L368
Hi,
I really like the work you've done here, good job!
For my purposes, I usually want to start a trace, and run it until I quit the program. In the current implementation Ferris spawns a new anonymous thread to do the processing, which means I need to either do a 'sleep forever' or 'wait for user input' hack.
Would it be possible to either:
Trace
struct, so I could do a .join on itI've hit a roadblock when using this library to gather stack walking information on sampled profiles. Even when wrangling the control handle from the trace with transmutes after calling start()
, it's too late to set what I need, and it ends up not working.
I noticed there was this comment:
// TODO: For kernel traces, implement enable_provider function for providers that require call to TraceSetInformation with extended PERFINFO_GROUPMASK
If either a simple callback with the control handle, or a full solution for managing kernel trace information could be added, that would save a lot of headaches.
Schema::properties()
is currently pub(crate)
, but is there any reason it shouldn't be just pub
? It's very useful for being able to inspect the properties in a schema; I couldn't find any other way of getting this list.
We're currently feeding String::as_ptr()
to A versions of the Windows API.
This means we'll have unexpected results and/or UB when using non-ASCII traces names for instance.
There is no specific synchronization mechanism in trace_callback_thunk()
.
Which means a thread could be closing the trace session and destroying the TraceData
, just at the same time the last callback was triggered. And the event_record.user_context
dereferenced in another thread, supposed to point to a TraceData
could be dangling here
If a thread closes the trace session and destroys the TraceData
, the associated Provider
, hence the associated callbacks
, which are closures, are dropped. That's a problem in case a callback is still in progress (e.g. a callback is stuck in a blocking function call), because the closure may contain state (e.g. in a move ||
closure).
krabsetw may have the same issues. See the issue in krabsetw
Here are a few ideas I had when reading the code before profiling it.
Feel free to add any remarks and comment :)
How important and efficient are all these ideas?
TODO: use a profiler to benchmark the few places that look time-consuming.
EVENT_RECORD
is Copy
. Depending on how the compiler optimizes it, it is possibly copied at every function call:
ctx.on_event(*event_record);
prov.on_event(record, locator);
, once for each providercallbacks.iter_mut().for_each(|cb| cb(record, locator))
, once for each callback of each providerEVENT_RECORD
is quite large).Solution:
&EVENT_RECORD
.Schema
to not own itThat's for the event payload part.
Considering the ETW schema, it is properly cached in the SchemaLocator
and is retrieved quickly.
One of the first steps in the callback is to call Parser::create(&schema)
. This
PropertyIter::enum_properties()
for every event record, although this only depends on the schema, not on the record itself!
enum_properties()
builds a Vec<Property>
)parser.try_parse(...)
does many things. But most work is done in find_property()
Parser
...which depends on the event recordCurrently, the callbacks are passed an EVENT_RECORD
and a SchemaLocator
.
As stated in a TODO in the code, this is not straighforward. We could/should:
&EVENT_RECORD
, see above)SchemaLocator
, but the Schema
directlySchema
. Let's keep giving them the ability to retrieve it or not)Schema
would probably not own the event record (nor a ref to it).Parser
instead of a Schema
is probably not a good idea. The end user may want to avoid its creation on most events, and create it only for e.g. event IDs that interest himMaybe ferrisetw could be used to list and control already existing traces.
See QueryAllTracesW
: https://learn.microsoft.com/en-us/windows/win32/api/evntrace/nf-evntrace-queryalltracesw
Could it also stop them? Subscribe to them? etc.
windows-rs version in ferrisetw's Cargo.toml is 0.39, but windows-rs version in my project is 0.42. This will cause the following error when compiling:
error[E0277]: the trait bound `Parser<'_>: TryParse<GUID>` is not satisfied
--> src/main.rs:23:66
|
23 | let guid: GUID = TryParse::<GUID>::try_parse(&mut parser, "Guid").unwrap();
| --------------------------- ^^^^^^^^^^^ the trait `TryParse<GUID>` is not implemented for `Parser<'_>`
| |
| required by a bound introduced by this call
|
= help: the following other types implement trait `TryParse<T>`:
My project can only be compiled by using windows-rs with the same version as ferrisetw.
Is there any good solution?
EventFilter::ByPids
are only effective on kernel mode logger session.
see https://learn.microsoft.com/en-us/windows/win32/api/evntprov/ns-evntprov-event_filter_descriptor:
The PIDs based filter-blob is only valid for a kernel mode logger session because the private logger session runs inside a user-mode process
But this does not work for KernelTrace
s in ferrisetw. This would be good to support it.
Ideas:
ferrisetw::KernelTrace
one of them in the first place?If this eventually works, this should be added in an integration test
In ferrisetw 0.1, UserTrace::enable
should be used to add a Provider to a trace. But, IIRC, this has no effect on a trace that has started already (the provider is pushed to a list, but EnableTraceEx2
is not called on this new trace).
To fix this problem, the next_major_version
branch reworks this enable
function and forces to define all Providers before the trace is started.
But the Windows API allows enabling/disabling providers when the trace is running (using EnableTraceEx2
, so this could be worth supporting this in ferrisetw
Property::new()
can return PropertyError::UnimplementedType
, depending on the TDH property flags.
This would be good to support more types (e.g. PropertyFlags::PROPERTY_STRUCT
, or PropertyFlags::PROPERTY_PARAM_LENGTH
)
I have a provider binary that emits n
number of different events over the same provider GUID using TraceLogging. Here's an example:
static const SID sid = { SID_REVISION, 1, 5, { 18 } };
TRACELOGGING_DECLARE_PROVIDER(g_hProvider);
// 3B9CAB28-762A-4740-A82B-B6829CC90ADF
TRACELOGGING_DEFINE_PROVIDER(
g_hProvider,
"My-Test-Provider",
(0x3b9cab28, 0x762a, 0x4740, 0xa8, 0x2b, 0xb6, 0x82, 0x9c, 0xc9, 0xa, 0xdf));
int main() {
TraceLoggingRegister(g_hProvider);
EmitEID1();
for (int i = 0; i < 5; i++) {
EmitEID2();
}
TraceLoggingUnregister(g_hProvider);
}
void EmitEID1() {
TraceLoggingWrite(g_hProvider,
"ProcessCreation",
TraceLoggingUInt8(1, "EventId"),
TraceLoggingUInt32(44, "Pid"),
TraceLoggingUInt32(400, "ParentPid"),
TraceLoggingWideString(L"test-provider.exe", "ParentProcessName"),
TraceLoggingUInt32(600, "CreatorPid"),
TraceLoggingWideString(L"test-provider.exe", "CreatorProcessName"),
TraceLoggingWideString(L"test-provider.exe", "FileName"),
TraceLoggingBoolean(TRUE, "ExactFileName"),
TraceLoggingWideString(L"testing", "CommandLine"),
TraceLoggingSid(&sid, "Sid"),
TraceLoggingBoolean(FALSE, "SubsystemProcess")
);
}
void EmitEID2() {
TraceLoggingWrite(g_hProvider,
"ThreadCreation",
TraceLoggingUInt8(2, "EventId"),
TraceLoggingUInt32(6000, "CreatorPid"),
TraceLoggingWideString(L"test-provider.exe", "CreatorProcessName"),
TraceLoggingUInt32(444, "TargetPid"),
TraceLoggingWideString(L"test-provider.exe", "TargetProcessName"),
TraceLoggingUInt32(6464, "TargetThreadId"),
TraceLoggingSid(&sid, "Sid")
);
}
I've written a basic consumer application that parses the events into a struct.
use ferristetw::*;
fn main() {
let test_provider = provider::Provider
::by_guid("3B9CAB28-762A-4740-A82B-B6829CC90ADF")
.add_callback(test_callback)
.build();
let test_trace = UserTrace::new()
.enable(test_provider)
.start_and_process()
.unwrap();
std::thread::sleep(std::time::Duration::new(60, 0));
test_trace.stop().unwrap();
}
fn test_callback(record: &EventRecord, schema_locator: &SchemaLocator) {
match schema_locator.event_schema(record) {
Err(err) => println!("Unable to get the ETW schema for event: {:?}", err),
Ok(schema) => parse_event(&schema, record)
}
}
fn parse_event(schema: &schema::Schema, record: &EventRecord) {
let parser = parser::Parser::create(record, schema);
match parser.try_parse::<u8>("EventId").unwrap_or(0) {
2 => {
let event = ThreadCreatedEvent {
id: 2,
description: String::from("Thread created"),
creator_pid: parser.try_parse::<u32>("CreatorPid").unwrap_or(0),
creator_process_name: parser.try_parse::<String>("CreatorProcessName").unwrap_or_else(|_| String::from("")),
target_pid: parser.try_parse::<u32>("TargetPid").unwrap_or(0),
target_process_name: parser.try_parse::<String>("TargetProcessName").unwrap_or_else(|_| String::from("")),
target_thread_id: parser.try_parse::<u32>("TargetThreadId").unwrap_or(0),
sid: parser.try_parse::<String>("Sid").unwrap_or_else(|_| String::from(""))
};
println!("{:?}", event);
}
_ => {}
}
}
In the example, I emit Event 1
once before emitting Event 2
five times. My problem is that all instances of Event 2
will fail to parse correctly (specifically with TdhNativeError(IoError(Os { code: 1168, kind: Uncategorized, message: "Element not found." }))
. If I remove the line containing EmitEID1()
from the example, the events will parse properly. I'm not exactly sure why I'm unable to parse multiple events and any help would be much appreciated.
Currently, we have
EventTraceProperties {
etw_trace_properties,
trace_name: [0; 1024],
log_file_name: [0; 1024],
};
We could avoid using two arrays of 1024 bytes to store a string that's probably shorter than this. Maybe we could make EventTraceProperties
generic on the name length, or really support dynamic allocation of the name
SchemaKey currently ignores the EVENT_HEADER_EXT_TYPE_EVENT_SCHEMA_TL metadata which describes trace logging events. This cause the wrong schema to be used for these events.
jrmuizel/etw-profiler@8a20fa2 is how I worked around this in etw-reader.
Hi there,
I get a panic when I try to call .stop()
on a Kernel Trace. Basic code:
let provider_io = Provider::kernel(&kernel_providers::FILE_IO_PROVIDER)
.build()
.unwrap();
let mut trace = KernelTrace::new()
.named(String::from("HijackWatcher"))
.enable(provider_io)
.start()
.unwrap();
std::thread::sleep(Duration::new(3, 0));
trace.stop();
Strack Trace:
thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', C:\Users\xxx\.cargo\registry\src\github.com-1ecc6299db9ec823\ferrisetw-0.1.1\src\trace.rs:112:30
stack backtrace:
0: 0x7ff6de00a782 - std::backtrace_rs::backtrace::dbghelp::trace
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library\std\src\..\..\backtrace\src\backtrace\dbghelp.rs:98
1: 0x7ff6de00a782 - std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66
2: 0x7ff6de00a782 - std::sys_common::backtrace::_print_fmt
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library\std\src\sys_common\backtrace.rs:66
3: 0x7ff6de00a782 - std::sys_common::backtrace::_print::impl$0::fmt
at /rustc/897e37553bba8b42751c67658967889d11ecd120/library\std\src\sys_common\backtrace.rs:45
Windows version:
OS Name: Microsoft Windows 11 Pro
OS Version: 10.0.22621 N/A Build 22621
hello, I have used the code of this commit(#42). An PropertyError("Invalid widestring pointer")
error occurred when I used the following code:
TryParse::<String>::try_parse(&parser, "appname")
This problem will not occur if i switch to the previous commits.
Trace AlreadyExist,but logman query -ets trace not found,this library has been unregistered method?
In ferrisetw 0.x, there was a macro doing most of the work.
It has been removed in ferrisetw 1.0 because of the different UserTraceBuilder
and KernelTraceBuilder
.
I tried to de-duplicate them, using a TraceBuilder<T: TraceTrait>
, but that is hard because I've ended up needing to make NativeEtw
pub
because they would otherwise be private types leaking to public interfaces. So I gave up.
But there should be a way, one just has to find the correct balance between traits, macros and dedicated impl ... for ...
.
Also, do we need to have separate types after all? There is only one Provider
, we may be OK with a single Trace
type
I've never used Github Actions before, but it would be a nice addition to be able to build, test and check coverage.
See: https://github.com/BamPeers/rust-ci-github-actions-workflow
Since (a PR to come, probably #63), we're ignoring the very last ETW events that were still in the buffers when we called CloseTrace
We may want to process them, and drop the memory structures when we're sure every event is processed.
See
/// TODO: it _might_ be possible to know whether we've processed the last buffered event, as /// ControlTraceW(EVENT_TRACE_CONTROL_QUERY) _might_ tell us if the buffers are empty or not. /// In case the trace is in ERROR_CTX_CLOSE_PENDING state, we could call this after every /// callback so that we know when to actually free memory used by the (now useless) callback. /// Maybe also setting the BufferCallback in EVENT_TRACE_LOGFILEW may help us.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.