This will allow external partners to update the build definitions if they want to add new changes and offloads some of the maintainability from internal teams.

For PullRequests, collabors/admins must comment "/azp run" in order to actually invoke the PR build.

For a method that has generic parameter and/or return value types, I am unable to get more than the CorElementType : ELEMENT_TYPE_GENERICINST (which is 0x15).

I'm doing this from the InstrumentMethod callback. The method that I am inspecting (through InstrumentMethod has a return value type which is a generic class:

System.Collections.Generic.IEnumerable<netfwWebapi.Models.Customer>

My code gets the CorSignature and parses it using the ISignatureParser packaged in the CLR Instrumentation Engine. Here is some code:

	// Get CorSignature
	BYTE pSignature[256] = {};
	DWORD cbSignature = 0;
	pMethodInfo->GetCorSignature(_countof(pSignature), pSignature, &cbSignature);

	CComPtr<IModuleInfo> pModuleInfo;
	pMethodInfo->GetModuleInfo(&pModuleInfo);

	ModuleID moduleId;
	pModuleInfo->GetModuleID(&moduleId);

	// Parse method signature
	CComPtr<ITypeCreator> pTypeFactory;
	pModuleInfo->CreateTypeFactory(&pTypeFactory);
	CComQIPtr<ISignatureParser> pSigParser(pTypeFactory);

	CComPtr<IType> pReturnType;
	CComPtr<IEnumTypes> pEnumTypes;
	DWORD genericParameterCount = 0;
	DWORD sizeUsed = 0;

	HRESULT hr = pSigParser->ParseMethodSignature(
		pSignature,
		cbSignature,
		nullptr,
		&pReturnType,
		&pEnumTypes,
		&genericParameterCount,
		&sizeUsed);
        if (FAILED(hr))
        {
             // logs error and exit out
        }

	// Return Value type
	CorElementType etReturnValue;
	pReturnType->GetCorElementType(&etReturnValue);
	m_returnValueType = getTypeString(etReturnValue, pReturnType, pMethodInfo);

My getTypeString method does something special if the element type is ELEMENT_TYPE_GENERICINST. Here is what it does for generic types:

		CComPtr<IType> pDeclaringType;
		pMethodInfo->GetDeclaringType(&pDeclaringType);
		CComPtr<ITokenType> pDeclaringTokenType;
		pDeclaringType->QueryInterface(__uuidof(ITokenType), (LPVOID*)&pDeclaringTokenType);
		mdToken declaringTypeToken = mdTokenNil;
		pDeclaringTokenType->GetToken(&declaringTypeToken);
		PLog.Write(LogLevel::Info, "Declaring Type Token: 0x%x\n", declaringTypeToken);

		mdToken itemTypeToken = mdTokenNil;
		CComPtr<ITokenType> pItemTokenType;
		pType->QueryInterface(__uuidof(ITokenType), (LPVOID*)&pItemTokenType);

		if (pItemTokenType != nullptr)
		{
			pItemTokenType->GetToken(&itemTypeToken);
			PLog.Write(LogLevel::Info, "ItemTypeToken: 0x%x\n", itemTypeToken);
		}
		else
		{
			PLog.Write(LogLevel::Info, "ItemTypeToken is null\n");
		}

As you can see, I ultimately want to get a type token for the return value type passed to my getTypeString method. What I get is that pItemTokenType is a null pointer. Is there something I am missing that's available to an InstrumentationMethod to get the generic type information? Do I need to get an IMetaDataImport interface pointer (yes, I know I can get one easily) to retrieve this info, or is there a simpler way?

Thanks!

Azure App Service preinstalled site extension allows for automatic updates to the latest version using the tilda '~' semantic. This behavior is desirable as it allows for automatic security bugfixes and maintainability.

For the Windows distribution model using MSI, having similar behavior helps different profiler products install different versions of the CLRIE while guaranteeing the latest versions will be used. This also decouples updates to CLRIE from any specific profiler as users can install the latest engine themselves.

Have the CLR Instrumentation Engine parse the data during the attach process in order to configure instrumentation method settings (e.g. from where to load configurations, key-value pairs to pass to method instance, etc) for attach.

This type of logging should be granular to the InstrumentationMethod level, allowing easier triaging of conflicts. Consider introducing a new logging flag for this.

Create a JSON schema that allows for passing engine and instrumentation method configuration information to the CLR Instrumentation Engine during profiler attach.

This is configured by the Azure team, but we can keep the configuration files provided to them in this repo.

I found this while debugging my instrumentation method. I was trying to match the name of an async state machine method, and it seems that ::GetFullName() is just returning the typename of the inner type. It is not encapsulating it in the outer type and namespace. The way that ILDASM does this is:

<namespace>.<TypeName>+<InnerType>.MethodName

Background

I've worked on .NET profilers before, and in fact have one that works (with the ICorProfilerCallback interfaces). This is my first stab at adding my profiler as an InstrumentationMethod.

Configuration setup

I'm trying to use the engine in an IIS-based web app, so I've setup the environment variables in the
I've setup the following environment variables in HKLM\SYSTEM\CurrentControlSet\Services\W3WC and ...\Services\WAS. This is just how I would do it with a raw profiler hook. Environment looks like this:

COR_PROFILER={324F817A-7420-4E6D-B3C1-143FBED6D855}
COR_ENABLE_PROFILING=1
COR_PROFILER_PATH_64=c:\path-to\MicrosoftInstrumentationEngine_x64.dll
MicrosoftInstrumentationEngine_Host={CA487940-57D2-10BF-11B2-A3AD5A13CBC0}
MicrosoftInstrumentationEngine_HostPath_64=C:\path-to\Microsoft.InstrumentationEngine.ExtensionsHost_x64.dll
MicrosoftInstrumentationEngine_LogLevel=Errors
MicrosoftInstrumentationEngine_ConfigPath64_ShiftLeft=C:\path-to\MyInstrumentation.config

MyInstrumentation.config:

<InstrumentationEngineConfiguration>
  <InstrumentationMethod>
    <Name>MyInstrumentation</Name>
    <Description>My Instrumentation for Monitoring</Description>
    <Module>Profiler.dll</Module>
    <ClassGuid>{8539E781-D72A-42FE-85A6-E27D6162F5B5}</ClassGuid>
    <Priority>50</Priority>
  </InstrumentationMethod>
</InstrumentationEngineConfiguration>

InstrumentationMethod

I've got a class that implements IInstrumentationMethod.Initialize and IInstrumentationMethod.ShouldInstrumentMethod.

Behavior

When I make requests against the web application (it's a simple .NET Framework-based WebApi), I noticed that MicrosoftInstrumentationEngine_x64.dll is loaded into the w3wp.exe process, but not my Profiler.dll. I'm not sure why it's not picking up my InstrumentationMethod from the config file.

This code unloads dynamically-loaded instrumentation methods:

https://github.com/Microsoft/CLRInstrumentationEngine/blob/0807cc84afec64f95b7fc5350331f1f8c7759efc/src/Extensions.BaseLib/HostExtensionBase.cpp#L328-L339

It seems that references on those instrumentation methods are leaking as destructor is not being called on dynamically loaded instrumentation method

Create new instrumentation method interfaces that allow instrumentation methods to participate in attaching the CLR Instrumentation Engine to a .NET CLR instance. These interfaces should allow for the initialization of the instrumentation method, passing configuration information during attach initialization, and notifying of attach completion.

Sometimes finding/reading the IDL file can be difficult. It may be easier to list all of the interfaces in a docs markdown (user-friendly)

The current Instrumentation Engine supports initializing for the first CLR instance in a process. All subsequent CLR instances that are loaded into the process are not profiled by the Instrumentation Engine. While the vast majority of managed processes will only ever have a single CLR instance, there are scenarios where there are multiple CLRs within a process.

For example, an IIS worker process that has managed IIS modules and an in-process ASP.NET Core 2.2+ application will have two CLRs: the .NET Framework CLR and the .NET Core 2.2+ CLR. In this example, the Instrumentation Engine typically attaches to the .NET Framework CLR instance and rejects profiling of the .NET Core CLR instance. Typically, it is more interesting to profile the .NET Core CLR instance rather than or in addition to the .NET Framework CLR instance.

In order to support multiple CLR instances in the same process, the Instrumentation Engine will be changed as follows:

• The Engine will create a new instance of CProfilerManager for each CLR instance. Each of these instances are completely separate so that there is no intermingling of CLR information across the instances.
• Each available Instrumentation Method (IM) will have a new instance of its IInstrumentationMethod implementation created and initialized per CLR instance. Thus, each IM instance will only be associated a single IProfilerManager instance (which is associated with a single CLR instance). It will be left to the IM instance to determine if it wants to participate in profiling the associated CLR instance.

https://docs.microsoft.com/en-us/cpp/standard-library/filesystem?view=vs-2019

We are getting reports that Instrumentation Engine shows up on hot path in Application Insights Service Profiler. Like below:

@juliusl investigated an issue and believe this is caused by global lock Instrumentation Engine sets on JIT Compilation and other callbacks.

Code that loads clr.dll (in extension base component of ApplicationInsights)

if (bstrModuleName == L"mscorlib.dll")
    {
        std::wstring strModulePath;
        IfFailRet(GetModulePath(strModulePath));

        std::shared_ptr<CModuleMetaDataLoader> spLoader;
        IfFailRet(CreateAndBuildUp(spLoader, strModulePath + ExtensionsBaseOriginalModuleName));

HRESULT CModuleMetaDataLoader::Load()
{
    HRESULT hr = S_OK;

    ATL::CComPtr<IMetaDataDispenserEx> pDisp;

    ATL::CComPtr<ICLRMetaHost> pMetaHost;
    ATL::CComPtr<ICLRRuntimeInfo> pRuntime;
    IfFailRet(CLRCreateInstance(CLSID_CLRMetaHost, IID_ICLRMetaHost, reinterpret_cast<void **>(&pMetaHost)));
    IfFailRet(pMetaHost->GetRuntime(L"v4.0.30319", IID_ICLRRuntimeInfo, reinterpret_cast<void **>(&pRuntime)));
    IfFailRet(pRuntime->GetInterface(CLSID_CorMetaDataDispenser, IID_IMetaDataDispenserEx, reinterpret_cast<void **>(&pDisp)));

The highlighted line causes the following dll’s to load, mentioned earlier in the thread:

clr.dll     clr.dll     C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll            N/A        N/A        Symbols loaded.                C:\Users\wiktork\AppData\Local\Temp\SymbolCache\clr.pdb\8354a7bf1cd24de5827c9128420bda702\clr.pdb                25           4.7.3190.0 built by: NET472REL1LAST_C 8/6/2018 11:16 PM         742B0000-7499F000        [1616] ConsoleApplication9.exe                              
msvcr120_clr0400.dll      msvcr120_clr0400.dll      C:\Windows\SysWOW64\msvcr120_clr0400.dll  N/A        N/A                Symbols loaded. C:\Users\wiktork\AppData\Local\Temp\SymbolCache\msvcr120_clr0400.i386.pdb\915fa55e7772460c947f83560680967a1\msvcr120_clr0400.i386.pdb                27           12.00.52519.0 built by: VSWINSERVICING             12/8/2016 4:21 PM         741B0000-742A5000                [1616] ConsoleApplication9.exe                

Currently, the LogLevel Message causes extremely high overhead (to the point of uselessness for anything except development time, and even then it's quite large).

In addition, the current logging infrastructure could be refactored/restructured to delay dynamic allocations of strings for prefixes such as time from the InstrumentationMethod to some public interface supporting logging with VarArgs by the ProfilerManager. This would reduce memory size taken by CLRIE (and any snapshots generated) by reducing the memory page changes due to logging. This latter discussion could be moved to another Issue, but I want to jot it down somewhere for now.

Profiler client processes know the applications/agents that do not require profiling. It would help to provide a declarative model that avoids CLRIE accidentally picking those processes up.

Current implementation only supports one variable which doesn't make sense as we aren't guaranteed the bitness of the process. Solution would be to separate into two variables (_32 & _64) and have the non-bitness variable as a fallback.

In the introductory pages: Getting Started, Design Notes, and any other ancillary docs there is no mention whatsoever of the CreateBaseline API. This API is mentioned in passing here.
It is extremely important to make implementors aware of it early on in the intro pages, otherwise they may be flying blind, making it difficult to estimate time and effort required.

This seems to be an error - while copying SEH clauses - tokens are only converted and copied for filters, not for regular catch statements:

https://github.com/Microsoft/CLRInstrumentationEngine/blob/99c628e15018fcd445c5e101873818e4279a216f/src/InstrumentationEngine/AssemblyInjector.cpp#L713-L720

This causes a failure when you trying to inject an assembly with try{}catch statement into another assembly.

Consider the following folder structure which contains the malformed version string 1.99

snikolov@DESKTOP-5UO7643:/mnt/c/Program Files/Microsoft CLR Instrumentation Engine$ find
. -type f

./1.99/Instrumentation32/MicrosoftInstrumentationEngine_x86.dll
./1.99/Instrumentation64/MicrosoftInstrumentationEngine_x64.dll
./Proxy/v1/InstrumentationEngine.ProfilerProxy_x64.dll
./Proxy/v1/InstrumentationEngine.ProfilerProxy_x86.dll

Observed behavior - on activation the profiler proxy becomes permanently stuck.
Expected behavior - the profiler proxy shall exit with an error code and print a message either on stdout/stderr or the file pointed to by MicrosoftInstrumentationEngine_FileLogPath.

Rename folder 1.99 to something more sensible like 1.0.28 and observe that now the profiler proxy works as designed.

during build, there's currently no checks for if packages are missing files.

Create downloadable links to released MSI/MSM files once available.

Out parameters for basic non-interface and non-VARIANT types (e.g., [out, optional] ULONG*) are incorrect. Midl does not generate SAL for optional basic types of out paramters. I believe the correct annotation is [in, out, ptr] for these types, but I will have to double check.

Current implementation of the Linux ExtensionsHost (which deals with loading Instrumentation Methods) specifically loads Snapshot Debugger's method "Production Breakpoints". This needs to be generalized to have functional parity with its Windows counterpart.

CGenericInstance can provide the related parent type, but does not expose any generic parameters. These are stored in std::vector<CComPtr<IType>> m_genericParameters

After re-writing a method using the InstrumentMethod callback, it should be possible to get the entire method's IL bytes as an array in the OnInstrumentationComplete callback for validation and debugging purposes.

ClrInstrumentationEngine allows you to debug your instrumentation method by setting an environment variable to wait for a debugger attach. This is not practical for customers, though, because the ProfilerManager.cpp sets up a wait time that is only overridden for debugging when the dll is compiled for debugging.

Though clients can attempt to locate the correct release tag, and rebuild with debug if they are on Windows, I expect most clients would rather just use retail dlls, and debug their own code.

I suggest that we remove the #ifdef Debug for the override of the wait time.

In order to support a cooperative model for our partners, we want to provide an MSI/MSM installer which defines the common installation behavior for everyone to use on Windows Desktop scenarios.

CLRIE currently depends on VS 2017 toolsets and builds. VS 2019 was released on April 2nd, 2019 and we are internally using it for daily development.

Unless there's strong pushback to continue developing using VS 2017, we will migrate this project to build with the latest tools provided by VS 2019.

Environment

Azure App Service

Scenario

I have a .NET profiler installed in the App Service, and have InstrumentationEngine_EXTENSION_VERSION set to ~1 in the Application settings, via the portal:

I enabled Application Insights via the portal (Deployment -> Application Insights). When I go back to look at the Application settings, InstrumentationEngine_EXTENSION_VERSION is reset to disabled.

I noticed that it gets reset to disabled for any Application Insights feature I enable, and even when I disabled Application Insights, it reset it!

This is clearly a bug or at least not acceptable behavior because, with CLRIE, it's possible to have multiple profilers running. My workaround has been to manually check the InstrumentationEngine_EXTENSION_VERSION application setting and set it when necessary.

Reproducing this bug should be easy given the explanation above.

Although this is not necessarily a CLR Instrumentation Engine bug, I was advised that reporting it here was one way to get some movement on fixing it.

Providing Instrumentation Engine as a NuGet package would allow profilers to deploy with the app.

This model however would be difficult to cooperate on (if two profilers require different CLRIE versions, the user must resolve this). Consider providing guidance/documentation on how to use CLRIE NuGet in a cooperative manner.

Followed the build instructions and successfully generated all build artifacts.
Also ensured that all tests pass, namely 37 of 37, 42/42, 59/59 all pass.

In section Environment Variables, in the first occurrence of COR_PROFILER I see a bracketed UUID which implies that successful DLL registration is necessary, especially for framework versions < 4.

However, when I attempt DLL registration as shown below from an admin prompt, it fails with error 0x80070715, i.e. "The specified resource type cannot be found in the image file."

C:\Windows\SysWOW64\regsvr32.exe /i C:\tmp\CLRInstrumentationEngine\bin\Debug\x64\InstrumentationEngine\x64\MicrosoftInstrumentationEngine_x64.dll

The above DLL is missing typical COM sections like REGISTRY, TYPELIB, String Table.
See the attached image.

Build tool version: vs2019 Professional 16.3.5
Host OS: Windows 10, 1903, build 18362.418

What am I missing here?
See also a forward link to my previous message.

Create an automated testing for build/setup/E2E that allows testing CLRIE changes.

Use CComPtr.Attach(new (nothrow) TYPE) instead. This ensures that if the system is out of memory, the pointer is set to NULL rather than an exception being thrown.

Roadmap will be updated with some timeline cadence (quarters?) for targeted releases. This will help partners track expectations.

Have the CLR Instrumentation Engine parse the data during the attach process in order to configure engine settings for attach.

FileLoggerSink.cpp uses _fwopen/_fwsopen to create the file. This API will fail if the directory path does not exist.

Consider adding CreateIfNotExist directory check before file open. This has an added benefit of keeping the behavior consistent with our ProductionBreakpoints InstrumentationMethod.

Suggested from #111,
Currently we wait 1 minute for configs to load which is sufficient for a small number of configs. When more configs exist, this timeout may not be sufficient and we should expose configuration hooks to allow customers to get unblocked.

Information at https://aka.ms/onboard-azure-pipelines (may be internal-only)

https://dev.azure.com/ms is an Azure DevOps project for the Microsoft Org's OSS projects on GitHub (dev.azure.com/azure is for Azure OSS).

Since CLRIE build pipelines are currently internal-facing, migrating the builds to this public-facing project would allow external users to make build changes to CLRIE and see build/test results, logs, etc.

This task is necessary for root-causing and routing errors to affected profilers in the case where multiple profilers conflict with each other.

The link to the CLRIE Roadmap in the Overview section of the repository's README.md goes to a 404. Knowing the roadmap for this project would be helpful for vendors and users of the instrumentation engine.

In order for users and partners to write InstrumentationMethods, they would need to include InstrumentationEngine.h header file. This file is available in-repo, but to ease consumption of CLRIE it should be made available as a NuGet package in a public feed.

A change to the .NET Core CLR has been made such that the deconstructor of attached profilers will no longer be called: https://github.com/dotnet/coreclr/issues/15136#issuecomment-534325458

The advice given is that all shutdown logic must be contained within ICorProfilerCallback::Shutdown since the CLR will never release the COM object. The current Instrumentation Engine does some minimal work in CProfilerManager::~CProfilerManager, which needs to be done elsewhere (e.g. ICorProfilerCallback::Shutdown).

Overview

The CLR Instrumentation Engine's main purpose is to provide cooperation among profilers running in the same environment. This requires various users to cooperatively ensure that the latest version of the Engine can be detected and loaded into the process.

To achieve this goal, we are asking each partner team to participate and provide their needs and requirements for their scenarios. Once we have enough information, we can discuss distribution solutions that can hopefully meet everyone's needs.

Distribution Steps

The distribution story for the Engine can be described through a series of steps.

1. Detect if the Instrumentation Engine is installed.
2. Detect if the Instrumentation Engine is configured.
3. Install the Instrumentation Engine.
4. Configure/activate the Instrumentation Engine.

Requirements

These are the currently identified requirements we expect for the Engine distribution. Please let us know if you have feedback or conflicts with these requirements and reply with your responses to these questions about your product.

• Support single well-known installation folder to easily detect if installed.
• Support updating the Engine via side-by-side versions to avoid file locks on older versions.
• Support determining latest version so profiler clients can detect if it meets their minimum requirements. The Engine is required to be forward compatible
• Setting Environment Variables in a consistent and well-known manner.
• Support ASP.NET Desktop apps on Windows and ASP.NET Core apps on both Windows and Linux.
• Don't require clean-up of older versions. The feasibility of this is dependent on the installation technologies and techniques.
• Have a validation mechanism against the installation

Questions

• What are the installation technologies you are currently using (eg. MSI, Exe bundles, xcopy, scripts)? Please differentiate between different platforms.
• What are the minimum requirements on your target platform?
  • Windows versions & skus
  • Linux distros & versions
  • Framework requirements (.Net Framework/Core versions)
  • Supported application types (ASP.NET on IIS, WPF, etc.)
• Is installation expected or allowed to run with admin privileges (ie. be able to install into %programfiles%)? Do you require installing to a non-admin accessible folder?

Possible Solutions

There may be multiple solutions, but the two options we have considered internally are:

1. MSI and/or a MergedModule for Windows platform

Each partner team has mentioned using an MSI for Windows to install their product. MSI has the benefit of supporting the above requirements.

If we go this route, there are several implementation details to consider:

• Share at the component level or chain in our MSI
• Custom actions for detection
• Per-user versus per-machine installation
• 1 MSI per version

2. Installation Scripts

There are several options for scripting languages, including Python, Bash, and PowerShell which allow for cross platform support and customizable control over installation. This also allows for use of various packaging frameworks such as NuGet or distro-specific options like apk, deb files. Several concerns about this option:

• Ability for MSIs or existing distribution technologies to incorporate this into their installation.
• Introducing new requirements (eg. dotnet core 2.1 for PS Core)

ICorProfiler attach has existed for a long time but it's only recently in dotnet Core 3 that module metadata emit and ReJit have been provided.

https://github.com/dotnet/coreclr/blob/master/Documentation/Profiling/Profiler%20Attach%20on%20CoreCLR.md

https://github.com/dotnet/coreclr/blob/master/Documentation/Profiling/ReJIT%20on%20Attach.md

This PR removes the notion of a customizable ExtensionsHost in favor of the default host (which has been used for years). This reduces the attack surface of the Engine, reduces contention points for collaboration over different hosts, and simplifies the onboarding process for the Engine.