From @sebastienros on April 20, 2018 0:3

In a microsoft/dotnet:2.0-sdk-jessie image, running perfcollect install errors with an unsupported linux distribution. uname returns Linux which isn't handled.

I tried to install manually based on Debian instructions, however after installing zip and linux-tools I still get a

Perf not installed.
Install perf to proceed.

Copied from original issue: dotnet/coreclr#17686

In a system without lttng ./perfcollect collect sampleTrace -nolttng errors out with "LTTng not installed. Install LTTng to proceed."

This is because in the script EnsurePrereqsInstalled comes before ProcessArguments has a chance to set useLTTng=0

workaround: sed -ie 's/useLTTng=1/useLTTng=0/' perfcollect

StackParser crashes when using /in and /out parameter because the StreamReader/StreamWriter instances "sr" and "sw" are closed too early.

Example: Failed to open mscorlib.ni.dll, continuing without symbols

This will never work because native images aren't ELF. We should filter these out.

From dotnet/cli#3223 it appears that crossgen now requires the -JITPath parameter.

@gkhanna79, can you provide details on this change and what is needed? Perfcollect calls crossgen to generate perfmap files for tracing, and it sounds like it will be broken by this change.

Are there any plans to support perfcollect on Mac OS? Attempting to install via ./perfcollect install responds with

ERROR: Auto install unsupported for this distribution. Install perf manually to continue.
but there are no instructions to be found on how to install manually.

Attempting to run ./perfcollect collect myAppCollect before performing the install results in

Perf not installed.
Install perf to proceed.

Async APIs are usually used with mix of await keyword, so that should be default behavior without await keyword and if someone has to consume async API without wait, a new keyword like "nowait" shall serve the purpose. This will help people write async APIs like simple synchronous APIs yet they will be doing non-blocking calls.

When running :
sudo ./perfcollect install it should have install perf (as shell script in /usr/bin and its dependent executable)

but after installation , when running perf command ,we get issue like :

/usr/bin/perf: line 13: exec: perf_4.4: not found

this might possibly due to reason that uname -r for K8 container is : 4.4.0-139-generic
while for docker it is : 4.9.125-linuxkit

Moreover , if we use this perf_4.9 as perf , then we receive segmentation fault error.

Hallo Community,
Maybe im here right for following problem.
Im facing following problems building the build-managed.sh for an corefx x86 build:

/home/arbeit/Tools/depProj.targets(83,5): error : Error no assets were resolved from NuGet packages. [/home/arbeit/corefx/external/netstandard/netstandard.depproj]
/home/arbeit/Tools/depProj.targets(83,5): error : Error no assets were resolved from NuGet packages. [/home/arbeit/corefx/external/docs/docs.depproj]
/home/arbeit/Tools/depProj.targets(83,5): error : Error no assets were resolved from NuGet packages. [/home/arbeit/corefx/external/ILLink/ILLink.depproj]

I was following these steps:

git clone https://github.com/dotnet/corefx.git
cd corefx
sudo ./cross/build-rootfs.sh x86

./build-native.sh -debug -- clang3.8
./build-native.sh -debug -buildArch=x86 -- cross clang3.8
./build-managed.sh -BuildTests=false

The build-native.sh builds working fine, but the managed build not. I also tried -release instead of -debug.

I found these issue with a workaround but i have no
src/Tools/CoreFx.Tools/CoreFx.Tools.csproj.

Can someone help me with that problems?

Greetings

Philipp.

We'd like to have this option in order to get accurate GC times.

Originally reported in https://github.com/dotnet/coreclr/issues/18452.

From @paraspatidar:

FYI , apart from kubernetes , same behavior is observed on Ubuntu VM on azure.
However on Ubuntu it happens only when we pass -pid in perfcollect as argument to specifically target perticular dotnet process.
for normal run where it collect for all CPUs (-a) , then it is successfully able to get CPU traces.
(however on kubernetes we have no other option but to use -pid as -a failed due to access issue)
when running that command manually :
/usr/bin/perf script -i perf.data.merged -f comm,pid,tid,cpu,time,event,ip,sym,dso,trace > perf.data.txt :
output : Samples for 'cpu-clock' event do not have CPU attribute set
OS :
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
uname -r : 4.18.0-1011-azure

Currently it is hard-coded 999.

@brianrob

While perfcollect collect <tracename> goes to the effort to run crossgen /CreatePerfMap <dll>, the perfcollect view <tracename> fails to use the perfmap to resolve the NGEN symbols.

It seems to me that the PerfMap is a format which is unsupported by perf for mmap-ped files.

The simplest solution might be creating DWARF debug info for the NGEN image.

There may actually be code in coreclr to already support writing this. GDB debug support in CoreCLR may actually be writing a ELF debug info to memory. Not certain, but that is what I remember from enabling GDB debug on Arm64.

Other options/hacks.

• Use the assembler to generate elf with debug info ?
• Use binutils objcopy to generate elf from symbolsrec format ?
• ZapDisable=1

From @lbthanh on September 5, 2018 5:34

perfcollect still not support to Amazon Linux distribution.

https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md

I work around with RHEL repo but dependency didn't work.

$ wget -P /etc/yum.repos.d/ https://packages.efficios.com/repo.files/EfficiOS-RHEL7-x86-64.repo

$ sudo yum --skip-broken install lttng-tools lttng-ust kmod-lttng-modules

Bellow error messages:
Packages skipped because of dependency problems:
kernel-4.9.85-47.59.amzn2.x86_64 from amzn2-core
kmod-lttng-modules-2.10.6-1.el7.x86_64 from EfficiOS-RHEL7-x86-64
lttng-tools-2.10.4-1.el7.x86_64 from EfficiOS-RHEL7-x86-64
lttng-ust-2.10.1-1.el7.x86_64 from EfficiOS-RHEL7-x86-64

Copied from original issue: dotnet/coreclr#19832

Docker Image: mcr.microsoft.com/dotnet/core-nightly/sdk:3.0.100-preview5

Preview5 moved from Debian Stretch to Debian Buster and linux-tools is not available in the stable feed (yet?) We can work around this by installing linux-perf, but will also need to make sure that we point at the right version (it is looking for 4.9 but installing 4.19).

@brianrob - I ran perfcollect install for the first time yesterday, and it is a rather lengthy install process that does some potentially dangerous things like adding additional feeds to your apt-get sources.

It seems that at a minimum we should give some warning to the user when we suggest running install that it is a lengthy install process with a lot of dependencies. It would also be good to mention that they can either install it through our nice automated process or point them at a list of what needs installed so they can do it manually if they are not trusting of external scripts.

This is the output that gets printed after stopping a capture. I was running an Hello World console app in a microsoft/dotnet:2.0.6-sdk-2.1.104 container.

...STOPPED.

Starting post-processing. This may take some time.

Generating native image symbol files
...FINISHED
Saving native symbols
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
/perf/perfcollect: line 1468: objdump: command not found
...FINISHED
Exporting perf.data file
...FINISHED
Compressing trace files
...FINISHED
Cleaning up artifacts
...FINISHED

Trace saved to session7.trace.zip

Although ./perfcollect install has been run, it seems that it didn't install the package required to run objdump.

Hi,
I am running dotnet core (2.2.2) process on K8 pod with elevated rights and could do a perf collect without any issue and trace is created as well as CPU stack (perf.data.txt) is created.

However , when coping to windows and opening the CPU stack , it fails with exception :

System.FormatException: Input string was not in a correct format

this happens when we do
`sudo ./perfcollect collect sample'

however if we pass on the process id (-pid param) , then CPU traces doesnt gather at all as mentioned in another issue :
#84
&

https://github.com/dotnet/coreclr/issues/18452

this format issue was said to be fixed in microsoft/perfview#166
but i am using perfview 2.0.26 and facing this issue.
here is sample trace :
https://drive.google.com/file/d/1eyQWCIAefVyCCp0zm5vCSr0f0Jp99EGC/view?usp=sharing

Running docker build with Dockefile produces the following error:

Step 7/10 : RUN apt-get -y install linux-tools-common linux-tools-`uname -r` linux-cloud-tools-`uname -r` lttng-tools lttng-modules-dkms liblttng-ust0 zip
 ---> Running in bcc38dd5d269
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package linux-tools-4.9.49-moby
E: Couldn't find any package by regex 'linux-tools-4.9.49-moby'
E: Unable to locate package linux-cloud-tools-4.9.49-moby
E: Couldn't find any package by regex 'linux-cloud-tools-4.9.49-moby'
The command '/bin/sh -c apt-get -y install linux-tools-common linux-tools-`uname -r` linux-cloud-tools-`uname -r` lttng-tools lttng-modules-dkms liblttng-ust0 zip' returned a non-zero code: 100

After the last update https://github.com/dotnet/corefx-tools/blob/master/src/performance/perfcollect/perfcollect contains syntax error at line 1275.

./perfcollect: line 1276: syntax error near unexpected token `then'
./perfcollect: line 1276: `    then'

elseif should be elif

I ran perfcollect from ubuntu VM for asp.net core 2.1 application by specifying pid of the dotnet process. The perf trace file was generated successfully and I was able to view the data on ubuntu with ./perfcollect view session4.trace.zip command.
However, when I copied the trace file on windows and opened it in Perfview, it could not load the CPU Stacks.
PFB snapshot for the same.

The events data is visible though.
I checked the underlying 'perf.data.txt' file and found it empty, but the 'perf.data' file is not empty.
Can you please help here to see what's going wrong?

Hello,

There was a PR in coreclr which changed the public API of crossgen tool (dotnet/coreclr#23462). Instead of /Trusted_Platform_Assemblies path[:path...] it now uses /r path [/r path...]. We need to update how perfcollect uses crossgen to respect the new key.

WIP is here: https://github.com/criteo-forks/corefx-tools/tree/crossgen_fix . For now I did a simple conversion. However I thought that maybe we should check crossgen version in perfcollect to support both old and new formats.

What do you think? Thanks!

namely the following:

DotNETRuntime:GCJoin_V2
DotNETRuntime:PinObjectAtGCTime

These can artificially increase the GC pause by potentially a lot (especially the pinobject one)

there are other verbose level events like

DotNETRuntime:FinalizeObject
DotNETRuntime:IncreaseMemoryPressure
DotNETRuntime:DecreaseMemoryPressure

but they are outside of GC and are ok to leave in.

Also I think it might make sense to have a separate option that includes the GCHeapSurvivalAndMovementKeyword events as they can artificially increase the GC pause by really a lot (as in, it can make GC 2x+ as long). It would be nice we have GCCollectOnly and GCOnly (that collects alloctick + verbose events like join/pinobject) and GCWithHeap (that collects GCOnly + GCHeapSurvivalAndMovementKeyword).

This class does cleanup in it's finalizer which throws an exception. Because this is on the finalizer it cannot be handled and tears down the stress mix. These tests need to be identified and excluded from generated stress mixes.

libc.so.6!gsignal
libc.so.6!abort
libcoreclr.so!PROCEndProcess
libcoreclr.so!UnwindManagedExceptionPass1
libcoreclr.so!DispatchManagedException
libcoreclr.so!IL_Throw
System.IO.FileSystem!System.IO.DirectoryInfo..ctor
System.IO.FileSystem.Tests!FailSafeDirectoryOperations.DeleteDirectory
System.IO.FileSystem.Tests!ManageFileSystem.Dispose
System.IO.FileSystem.Tests!ManageFileSystem.Finalize
libcoreclr.so!FastCallFinalizeWorker
libcoreclr.so!MethodTable::CallFinalizer
libcoreclr.so!MethodTable::CallFinalizer
libcoreclr.so!FinalizerThread::DoOneFinalization
libcoreclr.so!FinalizerThread::DoOneFinalization
libcoreclr.so!FinalizerThread::FinalizeAllObjects
libcoreclr.so!FinalizerThread::FinalizerThreadWorker
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase::FinalizerBase
libcoreclr.so!ManagedThreadBase::FinalizerBase
libcoreclr.so!FinalizerThread::FinalizerThreadStart
libcoreclr.so!CorUnix::CPalThread::ThreadEntry
libpthread.so.0!start_thread
libc.so.6!clone

Some configurations result in the CPU attribute not being set on cpu-clock events. We get an explicit failure on calls to 'perf script' when this happens because we ask for the CPU number.

Consider removing this or looking for a way to optionally get the CPU number.

I'm trying to run perfcollect in a docker container following these instructions. https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md

When I run
./perfcollect collect sampletrace
It immediately terminates with the following output.

Collection started. Press CTRL+C to stop.

...STOPPED.

Starting post-processing. This may take some time.

zero-sized file (perf.data), nothing to do!
Generating native image symbol files
zero-sized file (perf.data), nothing to do!
libcoreclr.so not found in perf data. Please verify that your .NET Core process is running and consuming CPU.
Saving native symbols
zero-sized file (perf.data), nothing to do!
...FINISHED
Exporting perf.data file
...FINISHED
Compressing trace files
...FINISHED
Cleaning up artifacts
...FINISHED

Trace saved to sampleTrace.trace.zip

I've verified that the dotnet process is running and consuming 100% CPU. The image I'm trying to run this in is microsoft/aspnetcore:2.0.8-stretch

I can't figure out why this won't run. Does anyone have any suggestions?

Right now, all events are enabled. We should reduce this set to just what is useful for a general investigation.

I'm following the instructions here: https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md

I ran ./perfcollect collect trace and am seeing the error: ./perfcollect: line 641: 363 Segmentation fault (core dumped) $* >> $logFile 2>&1

It looks like the seg fault is happening on this command: /usr/bin/perf_4.9 record -g -a -F 999 -e cpu-clock

Any suggestions on how to resolve this?

EventSource data isn't yet usable by the viewer other than by looking at raw event data. Since EventSources can be quite verbose, disable them by default but allow them to be collected with a command line switch.

There still appear to be some tests using ThreadHelpers which are not excluded from our stress mixes. These remaining tests need to be identified and excluded to stop polluting stress run results. Below is an example failure callstack:

libc.so.6!gsignal
libc.so.6!abort
libcoreclr.so!PROCEndProcess
libcoreclr.so!UnwindManagedExceptionPass1
libcoreclr.so!DispatchManagedException
libcoreclr.so!PreStubWorker
libcoreclr.so!ThePreStub
ThreadJoin1!Interop+mincore.CloseHandle
ThreadJoin1!System.Threading.Thread.Finalize
libcoreclr.so!FastCallFinalizeWorker
libcoreclr.so!MethodTable::CallFinalizer
libcoreclr.so!MethodTable::CallFinalizer
libcoreclr.so!FinalizerThread::DoOneFinalization
libcoreclr.so!FinalizerThread::DoOneFinalization
libcoreclr.so!FinalizerThread::FinalizeAllObjects
libcoreclr.so!FinalizerThread::FinalizerThreadWorker
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase_DispatchOuter
libcoreclr.so!ManagedThreadBase::FinalizerBase
libcoreclr.so!ManagedThreadBase::FinalizerBase
libcoreclr.so!FinalizerThread::FinalizerThreadStart
libcoreclr.so!CorUnix::CPalThread::ThreadEntry
libpthread.so.0!start_thread
libc.so.6!clone

This section of code copy all .map files from home directory ~/ to /tmp folder.

	local mapFiles=`find -name *.map`
	for mapFile in $mapFiles
	do
		echo "Copying $mapFile to /tmp."
		cp $mapFile /tmp
	done

For example all matched files from node_modules...

./.../do.js.map
./.../last.js.map
./.../switchMapTo.js.map
./.../takeUntil.js.map
./.../buffer.js.map

CoreCLR now generates separate .dbg symbol files. Perfcollect needs to grab these files when zipping up a trace, otherwise, native symbols cannot be resolved. Ideally, for every ELF binary, we look for a pointer to a debug file if the binary doesn't contain symbols, and then copy that debug file into the trace.

I used ./perfcollect install to install perf tools on Ubuntu 14.04, and whatever version it pulled down (I tried to find the version couldn't) uses -F instead of -f for fields now. I see that in perfcollect this is special cased for RHEL, but now the update has propagated through to what is running on ubuntu.

From @jeremymeng on June 5, 2018 22:35

Currently users have to manually download and patch their Linux system before running perfcollect script. It involves multiple steps as described at https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md#resolving-framework-symbols.

It would be great if a matching crossgen can be downloaded automatically before attempting to run crossgen.

Copied from original issue: dotnet/coreclr#18314

PerfCollect docs say that as long as a developer execs into a Docker container with --privileged, perfcollect should work, even if the container wasn't itself started with --privileged.

Instead, collection immediately stops and I get this error:

root@2ffc1d52bc5f:/app/perfcollect# ./perfcollect collect test
Collection started. Press CTRL+C to stop.

...STOPPED.

Starting post-processing. This may take some time.

zero-sized file (perf.data), nothing to do!
Generating native image symbol files
zero-sized file (perf.data), nothing to do!
libcoreclr.so not found in perf data. Please verify that your .NET Core process is running and consuming CPU.
Saving native symbols
zero-sized file (perf.data), nothing to do!
...FINISHED
Exporting perf.data file
...FINISHED
Compressing trace files
...FINISHED
Cleaning up artifacts
...FINISHED

If the container to be profiled is started initially (at docker run time) with --privileged and --pid=host (as if I were going to be attaching LLDB), then perfcollect collection works as expected.

If perfcollect can be made to work with non-privileged containers, that would be great. Otherwise, we should update the docs to explain the necessary steps for collecting traces in a Docker container.