google / perf_data_converter Goto Github PK

Tool to convert Linux perf files to the profile.proto format used by pprof

License: BSD 3-Clause "New" or "Revised" License

C++ 97.97% C 0.03% Shell 0.16% Starlark 1.85%

perf_data_converter's Introduction

Introduction

The perf_to_profile binary can be used to turn a perf.data file, which is generated by the linux profiler, perf, into a profile.proto file which can be visualized using the tool pprof.

For details on pprof, see https://github.com/google/pprof

THIS IS NOT AN OFFICIAL GOOGLE PRODUCT

Prerequisites to build

Install dependencies

sudo apt-get -y install g++ git libelf-dev libcap-dev

At least g++-5 or clang-7

Compile and Test

To install all dependences and build the binary, run the following commands. These were tested on Ubuntu 14.04 LTS:

Compile

Install bazel by following the instructions here.

Install dependencies and build perf_to_profile using bazel

git clone https://github.com/google/perf_data_converter.git
cd perf_data_converter
bazel build src:perf_to_profile

Place the perf_to_profile binary in a place accessible from your PATH (e.g. /usr/local/bin).

Running tests

There are a small number of tests that verify the basic functionality. To run these, after successful compilation, run:
```
bazel test src:all
bazel test src/quipper:all
```

Note: Executables generated using bazel build are available under the directory bazel-bin/.

Usage

Profile a command using perf, for example:
```
perf record /bin/ls
```
Recent versions of pprof will automatically invoke perf_to_profile:
```
pprof -web perf.data
```

Contribution

We appreciate your help!

Note that perf data converter and quipper projects do not use GitHub pull requests, and that we use the issue tracker for bug reports.

perf_data_converter's People

Contributors

Stargazers

Watchers

perf_data_converter's Issues

build error using bazel

when I build perf-data-converter, I have this error, how I can solve it?

[root@opengauss-master perf_data_converter]# bazel build src:perf_to_profile
Starting local Bazel server and connecting to it...
DEBUG: Rule 'com_google_protobuf' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "7372397031f356341e932c4b0489489e4111152fc3a9e03660ede951e9b2d84d"
DEBUG: Repository com_google_protobuf instantiated at:
/root/perf/perf_data_converter/WORKSPACE:30:13: in
Repository rule http_archive defined at:
/root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in
DEBUG: Rule 'rules_python' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "12b37c991026c4ce136509fd3aeb4a194e98a29c0f328ecac53269a8c0af2906"
DEBUG: Repository rules_python instantiated at:
/root/perf/perf_data_converter/WORKSPACE:20:13: in
Repository rule http_archive defined at:
/root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in
INFO: Repository boringssl instantiated at:
/root/perf/perf_data_converter/WORKSPACE:37:13: in
Repository rule http_archive defined at:
/root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in
INFO: Repository bazel_skylib instantiated at:
/root/perf/perf_data_converter/WORKSPACE:5:13: in
Repository rule http_archive defined at:
/root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in
ERROR: /root/perf/perf_data_converter/src/BUILD:170:11: error loading package '@com_google_protobuf//': at /root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/rules_python/python/defs.bzl:20:6: at /root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/rules_python/python/py_runtime.bzl:19:6: at /root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/rules_python/python/private/common/py_runtime_macro.bzl:16:6: at /root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/rules_python/python/private/common/py_runtime_rule.bzl:21:6: at /root/.cache/bazel/_bazel_root/4a25792f71ccee4f60d59a36fdf7bb13/external/rules_python/python/private/common/py_internal.bzl:23:6: Unable to find package for @rules_python_internal//:py_internal.bzl: The repository '@rules_python_internal' could not be resolved: Repository '@rules_python_internal' is not defined. and referenced by '//src:builder'
ERROR: Analysis of target '//src:perf_to_profile' failed; build aborted:
INFO: Elapsed time: 9.419s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (39 packages loaded, 206 targets configured)
currently loading: @com_google_protobuf//
Fetching https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz

perf_to_profile fails with large (18gb) perf data

I generated perf data with a clang linux build
$ perf record -e cycles:pp --output=arm64_clang_built_def_config.data --call-graph lbr -- make LLVM=1 -j72 ARCH=arm64

Which had a (quite large) size of:

$ du -h arm64_clang_built_def_config.data 
18G     arm64_clang_built_def_config.data

Invoking the perf_to_profile tool yields this output:

$ perf_to_profile -i arm64_clang_built_def_config.data -o profile_arm64_clang_built_defconfig
[ERROR:src/quipper/perf_reader.cc:197] perf_file_section.size 19150652112 cannot exceed remaining perf data input size 1970824676 at offset 824
[ERROR:src/quipper/perf_reader.cc:813] Error reading header file section info.
[ERROR:src/perf_data_converter.cc:749] Could not read input perf.data
[FATAL:src/perf_to_profile.cc:33] Expected profile vector to have one element.

For reference, that first number 19150652112 is ~19gb and the second 1970824676 is ~1.9gb.

Steps to reproduce:

run the perf_to_profile tool on a large perf.data. (I am not exactly sure how large of a file is required to reproduce, but I know for sure >=18gb does the trick)

edit: as per my comment (and fixes) below, the critical size is ~2.14gb.

should kernel addr-- ?

i find that you get addr from perf record.
and add addr with -1 to get right addr.

i find this could change the addr's kernel symbol.

give /proc/kallsyms:

[root@plat-sh-infra-testing-common-parca001 perfcheck]# cat /proc/kallsyms | head -n 34175 | tail
ffffffff92d83fe0 T tty_unlock_slave
ffffffff92d83fff T __lock_text_end
ffffffff92d84000 t perf_event_nmi_handler
ffffffff92d84000 T __kprobes_text_start
ffffffff92d84050 t perf_ibs_nmi_handler
ffffffff92d840c0 T save_paranoid
ffffffff92d84200 t common_interrupt
ffffffff92d8436a t ret_from_intr
ffffffff92d8437f t exit_intr
ffffffff92d8439d t retint_with_reschedule

the perf script would get ret_from_intr for addr ffffffff92d8436a:

[root@plat-sh-infra-testing-common-parca001 .parca-agent]# perf script | head -n 4240 | tail
        ffffffffc04961f5 virtnet_poll+0x265 ([kernel.kallsyms])
        ffffffff92c5057f net_rx_action+0x26f ([kernel.kallsyms])
        ffffffff926a5435 __do_softirq+0xf5 ([kernel.kallsyms])
        ffffffff92d9142c call_softirq+0x1c ([kernel.kallsyms])
        ffffffff9262f715 do_softirq+0x65 ([kernel.kallsyms])
        ffffffff926a57b5 irq_exit+0x105 ([kernel.kallsyms])
        ffffffff92d92886 __irqentry_text_start+0x56 ([kernel.kallsyms])
        ffffffff92d8436a ret_from_intr+0x0 ([kernel.kallsyms])
                  50c5d3 ZSTD_compressBlock_greedy_extDict_row+0x2d3 (/root/zstd/zstd)
        9f5573aeed0b19a2 [unknown] ([unknown])

but with addr--, we would get common_interrupt for addr ffffffff92d8436a.

so with addr--, our result would diff from perf script output.
which is correct?

Compiler warnings when building perf_data_converter

$ bazel build src:perf_to_profile
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: SHA256 (https://github.com/google/boringssl/archive/master-with-bazel.zip) = cec1a5180dc44d083573015694a5cc40a194920a1e7665dc005592a08d959597
DEBUG: Rule 'boringssl' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "cec1a5180dc44d083573015694a5cc40a194920a1e7665dc005592a08d959597"
DEBUG: Call stack for the definition of repository 'boringssl' which is a http_archive (rule definition at /usr/local/google/home/aalexand/.cache/bazel/_bazel_aalexand/f6d7d2335bdf63a7813e232f542cf0f2/external/bazel_tools/tools/build_defs/repo/http.bzl:262:16):
 - /usr/local/google/home/aalexand/src/perf_data_converter/WORKSPACE:27:1
INFO: Analyzed target //src:perf_to_profile (19 packages loaded, 1079 targets configured).
INFO: Found 1 target...
INFO: From Compiling src/quipper/perf_reader.cc:
src/quipper/perf_reader.cc: In member function 'bool quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, size_t)':
src/quipper/perf_reader.cc:1525:23: warning: comparison of integer expressions of different signedness: 'int' and 'uint32_t' {aka 'unsigned int'} [-Wsign-compare]
     for (int i = 0; i < nrcpus; ++i) {
                     ~~^~~~~~~~
INFO: From Compiling src/perf_data_converter.cc:
src/perf_data_converter.cc: In member function 'perftools::ProcessProfiles perftools::{anonymous}::PerfDataConverter::Profiles()':
src/perf_data_converter.cc:663:21: warning: comparison of integer expressions of different signedness: 'int' and 'std::deque<perftools::profiles::Builder>::size_type' {aka 'long unsigned int'} [-Wsign-compare]
   for (int i = 0; i < builders_.size(); i++) {
                   ~~^~~~~~~~~~~~~~~~~~
INFO: From Compiling src/quipper/perf_serializer.cc:
src/quipper/perf_serializer.cc: In member function 'bool quipper::PerfSerializer::DeserializeAuxtraceInfoEvent(const quipper::PerfDataProto_AuxtraceInfoEvent&, quipper::event_t*) const':
src/quipper/perf_serializer.cc:1006:21: warning: comparison of integer expressions of different signedness: 'quipper::u64' {aka 'long unsigned int'} and 'int' [-Wsign-compare]
   for (u64 i = 0; i < sample.unparsed_binary_blob_priv_data_size(); ++i) {
                   ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/quipper/perf_serializer.cc: In member function 'const quipper::SampleInfoReader* quipper::PerfSerializer::GetSampleInfoReaderForEvent(const event_t&) const':
src/quipper/perf_serializer.cc:1467:24: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'ssize_t' {aka 'long int'} [-Wsign-compare]
         if (array_size <= event_id_pos) {
             ~~~~~~~~~~~^~~~~~~~~~~~~~~
src/quipper/perf_serializer.cc:1482:24: warning: comparison of integer expressions of different signedness: 'size_t' {aka 'long unsigned int'} and 'ssize_t' {aka 'long int'} [-Wsign-compare]
         if (array_size < event_id_pos) {
             ~~~~~~~~~~~^~~~~~~~~~~~~~
Target //src:perf_to_profile up-to-date:
  bazel-bin/src/perf_to_profile
INFO: Elapsed time: 60.432s, Critical Path: 17.10s
INFO: 560 processes: 560 linux-sandbox.
INFO: Build completed successfully, 567 total actions

Build broken due to bad dependencies import

❯ bazel build src:all                                  
INFO: Repository com_google_googletest instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
WARNING: Download from https://github.com/google/googletest/archive/master.zip failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
ERROR: An error occurred during the fetch of repository 'com_google_googletest':
   Traceback (most recent call last):
	File "/home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/bazel_tools/tools/build_defs/repo/http.bzl", line 111, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/google/googletest/archive/master.zip] to /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/com_google_googletest/master.zip: GET returned 404 Not Found
INFO: Repository com_google_protobuf instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: Repository remote_coverage_tools instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: Repository boringssl instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: /home/ahh/repo/perf_data_converter/src/compat/BUILD:27:11: //src/compat:test_compat depends on @com_google_googletest//:gtest_main in repository @com_google_googletest which failed to fetch. no such package '@com_google_googletest//': java.io.IOException: Error downloading [https://github.com/google/googletest/archive/master.zip] to /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/com_google_googletest/master.zip: GET returned 404 Not Found
ERROR: Analysis of target '//src:intervalmap_test' failed; build aborted: Analysis failed
INFO: Elapsed time: 1.383s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

But it's not just tests:


❯ bazel build src:perf_to_profile
INFO: SHA256 (https://codeload.github.com/protocolbuffers/protobuf/zip/master) = 7138c6ec951e60f45efcdf4b2b5ab795a082cd50ae12f6ef79d052b864995218
DEBUG: Rule 'com_google_protobuf' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "7138c6ec951e60f45efcdf4b2b5ab795a082cd50ae12f6ef79d052b864995218"
DEBUG: Repository com_google_protobuf instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: Repository boringssl instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /home/ahh/.cache/bazel/_bazel_ahh/b0269a7be5e98a894844c52eb8573a30/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: /home/ahh/repo/perf_data_converter/src/quipper/BUILD:25:14: every rule of type proto_library implicitly depends upon the target '@com_google_protobuf//:protoc', but this target could not be found because of: error loading package '@com_google_protobuf//': Unable to find package for @rules_pkg//:mappings.bzl: The repository '@rules_pkg' could not be resolved.
ERROR: Analysis of target '//src:perf_to_profile' failed; build aborted: Analysis failed
INFO: Elapsed time: 1.671s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 0 targets configured)
    Fetching @bazel_skylib; fetching
    Fetching @rules_proto; fetching
    Fetching @rules_python; fetching

build error in centos

build error in centos:

clang version:

gcc version:

bazel version:

perf_to_profile: All kernel symbols are collapesed to `[[kernel.kallsyms]]`

Hello up there.

I'm on latest perf_to_profile (b86dd52) and on latest pprof (4b386c29ec) and I've tried to use this tools to have better visualsiation for my call-graph profiles. While for user-space it seems to be working, I've found that

all kernel symbols are collapesd to `[[kernel.kallsyms]]` by perf_to_profile.

For example:

root@deco:/home/kirr# perf record dd if=/dev/urandom of=/dev/null bs=1K count=1M
1048576+0 записей получено
1048576+0 записей отправлено
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 6,90417 s, 156 MB/s
[ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 1.073 MB perf.data (27575 samples) ]

root@deco:/home/kirr# perf report --stdio |head -50
no symbols found in /bin/dd, maybe install a debug package?
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 27K of event 'cycles:ppp'
# Event count (approx.): 23116095014
#
# Overhead  Command  Shared Object     Symbol                              
# ........  .......  ................  ....................................
#
    52.98%  dd       [kernel.vmlinux]  [k] _extract_crng
    31.96%  dd       [kernel.vmlinux]  [k] chacha20_block
     4.82%  dd       [kernel.vmlinux]  [k] copy_user_enhanced_fast_string
     2.41%  dd       [kernel.vmlinux]  [k] _raw_spin_lock_irqsave
     1.78%  dd       [kernel.vmlinux]  [k] _raw_spin_unlock_irqrestore
     1.10%  dd       [kernel.vmlinux]  [k] urandom_read
     0.84%  dd       [kernel.vmlinux]  [k] __check_object_size
     0.39%  dd       [kernel.vmlinux]  [k] check_stack_object
     0.38%  dd       [kernel.vmlinux]  [.] system_call_fast_compare_end
     0.30%  dd       [kernel.vmlinux]  [k] __virt_addr_valid
     0.27%  dd       [kernel.vmlinux]  [k] entry_SYSCALL_64
     0.26%  dd       [kernel.vmlinux]  [k] extract_crng
     0.23%  dd       [kernel.vmlinux]  [k] _copy_to_user
     0.21%  dd       [kernel.vmlinux]  [k] fsnotify
     0.20%  dd       [unknown]         [k] 0x00007f67816e374e
     0.14%  dd       [unknown]         [k] 0x00007f67816e37ae
     0.13%  dd       [kernel.vmlinux]  [k] memset_erms
     0.12%  dd       [kernel.vmlinux]  [k] _crng_backtrack_protect
     0.11%  dd       [kernel.vmlinux]  [k] vfs_write
     0.10%  dd       [kernel.vmlinux]  [k] common_file_perm
     0.09%  dd       [kernel.vmlinux]  [k] vfs_read
     0.09%  dd       [kernel.vmlinux]  [k] security_file_permission
     0.09%  dd       [kernel.vmlinux]  [k] __fdget_pos
     0.08%  dd       [kernel.vmlinux]  [k] __fsnotify_parent
     0.08%  dd       [kernel.vmlinux]  [k] __fget_light
     0.08%  dd       [kernel.vmlinux]  [k] rw_verify_area
     0.07%  dd       [kernel.vmlinux]  [k] aa_file_perm
     0.07%  dd       [kernel.vmlinux]  [k] sys_write
     0.05%  dd       [kernel.vmlinux]  [k] crng_backtrack_protect
     0.04%  dd       [kernel.vmlinux]  [k] sys_read
     0.04%  dd       [kernel.vmlinux]  [k] entry_SYSCALL_64_fastpath
     0.03%  dd       dd                [.] 0x0000000000002ef8
     0.03%  dd       dd                [.] 0x0000000000004b38
     0.03%  dd       [kernel.vmlinux]  [k] apparmor_file_permission
     0.02%  dd       dd                [.] 0x0000000000004a80
     0.02%  dd       dd                [.] 0x0000000000002ef0
     0.02%  dd       dd                [.] 0x0000000000004bd6
     0.01%  dd       dd                [.] 0x0000000000004bb2
     0.01%  dd       dd                [.] 0x0000000000004c73

root@deco:/home/kirr# ./bin/perf_to_profile perf.data perf.pprof
[INFO:quipper/perf_reader.cc:847] Number of events stored: 27757
[ERROR:quipper/perf_parser.cc:217] Unknown event type: 79
[INFO:perf_data_handler.cc:414] guessing main for pid: 6901 /bin/dd

root@deco:/home/kirr# ./src/tools/go/g.env/bin/pprof perf.pprof 
File: perf_4.13
perf-version:4.13.13
perf-command:/usr/bin/perf_4.13 record dd if=/dev/urandom of=/dev/null bs=1K count=1M
Type: cycles:ppp_event
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 22952170132, 99.29% of 23116095014 total
Dropped 5 nodes (cum <= 115580475)
      flat  flat%   sum%        cum   cum%
22952170132 99.29% 99.29% 22952170132 99.29%  [[kernel.kallsyms]]

From outside from a quick glance that seems to be happenning due to perf_to_profilie not handling kernel "dso" properly.

For workloads that depend performance-wise on proper in-kernel behaviour, instead of representing whole kernel integrally as one [[kernel.kallsyms]], it would be good to have pprof visualization (graph + flames) to be working for kernel symbols too.

Thanks beforehand,
Kirill

/cc @aalexand

perf_to_profile crashes on a 3GB perf.data file

perf record ./src/bitcoind -datadir=mainnet -server=0 -connect=127.0.0.1 -dbcache=16000

I ran the above command to capture a perf of a long running process (3-4 hours) and got the following error:

[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/stubs/stringpiece.cc:50] size too big: 18446744072542064660 details: string length exceeds max size
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  size too big: 18446744072542064660 details: string length exceeds max size
fish: Job 1, '../perf_data_converter/result/b…' terminated by signal SIGABRT (Abort)

The perf.data file generated by the perf was 3GB. It's too big to attach in the github issues but I can provide it on request.

EDIT: I just saw #123 and should mention that the binary I was using was built on b1c2b82a8ec75b016c36da31c845491c75b9adae

How to change the boringssl repo to github?

Sometimes, I failed to clone boringssl from googlesource, so I try to change the url of boringssl to its mirror on github. I can clone it correctly, but bazel always outputs "no such package '@boringssl//': BUILD file not found on package path and referenced by '//src/quipper:binary_data_utils'".

"Couldn't find any package by glob 'linux-tools-4.9.0-6-amd64" error when following bazel build instructions

$ sudo apt-get -y install g++ git libelf-dev libcap-dev linux-tools-`uname -r`
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package linux-tools-4.9.0-6-amd64
E: Couldn't find any package by glob 'linux-tools-4.9.0-6-amd64'
E: Couldn't find any package by regex 'linux-tools-4.9.0-6-amd64'

fips_break_test.h: No such file or directory

When trying to compile this project on both Ubuntu 18.04 and Debian 9 operating systems, I get the following error:

external/boringssl/src/crypto/fipsmodule/bcm.c:32:10: fatal error: fips_break_test.h: No such file or directory
 #include "fips_break_test.h"
          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
Target //src:perf_to_profile failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 68.100s, Critical Path: 9.56s
INFO: 534 processes: 8 internal, 526 processwrapper-sandbox.

Any ideas?

Thanks 🙏🏻

"PERF_SAMPLE_REGS_USER is not yet supported" fatal error on perf.data recorded with `--call-graph dwarf`

On Ubuntu Focal:

$ uname -a
Linux mark-t510 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ perf -v
perf version 5.11.22

To reproduce, I ran this command. It'd probably work with any other command too (where ninja.dot is https://graphviz.org/Gallery/directed/ninja.gv.txt):

$ sudo perf record -F 4000 -g --call-graph dwarf -o ninja.data dot -Tsvg ninja.dot

Running this command, I get these fatal errors:

$ projects/perf_data_converter/bazel-bin/src/perf_to_profile -i ninja.data -o  ninja_pprof.profile
[WARNING:src/quipper/perf_reader.cc:1247] Skipping 88 bytes of metadata: HEADER_CPU_TOPOLOGY
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event PERF_RECORD_CPU_MAP
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[ERROR:src/quipper/sample_info_reader.cc:385] PERF_SAMPLE_REGS_USER is not yet supported.
[ERROR:src/quipper/perf_reader.cc:1023] Couldn't read event PERF_RECORD_SAMPLE
[ERROR:src/perf_data_converter.cc:718] Could not read input perf.data
[FATAL:src/perf_to_profile.cc:33] Expected profile vector to have one element. Got:0

Looks like the root cause is the PERF_SAMPLE_REGS_USER is not yet supported. message.

Everything works fine if I drop the --call-graph dwarf and use the default frame-pointer-based callstack unwinding:

$ sudo perf record -F 4000 -g -o ninja-g.data dot -Tsvg ninja.dot
$ projects/perf_data_converter/bazel-bin/src/perf_to_profile -i ninja-g.data -o  ninja_pprof.profile
[WARNING:src/quipper/perf_reader.cc:1247] Skipping 88 bytes of metadata: HEADER_CPU_TOPOLOGY
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event PERF_RECORD_CPU_MAP
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:src/quipper/perf_reader.cc:1039] Skipping unsupported event UNKNOWN_EVENT_18
[INFO:src/quipper/perf_reader.cc:1030] Number of events stored: 441
[INFO:src/quipper/perf_parser.cc:267] Parser processed: 183 MMAP/MMAP2 events, 2 COMM events, 0 FORK events, 1 EXIT events, 252 SAMPLE events, 98 of these were mapped, 0 SAMPLE events with a data address, 0 of these were mapped
[ERROR:src/quipper/perf_parser.cc:289] Only 38% of samples had all locations mapped to a module, expected at least 95%
[INFO:src/perf_data_handler.cc:87] Using the build id found for the file name: [kernel.kallsyms], build id: 9ae838e603db988d68009a8c3deb0cb1184c6824.
[WARNING:src/perf_data_handler.cc:463] stat: missing_callchain_mmap 593/1659

"fatal error: 'byteswap.h' file not found" while building OSS quipper

 perf_data_converter (master) ✔ bazel build src:perf_to_profile                                                                                                                                          k8s [kubernetes-admin@kubernetes]

INFO: Invocation ID: 3c80c53b-bb6b-4ba4-82c2-1e2a29e678f2
DEBUG: Rule 'io_bazel' modified arguments {"sha256": "c290fbcc6de1e1a4d94408d32dd962aec90db6c670020784cf7a495385deb954"}
DEBUG: Rule 'boringssl' modified arguments {"sha256": "65991a30c481a2f753f2bca8f51ca84183d2d8be452514bd1fac1b774e478fb2"}
INFO: Analysed target //src:perf_to_profile (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: From Linking external/boringssl/libcrypto.a:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/cpu-aarch64-fuchsia.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/cpu-aarch64-linux.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/cpu-arm-linux.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/cpu-arm.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/cpu-ppc64le.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/poly1305.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/poly1305_arm.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/deterministic.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/fuchsia.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/windows.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/refcount_c11.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/thread_none.pic.o has no symbols
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/external/boringssl/_objs/crypto/thread_win.pic.o has no symbols
ERROR: /Users/lpp/Desktop/perf_data_converter/src/quipper/BUILD:96:1: C++ compilation of rule '//src/quipper:dso' failed (Exit 1) cc_wrapper.sh failed: error executing command external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 60 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from src/quipper/dso.cc:5:
In file included from src/quipper/dso.h:14:
In file included from src/quipper/data_reader.h:11:
src/quipper/binary_data_utils.h:8:10: fatal error: 'byteswap.h' file not found
#include <byteswap.h>
         ^~~~~~~~~~~~
1 error generated.
Target //src:perf_to_profile failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 19.511s, Critical Path: 3.27s
INFO: 239 processes: 239 darwin-sandbox.
FAILED: Build did NOT complete successfully
 perf_data_converter (master) ✔ bazel build src:perf_to_profile                   k8s [kubernetes-admin@kubernetes]

 perf_data_converter (master) ✔                                                   k8s [kubernetes-admin@kubernetes]
 perf_data_converter (master) ✔ git rev-parse HEAD                                k8s [kubernetes-admin@kubernetes]
Alias tip: g rev-parse HEAD
0a61af58974212f0ec3c40833b563fa587f9f9e3
 perf_data_converter (master) ✔                                                   k8s [kubernetes-admin@kubernetes]
 perf_data_converter (master) ✔ uname -a                                          k8s [kubernetes-admin@kubernetes]
Darwin nmiculinics-MacBook-Pro.local 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64

branch stack reader needs updating for new information

The branch stack entry reader will warn about reserved bits being non-zero, so the reader needs to know about all the fields that the kernel is producing:

diff --git a/src/quipper/kernel/perf_event.h b/src/quipper/kernel/perf_event.h
index 9f44a26..2a9ac28 100644
--- a/src/quipper/kernel/perf_event.h
+++ b/src/quipper/kernel/perf_event.h
@@ -1065,7 +1065,11 @@ struct perf_branch_entry {
       in_tx : 1,     /* in transaction */
       abort : 1,     /* transaction abort */
       cycles : 16,   /* cycle count to last branch */
-      reserved : 44;
+      type : 4,      /* branch type, or 15 to indicate new type */
+      spec : 2,      /* speculation outcome */
+      new_type : 4,  /* branch type (new types) */
+      priv : 3,      /* privilege level */
+      reserved : 31;
 };

 }  // namespace quipper
diff --git a/src/quipper/kernel/perf_internals.h b/src/quipper/kernel/perf_internals.h
index 3906e1f..f72122a 100644
--- a/src/quipper/kernel/perf_internals.h
+++ b/src/quipper/kernel/perf_internals.h
@@ -259,7 +259,11 @@ struct branch_flags {
   u64 in_tx : 1;
   u64 abort : 1;
   u64 cycles : 16;
-  u64 reserved : 44;
+  u64 type : 4;
+  u64 spec : 2;
+  u64 new_type : 4;
+  u64 priv : 3;
+  u64 reserved : 31;
 };

 struct branch_entry {

Probably, it should save at least some of this new information in the entry. The 'type' field may be needed by whoever is consuming this infomation, in order to ignore or distinguish some kinds of branches (e.g. interrupts). If type==15, new_type is used as an extended type field.

(Support for 'type' seems to have been added by the recent "update to latest internal version" merge, but "new_type" is still missing and we are already seeing kernels make use of that.)

failed to compile in linux environment

bazel version: 3.5.0

/root/.cache/bazel/_bazel_root/a08c2e4811c846650b733c6fc815a920/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in
ERROR: /root/.cache/bazel/_bazel_root/a08c2e4811c846650b733c6fc815a920/external/boringssl/BUILD:203:11: @platforms//os:linux is not a valid configuration key for @boringssl//:crypto

cc_library(
name = "crypto",
srcs = crypto_sources + crypto_internal_headers + crypto_sources_asm,
hdrs = crypto_headers + fips_fragments,
copts = boringssl_copts_c11,
includes = ["src/include"],
linkopts = select({
# Android supports pthreads, but does not provide a libpthread
# to link against.
"@platforms//os:android": [],
"@platforms//os:macos": [],
"@platforms//os:ios": [],
"@platforms//os:tvos": [],
"@platforms//os:watchos": [],
"@platforms//os:windows": ["-defaultlib:advapi32.lib"],
"//conditions:default": ["-lpthread"],
}),
visibility = ["//visibility:public"],
)

Could NOT find Protobuf (missing: Protobuf_LIBRARIES Protobuf_INCLUDE_DIR)

I hava download all the source code, and use the command flowed:

root@x86-centos7:build$ cmake -DCMAKE_INSTALL_PREFIX=/home/name/test/autofdo-master/install -DLLVM_PATH=/home/name/llvm-dev_12x/install/bin ../

when run the make, I got this error:

CMake Error at /usr/local/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find Protobuf (missing: Protobuf_LIBRARIES Protobuf_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/local/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/share/cmake-3.17/Modules/FindProtobuf.cmake:626 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:181 (find_package)
  CMakeLists.txt:513 (config_with_llvm)

how can i resolve this?

build failed with bazel 4.0, Arch Linux

I tried to build with bazel 4.0.0, and got an error:

 in proto_lang_toolchain rule @com_google_protobuf//:cc_toolchain: '@com_google_protobuf//:cc_toolchain' does not have mandatory provider 'ProtoInfo'.

system, archlinux:

Linux 5.10.13-arch1-2 #1 SMP PREEMPT Sat, 06 Feb 2021 11:07:29 +0000 x86_64 GNU/Linux

bazel version:

Build label: 4.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 21 14:04:24 2021 (1611237864)
Build timestamp: 1611237864
Build timestamp as int: 1611237864

I found bazelbuild/bazel#12887, and try to upgrade protobuf to v13.0, but there are some other dependency issues.

Modify .travis.yml to build quipper also with libc++

This is to make sure quipper builds with libc++ as chromium os uses the github copy of quipper.

perf_to_profile drops perf symbols

perf_to_profile drops symbol names when converting from perf output to pprof format.

$ perf record tree /usr
$ perf report
Samples: 4K of event 'cpu-clock:uhH', Event count (approx.): 1236250000
Overhead  Command  Shared Object     Symbol
  12.52%  tree     tree              [.] 0x000000000000c964
  11.06%  tree     libc-2.27.so      [.] vfprintf
   4.63%  tree     libc-2.27.so      [.] write
   4.33%  tree     libc-2.27.so      [.] __lxstat64
   3.84%  tree     libc-2.27.so      [.] _IO_file_xsputn
   3.01%  tree     libc-2.27.so      [.] __fprintf_chk
   2.75%  tree     libc-2.27.so      [.] __strcoll_l
   2.43%  tree     libc-2.27.so      [.] c32rtomb
   1.92%  tree     libc-2.27.so      [.] malloc
   1.80%  tree     libc-2.27.so      [.] 0x000000000016af98
   1.72%  tree     libc-2.27.so      [.] 0x00000000000e00a8
   1.62%  tree     libc-2.27.so      [.] 0x000000000018e5a5
   1.54%  tree     libc-2.27.so      [.] cfree
   1.38%  tree     libc-2.27.so      [.] iswprint
   1.01%  tree     libc-2.27.so      [.] 0x000000000018e590
   0.87%  tree     libc-2.27.so      [.] __open_nocancel
   0.87%  tree     libc-2.27.so      [.] 0x0000000000169d90
   0.75%  tree     libc-2.27.so      [.] __close_nocancel
   0.75%  tree     libc-2.27.so      [.] __fxstat64
   0.75%  tree     libc-2.27.so      [.] readdir64
   0.69%  tree     libc-2.27.so      [.] 0x0000000000169d7a
   0.69%  tree     libc-2.27.so      [.] 0x000000000018e59c
   0.67%  tree     libc-2.27.so      [.] 0x0000000000169d98
   0.67%  tree     libc-2.27.so      [.] 0x000000000018e230
   0.65%  tree     libc-2.27.so      [.] 0x000000000018e69d
   0.61%  tree     tree              [.] 0x000000000000d672
   0.59%  tree     tree              [.] 0x0000000000007bbe
   0.57%  tree     libc-2.27.so      [.] _IO_default_xsputn
   0.53%  tree     libc-2.27.so      [.] 0x000000000018e690
   0.51%  tree     tree              [.] 0x000000000000c949
   0.51%  tree     tree              [.] 0x000000000000c95c
   0.44%  tree     libc-2.27.so      [.] 0x0000000000169d6c
   0.42%  tree     libc-2.27.so      [.] 0x000000000018e22b
   0.42%  tree     tree              [.] 0x000000000000d675
   0.40%  tree     libc-2.27.so      [.] 0x00000000000e00b0
   0.40%  tree     libc-2.27.so      [.] 0x000000000016af9e
   0.38%  tree     libc-2.27.so      [.] _IO_file_overflow
   0.38%  tree     libc-2.27.so      [.] 0x000000000018e227
   0.36%  tree     libc-2.27.so      [.] _IO_file_write
   0.36%  tree     libc-2.27.so      [.] mbstowcs
   0.34%  tree     libc-2.27.so      [.] 0x0000000000169d60
   0.34%  tree     libc-2.27.so      [.] 0x0000000000169de5
   0.32%  tree     libc-2.27.so      [.] _dl_mcount_wrapper_check
...

After conversion pprof data only contains the "Shared Object" name pprof accumulates all profiles under shared object names instead of symbols.

(pprof) top
Showing nodes accounting for 1236000000, 100% of 1236250000 total
Dropped 1 node (cum <= 6181250)
      flat  flat%   sum%        cum   cum%
 956250000 77.35% 77.35%  956250000 77.35%  [libc-2.27.so]
 279750000 22.63%   100%  279750000 22.63%  [tree]

Build fails with 404 errors because bazel doesn't follow redirects

The github.com links now seem to redirect to codeload.github.com but Bazel 4.1.0 just reports 404 even though the redirect works in a browser.

The error looks like this:

WARNING: Download from https://github.com/protocolbuffers/protobuf/archive/master.zip failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
ERROR: An error occurred during the fetch of repository 'com_google_protobuf':
   Traceback (most recent call last):
	File "/home/jamcla02/.cache/bazel/_bazel_jamcla02/b2015f4763ecfa490ff19feafa4074f9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 111, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/protocolbuffers/protobuf/archive/master.zip] to /home/jamcla02/.cache/bazel/_bazel_jamcla02/b2015f4763ecfa490ff19feafa4074f9/external/com_google_protobuf/temp14020832879063832399/master.zip: GET returned 404 Not Found

I see that PRs are not supported, so I will leave the patch that fixes it here in case anybody has the same issue. It can be applied with git am <patchfile>:

From 30f5dcd29d92972434f0a9fa1e17ec834d278b9d Mon Sep 17 00:00:00 2001
From: James Clark <[email protected]>
Date: Mon, 2 Aug 2021 11:48:15 +0100
Subject: [PATCH] Fix build

The existing github links redirect to codeload.github.com
and redirection isn't supported by bazel so update them.
They also lost their extensions so add 'type' to force it.

python_rules also changed 'master' to 'main'
---
 WORKSPACE | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/WORKSPACE b/WORKSPACE
index c416ba5..16c9be2 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -18,16 +18,18 @@ http_archive(
 # rules_python is a dependency for protobuf.
 http_archive(
     name = "rules_python",
-    urls = ["https://github.com/bazelbuild/rules_python/archive/master.tar.gz"],
-    strip_prefix = "rules_python-master",
+    urls = ["https://codeload.github.com/bazelbuild/rules_python/tar.gz/main"],
+    strip_prefix = "rules_python-main",
+    type = "tar.gz"
 )
 
 # proto_library, cc_proto_library, and java_proto_library rules implicitly
 # depend on @com_google_protobuf for protoc and proto runtimes.
 http_archive(
     name = "com_google_protobuf",
-    urls = ["https://github.com/protocolbuffers/protobuf/archive/master.zip"],
+    urls = ["https://codeload.github.com/protocolbuffers/protobuf/zip/master"],
     strip_prefix = "protobuf-master",
+    type = "zip"
 )
 
 http_archive(
@@ -35,8 +37,9 @@ http_archive(
     # Gitiles creates gzip files with an embedded timestamp, so we cannot use
     # sha256 to validate the archives.  We must rely on the commit hash and
     # https. Commits must come from the master-with-bazel branch.
-    urls = ["https://github.com/google/boringssl/archive/master-with-bazel.zip"],
+    urls = ["https://codeload.github.com/google/boringssl/zip/master-with-bazel"],
     strip_prefix = "boringssl-master-with-bazel",
+    type = "zip"
 )
 
 # zlib is a dependency of protobuf.
-- 
2.28.0

Memory corruption when parsing malformed perf data files

Hi team,

Quipper, a perf library used in this repro for Google's perf_data_converter, crashes with signs of memory corruption when parsing malformed perf .data files. Quick automated triage info below as you can check if the issue appears to be an exploitable vulnerability or just a crash as no further analysis was performed.

Quipper library code can be found here:

https://github.com/google/perf_data_converter/tree/master/src/quipper

Repro steps:

$ git clone https://github.com/google/perf_data_converter

(install deps and build)

$ ./perf_data_converter/bazel-bin/src/perf_to_profile -i repro.data -o test -f

repro.data (rename from repro.data.txt, had to add .txt to the extension for github issue file upload)
repro.data.txt

observe crash on console and within a debugger

Crash was found using litefuzz to mutate sample perf files. See below repro demonstration and debug output.

$ ./perf_data_converter/bazel-bin/src/perf_to_profile -i repro.data -o test -f
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted (core dumped)

Debug output from gdb from initial crash triage below.

Note: this uses electric-fence malloc debugger, install with sudo apt-get install electric-fence to trigger the instrumented crash as you can see.

$ gdb -q perf_data_converter/bazel-bin/src/perf_to_profile
Reading symbols from perf_data_converter/bazel-bin/src/perf_to_profile...
(No debugging symbols found in perf_data_converter/bazel-bin/src/perf_to_profile)

(gdb) set environment LD_PRELOAD=/usr/lib/libefence.so

(gdb) r -i ~/repro.data -o /tmp/test.data -f

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff79489ab in mprotect () at ../sysdeps/unix/syscall-template.S:78

(gdb) bt
#0 0x00007ffff79489ab in mprotect () at ../sysdeps/unix/syscall-template.S:78
#1 0x00007ffff7dc623e in Page_AllowAccess () from /usr/lib/libefence.so
#2 0x00007ffff7dc5745 in ?? () from /usr/lib/libefence.so
#3 0x00007ffff7dc5e7e in memalign () from /usr/lib/libefence.so
#4 0x00007ffff7ae9b39 in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00005555555e16ca in __gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::allocate(unsigned long, void const*) ()
#6 0x00005555555e15b8 in std::allocator_traits<std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::allocate(std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >&, unsigned long) ()
#7 0x00005555555e13a6 in std::_Vector_base<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::_M_allocate(unsigned long) ()
#8 0x0000555555600284 in std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::_M_default_append(unsigned long) ()
#9 0x00005555555fc17b in std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > >::resize(unsigned long) ()
#10 0x00005555555f0328 in quipper::PerfReader::ReadCPUTopologyMetadata(quipper::DataReader*, unsigned long) ()
#11 0x00005555555ee39c in quipper::PerfReader::ReadMetadataWithoutHeader(quipper::DataReader*, unsigned int, unsigned long)::{lambda()#1}::operator()() const ()
#12 0x00005555555ee710 in quipper::PerfReader::ReadMetadataWithoutHeader(quipper::DataReader*, unsigned int, unsigned long) ()
#13 0x00005555555ede94 in quipper::PerfReader::ReadMetadata(quipper::DataReader*) ()
#14 0x00005555555f1717 in quipper::PerfReader::ReadFileData(quipper::DataReader*) ()
#15 0x00005555555e8ce6 in quipper::PerfReader::ReadFromData(quipper::DataReader*) ()
#16 0x00005555555e8b69 in quipper::PerfReader::ReadFromPointer(char const*, unsigned long) ()
#17 0x00005555555799ac in perftools::RawPerfDataToProfiles(void const*, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::less<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&, unsigned int, unsigned int, std::map<unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::less, std::allocator<std::pair<unsigned int const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) ()
#18 0x00005555555746c4 in StringToProfiles(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned int, unsigned int) ()
#19 0x00005555555731e5 in main ()

(gdb) i r
rax 0xfffffffffffffff4 -12
rbx 0x7ffff703e730 140737337616176
rcx 0x7ffff79489ab 140737347094955
rdx 0x3 3
rsi 0xd6e6c3000 57687158784
rdi 0x7fff8894c000 140735484837888
rbp 0xd6e6c4000 0xd6e6c4000
rsp 0x7fffffffc9d8 0x7fffffffc9d8
r8 0xffffffff 4294967295
r9 0x0 0
r10 0x22 34
r11 0x202 514
r12 0x7ffff703e758 140737337616216
r13 0x7ffff703e758 140737337616216
r14 0xd6e6c2e80 57687158400
r15 0x7ffff7fc8128 140737353908520
rip 0x7ffff79489ab 0x7ffff79489ab <mprotect+11>
eflags 0x10202 [ IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0

(gdb) exploitable
Description: Segmentation fault on program counter
Short description: SegFaultOnPc (3/22)
Hash: 434b871bba6df1ba821918d14d98847d.3c60d5a82288d85766b6290f18c589bb
Exploitability Classification: EXPLOITABLE
Explanation: The target tried to access data at an address that matches the program counter. This is likely due to the execution of a branch instruction (ex: 'call') with a bad argument, but it could also be due to execution continuing past the end of a memory region or another cause. Regardless this likely indicates that the program counter contents are tainted and can be controlled by an attacker.
Other tags: AccessViolation (21/22)

Recommendation/fix is to provide more input validation and bounds checking to ensure malformed files that could trigger error conditions are instead handled gracefully.

Build failing "fips.c: No such file or directory" since 15.03.2021 (boringssl)

Project build fails for me since yesterday (15.03.2021), it worked fine earlier. The problem seems to lie in boringssl dependency, this change in particular.

$ bazel build src:perf_to_profile
(...)
ERROR: /root/.cache/bazel/_bazel_root/9a84914462919c338e94f245491500b7/external/boringssl/BUILD:151:11: Compiling src/crypto/fipsmodule/bcm.c failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF ... (remaining 34 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF ... (remaining 34 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
external/boringssl/src/crypto/fipsmodule/bcm.c:82:10: fatal error: fips.c: No such file or directory
 #include "fips.c"
          ^~~~~~~~
compilation terminated.
Target //src:perf_to_profile failed to build
(...)

Build this Dockerfile to reproduce:

FROM ubuntu:18.04

RUN apt update && apt install -y npm g++ git libelf-dev libcap-dev
RUN npm install -g @bazel/bazelisk

RUN git clone https://github.com/google/perf_data_converter.git /usr/src/perf_data_converter
WORKDIR /usr/src/perf_data_converter

# uncomment to make the build succeed
# RUN sed -i 's/\/google\/boringssl/\/pbetkier\/boringssl/' WORKSPACE

RUN bazel build src:perf_to_profile

Note the commented out sed. After uncommenting the build does pass. It replaces the official boringssl dependency with my fork that moves history before the boringssl change linked in the beginning.

I'm not sure if this is the right place to file this problem, or should I rather file it in the boringssl project. My knowledge of the cpp ecosystem is also rather limited. I hope you can direct me to the right place :)

Could we get a command-line option for setting `kPidAndTidLabels`?

Would love to be able to get per-thread tags in the output, using this constant:

perf_data_converter/src/perf_data_converter.h

Line 32 in e8a6590

kPidAndTidLabels = 3,

perf_to_profile could take a command line argument for this?

Cannot extract function name from perf.data

Hello! We obtain the "perf.data" file by using "perf record -g -a ./my_execution", but it seems that the "perf.data" database does not include mapping information such as function names. After using "perf_to_profile" to convert the "perf.data" file to protobuf file, we still cannot extract function name information from it. Is it possible that our command is incorrect or lacks certain flags? Thanks for your time in advance.

stat: missing_callchain_mmap message should be a warning, not an error

It is currently logged as an error, but it is not very much actionable for users, so it should be a warning.

[ERROR:perf_data_handler.cc:310] stat: missing_callchain_mmap 14317/113208

Converter fails to parse perf.data files with PERF_SAMPLE_REGS_USER & PERF_SAMPLE_STACK_USER sample types.

When I run converter i get:

[ERROR:src/quipper/sample_info_reader.cc:277] PERF_SAMPLE_REGS_USER is not yet supported.
[ERROR:src/quipper/sample_info_reader.cc:591] Read/skipped 56 bytes, expected 8440 bytes.
[ERROR:src/perf_data_converter.cc:659] Could not read input perf.data
[FATAL:src/perf_to_profile.cc:27] Expected profile vector to have one element.

ERROR: Analysis of target '//src:perf_to_profile' failed; build aborted: error loading package '@com_google_protobuf//': Unable to find package for @rules_java//java:defs.bzl: The repository '@rules_java' could not be resolved.

Not experienced enough how bazel build system works. I am running into this issue when building.

$ bazel build src:perf_to_profile

ERROR: /home/mschulte/.local/opt/perf_data_converter/src/quipper/BUILD:25:1: every rule of type proto_library implicitly depends upon the target '@com_google_protobuf//:protoc', but this target could not be found because of: error loading package '@com_google_protobuf//': Unable to find package for @rules_java//java:defs.bzl: The repository '@rules_java' could not be resolved.
ERROR: Analysis of target '//src:perf_to_profile' failed; build aborted: error loading package '@com_google_protobuf//': Unable to find package for @rules_java//java:defs.bzl: The repository '@rules_java' could not be resolved.
INFO: Elapsed time: 0.105s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 21 targets configured)
    currently loading: @com_google_protobuf//

Here is some potentially useful information about my system that the issue could depend on

g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

bazel

Build label: 0.26.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jun 6 11:05:05 2019 (1559819105)
Build timestamp: 1559819105
Build timestamp as int: 1559819105

clang

clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

java

openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

javac

javac 11.0.11

protoc

libprotoc 3.6.1

perf_to_profile: [ERROR:perf_data_handler.cc:310] stat: missing_callchain_mmap 20547/294830

Triggered when attempting to convert a perf capture of an OCaml program. Unfortunately, I can't share the code, but I might be able to produce a freestanding reproduction case if someone's available to chase this.