Git Product home page Git Product logo

autofdo's Introduction

1. Install prerequisite

    $ sudo apt install libunwind-dev libgflags-dev libssl-dev libelf-dev protobuf-compiler

To build autofdo tool for llvm, it needs llvm major version >= 10. You can either install llvm using command like "sudo apt install llvm-10", or you can build the latest llvm from source.

To build autofdo tool for gcc, no llvm installation is needed.

2. Commands

2.1 Build autofdo tool for llvm

2.1.1 If build llvm from source

    $ git clone https://github.com/llvm/llvm-project.git
    $ mkdir build
    $ cd build
    $ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON \
        -DBUILD_SHARED_LIBS=OFF -DLLVM_PARALLEL_LINK_JOBS=1 -DLLVM_INCLUDE_TESTS=OFF \
        -DLLVM_ENABLE_RTTI=ON -DCMAKE_INSTALL_PREFIX=/path/to/llvm/install \
        -DLLVM_ENABLE_PROJECTS="clang" ../llvm-project
    $ ninja
    $ ninja install

2.1.2 Build autofdo tools

    $ git clone --recursive https://github.com/google/autofdo.git
    $ cd autofdo
    $ mkdir build
    $ cd build
    $ # Note: "-DCMAKE_INSTALL_PREFIX=." must be used, because there is a bug in the basil cmakelist.
    $ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=. -DLLVM_PATH=/path/to/llvm/install ../   
    $ ninja
    $ ninja test

2.2 Build autofdo tool for gcc

2.2.1 Build autofdo tools using system gcc

    $ git clone --recursive https://github.com/google/autofdo.git
    $ cd autofdo
    $ mkdir build
    $ cd build
    $ # Note: "-DCMAKE_INSTALL_PREFIX=." must be used, because there is a bug in the basil cmakelist.
    $ cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=. ../   
    $ ninja

2.2.2 Build autofdo tools using customized built gcc

  1. download gcc tarball, for example, gcc-11.2.0.tar.gz (other versions of gcc are similar), and extract
  2. download prerequisites by
    $ cd gcc-release-11.2.0
    $ ./contrib/download_prerequisites
  1. config, build and install gcc
    $ cd gcc-release-11.2.0
    $ mkdir build
    $ cd build
    $ ../gcc-releases-gcc-11.2.0/configure \
          --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu \
          --prefix=${HOME}/installs/gcc-11.2.0 \
          --enable-checking=release \
          --enable-languages=c,c++ \
          --disable-multilib \
          --program-suffix=-11.2
    $ make -j24 install
  1. config autofdo and build:
    $ export PATH=${HOME}/installs/gcc-11.2.0/bin:$PATH
    $ CXX=${HOME}/installs/gcc-11.2.0/bin/x86_64-linux-gnu-g++-11.2
    $ CC=${HOME}/installs/gcc-11.2.0/bin/x86_64-linux-gnu-gcc-11.2

    $  mkdir -p build-with-gcc
    $  cd build-with-gcc
    $  cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=. \
        -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" ../autofdo

autofdo's People

Contributors

algrant-arm avatar boomanaiden154 avatar danielboulby-arm avatar danielcdh avatar davidcallahan avatar denik82 avatar dnovillo avatar erozenfeld avatar gburgessiv avatar kim-phillips-arm avatar lannadorai avatar marksantaniello avatar mtrofin avatar rlavaee avatar rtoax avatar ruijiefang avatar rvanka avatar rwalkr avatar shenhanc avatar shenhanc78 avatar taewookoh avatar tmsri avatar vapier avatar wmi-11 avatar wmi0 avatar zerodefect avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

autofdo's Issues

functioninfo.h:163 Check failed: logical_num > 0 (0 vs. 0)

I'm using the gcc version of autofdo tool. And when I tried to convert my perf.data into gcov file, the tool reports an error as below:

F20210608 11:36:58.429226 49029 functioninfo.h:163] Check failed: logical_num > 0 (0 vs. 0)
*** Check failure stack trace: ***
Aborted

The commands that I'm using to collect perf.data and convert it are:
perf record -b "my binary"
./create_gcov --binary=../battle_data/battlesvr --profile=../battle_data/perf.data_battle --gcov=ver1.1_print.gcov -gcov_version=1

In addition, I have tried to build a llvm version of the autofdo tool and it successfully converts the perf.data. Any help would be appreciated!

Missing aclocal.m4

The aclocal.m4 is missing, so we need to regenerate every time. If users have a different version of automake, it complains.

Unknown event type 79

Hello.

I see probably the same as discussed in #37. My env is:

$ uname -a
Linux marxinbox 4.10.8-1-default #1 SMP PREEMPT Fri Mar 31 17:16:00 UTC 2017 (ea9dcd4) x86_64 x86_64 x86_64 GNU/Linux

$ perf --version
perf version 4.8.4

I tested both autotools 0.15 and current trunk, both behave the same:
There's content of my perf.data file

$ perf script -i ~/Downloads/indir-call-prof.perf.data | head -n20
Failed to open /home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01, continuing without symbols
 indir-call-prof 13298  6018.058609:          1 cpu/event=0xC4,umask=0x20/pu:      7f2bac7ffca7 _dl_start+0xa7 (/lib64/ld-2.25.so)
 indir-call-prof 13298  6018.058614:          1 cpu/event=0xC4,umask=0x20/pu:      7f2bac7ffca3 _dl_start+0xa3 (/lib64/ld-2.25.so)
 indir-call-prof 13298  6018.058617:          7 cpu/event=0xC4,umask=0x20/pu:      7f2bac7ffc98 _dl_start+0x98 (/lib64/ld-2.25.so)
 indir-call-prof 13298  6018.058620:         76 cpu/event=0xC4,umask=0x20/pu:      7f2bac7ffd59 _dl_start+0x159 (/lib64/ld-2.25.so)
 indir-call-prof 13298  6018.058625:        813 cpu/event=0xC4,umask=0x20/pu:      7f2bac812808 _dl_sysdep_start+0x48 (/lib64/ld-2.25.so)
 indir-call-prof 13298  6018.058629:       6330 cpu/event=0xC4,umask=0x20/pu:      7f2bac811aa0 __tunables_init+0x60 (/lib64/ld-2.25.so)
 indir-call-prof 13298  6018.058638:      52447 cpu/event=0xC4,umask=0x20/pu:      7f2bac811b13 __tunables_init+0xd3 (/lib64/ld-2.25.so)
 indir-call-prof 13298  6018.058813:     217332 cpu/event=0xC4,umask=0x20/pu:            400510 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.058928:     229110 cpu/event=0xC4,umask=0x20/pu:            400510 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.059049:     262561 cpu/event=0xC4,umask=0x20/pu:            400510 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.059187:     297514 cpu/event=0xC4,umask=0x20/pu:            400422 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.059344:     327593 cpu/event=0xC4,umask=0x20/pu:            400408 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.059515:     352199 cpu/event=0xC4,umask=0x20/pu:            400422 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.059699:     372308 cpu/event=0xC4,umask=0x20/pu:            400510 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.059894:     388913 cpu/event=0xC4,umask=0x20/pu:            400408 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.060097:     402754 cpu/event=0xC4,umask=0x20/pu:            400422 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.060307:     414372 cpu/event=0xC4,umask=0x20/pu:            400408 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.060524:     424153 cpu/event=0xC4,umask=0x20/pu:            400408 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.060745:     432436 cpu/event=0xC4,umask=0x20/pu:            400510 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)
 indir-call-prof 13298  6018.060971:     439451 cpu/event=0xC4,umask=0x20/pu:            400422 [unknown] (/home/marxin/Programming/gcc/objdir/gcc/testsuite/gcc/indir-call-prof.x01)

indir-call-prof.perf.data.zip.tar.gz

Thanks for help.
Martin

make failure with 'configure: error: cannot run /bin/sh ./../../config.sub'

$make
make all-am
make[1]: Entering directory /home/longfei.alf/autofdo' echo "[MAKE] Building protobuf" [MAKE] Building protobuf (cd third_party/protobuf/; CC="gcc" CXX="g++" LDFLAGS=" -g " CPPFLAGS=" -g " ./configure --disable-shared --enable-static ) checking whether to enable maintainer-specific portions of Makefiles... yes configure: error: cannot run /bin/sh ./../../config.sub make[1]: *** [third_party/protobuf/src/protoc] Error 127 make[1]: Leaving directory /home/longfei.alf/autofdo'
make: *** [all] Error 2

Create_gcov always dump 1600 bytes

I am trying to covert captured perf data to gcc format.But always 1600 bytes are dumped.

Machine where I captured perf data is having LBR support.
perf record -o /tmp//perf.data -b -e cpu/event=0xc4,umask=0x20,name=br_inst_retired_near_taken,period=400009/pp -p 11045 -F 10 -- sleep 900

Then I used following command to create gcov data
create_gcov --binary=<> --profile=perf.data --gcov=perf-prod.afdo --profiler=perf -log_dir=. -gcov_version=1

Binary is unstripped and have debug symbols

In log following
Log file created at: 2019/05/10 12:04:54
Running on machine: a107-2-2-3
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0510 12:04:54.359030 5412 module_grouper.cc:60] Cannot read compilation info from binary. Please use -frecord-compilation-info-in-elf when building the binary
I0510 12:04:55.822785 5412 perf_reader.cc:798] Number of events stored: 1164694
I0510 12:05:03.224366 5412 perf_parser.cc:256] Parser processed: 1066560 MMAP/MMAP2 events, 405 COMM events, 0 FORK events, 0 EXIT events, 90695 SAMPLE events, 73020 of these were mapped

There is no other information.I tried many combinations but it is not working.What could be the issue?

LLVM profile (text format) has corrupt names for called functions

Called function names in LLVM profiles are sometimes corrupt - see the sixth line here:

hot:8187856:16040
 3: 12976
 4: 12976
 5: 12976
 6: 12976
 3: P▒}:2335680
  2: 12976
  3: 12976

This looks to be a string reference counting problem. This fixes it:

diff --git a/llvm_profile_writer.cc b/llvm_profile_writer.cc
index 3da7a1f..04b3f7c 100644
--- a/llvm_profile_writer.cc
+++ b/llvm_profile_writer.cc
@@ -94,10 +94,10 @@ void LLVMProfileBuilder::VisitCallsite(const Callsite &callsite) {
     inline_stack_.pop_back();
   }
   auto &caller_profile = *(inline_stack_.back());
-  auto CalleeName = GetNameRef(Symbol::Name(callsite.second)).str();
+  auto CalleeName = GetNameRef(Symbol::Name(callsite.second));
   auto &callee_profile =
       caller_profile.functionSamplesAt(llvm::sampleprof::LineLocation(
-          line, discriminator))[CalleeName];
+          line, discriminator))[CalleeName.str()];
   callee_profile.setName(CalleeName);
   inline_stack_.push_back(&callee_profile);
 }

Compile error in llvm_profile_writer.cc

Error details:

llvm_profile_writer.cc: In member function ‘virtual void autofdo::LLVMProfileBuilder::VisitTopSymbol(const string&, const autofdo::Symbol*)’:
llvm_profile_writer.cc:84:11: error: ‘class llvm::sampleprof::FunctionSamples’ has no member named ‘setName’
profile.setName(name_ref);
^
llvm_profile_writer.cc: In member function ‘virtual void autofdo::LLVMProfileBuilder::VisitCallsite(const Callsite&)’:
llvm_profile_writer.cc:100:31: error: no matching function for call to ‘llvm::sampleprof::FunctionSamples::functionSamplesAt(llvm::sampleprof::LineLocation)’
line, discriminator));
^
llvm_profile_writer.cc:100:31: note: candidate is:
In file included from llvm_profile_writer.h:8:0,
from llvm_profile_writer.cc:30:
/usr/include/llvm/ProfileData/SampleProf.h:243:20: note: llvm::sampleprof::FunctionSamples& llvm::sampleprof::FunctionSamples::functionSamplesAt(const llvm::sampleprof::CallsiteLocation&)
FunctionSamples &functionSamplesAt(const CallsiteLocation &Loc) {
^
/usr/include/llvm/ProfileData/SampleProf.h:243:20: note: no known conversion for argument 1 from ‘llvm::sampleprof::LineLocation’ to ‘const llvm::sampleprof::CallsiteLocation&’
Makefile:1065: recipe for target 'create_llvm_prof-llvm_profile_writer.o' failed
make[1]: *** [create_llvm_prof-llvm_profile_writer.o] Error 1
make[1]: Leaving directory '/home/koukos/Workspace/tools/autofdo'
Makefile:504: recipe for target 'all' failed
make: *** [all] Error 2

I could definitely fix it in the sourcecode; however, i am not entirely sure if it's indeed a bug or if it comes from a mis-configuration on my system. I am using gcc/g++ 4.8.5 on OpenSuse LEAP 42.2 system with linux kernel 4.4 (LTS) and libc 2.22. LLVM and clang version 3.8.0

Also confirmed on Ubuntu 16.04 LTS with similar compiler and kernel versions. LLVM 3.8, kernel 4.4, gcc 5.4, libc 2.23, ...

Before that i had some issues with autoconf/automake as mentioned in previous posts that i somehow worked around with: autoreconf -ivf

Any ideas?

Thanks in advance,
Konstantinos.

Please give an example in order to pass several binaries to create_gcov?

One can convert profile data using the following command

create_gcov --binary=your_program.unstripped --profile=perf.data --gcov=profile.afdo

But how to do this when the program is proprietary binary and the profiled code consists of several optional shared object file? I’m meaning, how to pass several binaries to create_gcov when there’s still a single perf.data for all of them?

create_gcov report "No buildid found in binary"

Hi,

The create_gcov always report cannot found binary buildid, but my kernel
already compiled with buildid.

[root@localhost autofdo]# ./create_gcov --binary=../linux/vmlinux --profile=../linux/perf.data --gcov=1
E0512 10:06:07.650653 14367 elf_reader.cc:789] Malformed .note.gnu.build-id section.
E0512 10:06:07.657847 14367 sample_reader.cc:233] No buildid found in binary
[root@localhost autofdo]#
[root@localhost autofdo]#
[root@localhost autofdo]# readelf -n ../linux/vmlinux|grep build
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)

Can't build auto after latest commits

build with:
git clone --recursive https://github.com/google/autofdo.git
aclocal -I .; autoheader; autoconf; automake --add-missing -c
./configure
make

depbase=`echo third_party/perf_data_converter/src/quipper/perf_data_utils.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src -I./third_party/perf_data_converter/src/quipper  -std=gnu++11 -I./third_party/protobuf/src -g -O2 -std=gnu++11 -MT third_party/perf_data_converter/src/quipper/perf_data_utils.o -MD -MP -MF $depbase.Tpo -c -o third_party/perf_data_converter/src/quipper/perf_data_utils.o third_party/perf_data_converter/src/quipper/perf_data_utils.cc &&\
mv -f $depbase.Tpo $depbase.Po
In file included from third_party/perf_data_converter/src/quipper/perf_data_utils.cc:10:0:
third_party/perf_data_converter/src/quipper/compat/proto.h:14:26: fatal error: perf_data.pb.h: No such file or directory
compilation terminated.
Makefile:882: recipe for target 'third_party/perf_data_converter/src/quipper/perf_data_utils.o' failed
make[1]: *** [third_party/perf_data_converter/src/quipper/perf_data_utils.o] Error 1
make[1]: Leaving directory '/usr/src/autofdo'
Makefile:539: recipe for target 'all' failed
make: *** [all] Error 2

System is a Ubuntu 16.04.4 LTS

how to use this tools for autoLIPO(autoFDO + LIPO)in gcc google 4_9 branch ?

hi, @danielcdh
I recently read the code of LIPO (Profile Feedback Based Lightweight IPO) in gcc and find that the theory is very good. In this paper and this patches , I see that LIPO has realized the combination with autoFDO, avoiding the disadvantage of LIPO having to recompile the FDO process.

I tried to use the autoFDO tool to generate the gcda and imports files required by LIPO, and although imports were automatically generated, it was always empty, causing LIPO not to optimize the program.

Therefore, I want to know how to make the imports file generated by the autoFDO tool have content instead of empty and make LIPO effective.

Looking forward to your reply!

thanks,
ijinma

Mention -g1 with -gmlt

README has: "This means that you need to compile the binary with "-gmlt""

Did -gmlt become -g1 in newer gcc? README maybe should mention both (in case someone is on a -gmlt version).

sample_merger: Unable to map MMAP2 event!

I sometimes see the following problem with sample_merger. Often I can just take a new perf collection and this will go away. I will try to get a small repro for this.

E0801 10:19:00.549764 1907491 address_mapper.cc:148] Could not find space to map addr=7fd94e205000 with size 206000
F0801 10:19:00.550119 1907491 perf_parser.cc:151] Check failed: MapMmapEvent(&event.mmap2, i) Unable to map MMAP2 event!

*** Check failure stack trace: ***
    @           0x452f55  google::LogMessage::Flush()
    @           0x4537c1  google::LogMessageFatal::~LogMessageFatal()
    @           0x44c0a0  quipper::PerfParser::ProcessEvents()
    @           0x44c713  quipper::PerfParser::ParseRawEvents()
    @           0x42726a  autofdo::PerfDataSampleReader::Append()
    @           0x4269b4  autofdo::SampleReader::ReadAndSetTotalCount()
    @           0x41c8fc  autofdo::ProfileCreator::ReadSample()
    @           0x41d055  autofdo::MergeSample()
    @           0x408b67  main
    @     0x7f2d20e6a0f6  __libc_start_main
    @           0x409e9d  (unknown)
    @              (nil)  (unknown)

line numbers don't match with autofdo output for llvm

Hello,
I'm working on a simple bubblesort code. I'm on Ubuntu 16.04, I compile it with LLVM5 and generate the profile output using autofdo as shown below.

clang -O2  -g sort.c -o sort_g_O0
sudo perf record -e cpu/event=0xc4,umask=0x20,name=br_inst_retired_near_taken,period=400009/pp -b ./sort_g_O0
sudo ../../autofdo/create_llvm_prof --binary=./sort_g_O0 --out=sort_g_O0.prof

Then I use llvm-prof to read this sampled profile. The output gives me some weird line numbers that do not match with the source code.

The (partial) output of
llvm-profdata show -sample sort_g_O0.prof
is this:
2: inlined callee: sort_array: 583159, 0, 4 sampled lines
Samples collected in the function's body {
1: 0
3: 1
4: 1
65501: 1
}
Samples collected in inlined callsites {
6: inlined callee: bubble_sort: 583142, 0, 6 sampled lines
Samples collected in the function's body {
2: 3
4: 12608
5: 16740
7: 4209
8: 4209
65516: 12673
}
No inlined callsites in this function

I tried to figure it out myself,

  • I changed the optimization level to 0 and I still get the weird line number in sort_array function.
  • I printed the offset and line numbers in llvm_profile_writer.cc and indeed something is wrong. But I'm lost tracking it back to dwarf info.

Can somebody help me on this? Am I doing something wrong, if not, why am I seeing these line numbers?

I attached the files.
llvm_autofdo.zip

Thanks!

create_gcov produces invalid output files

Hi,

I've download and built the autofdo project, however, when I do perf run and try to reformat the data using create_gcov the resulting files do not look correct. This is what I am seeing:

[sysprog@nucky ~] $ /usr/local/bin/create_gcov --binary=/usr/bin/lz_batch --profile=/tmp/profiling-data/prof20k-bir.raw -gcov_version 1 --gcov=/tmp/profiling-data/prof20k-bir.afdo
[sysprog@nucky ~] $ ls -ltr /tmp/profiling-data/prof20k-bir.*
-rwxrwxrwx 1 lzsystem lzsystem 1659536340 Apr 20 21:20 /tmp/profiling-data/prof20k-bir.raw
-rw-rw-r-- 1 sysprog  sysprog           0 Apr 20 23:09 /tmp/profiling-data/prof20k-bir.afdo.imports
-rw-rw-r-- 1 sysprog  sysprog        1600 Apr 20 23:09 /tmp/profiling-data/prof20k-bir.afdo

I have tried a number of profile data files but the result is always the same. The imports file is size 0 and the gcov file is 1600 bytes. The beginning of the file looks like the following:

0000000: 6164 6367 0100 0000 0000 0000 0000 00aa  adcg............
0000010: 0300 0000 0100 0000 0100 0000 0000 0000  ................
0000020: 0000 00ac 0100 0000 0000 0000 0000 00ae  ................
0000030: 0100 0000 0000 0000 0000 00af 8001 0000  ................
0000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................

The rest of the file is filled with all 0x0.

configure problem

Building on a CentOS6 platform, configure failed to detect the static LLVM libraries I had built. I removed the -ltinfo flag from LLVM_LDFLAGS variable in configure and all went smoothly from that point.

create_gcov: support for gnu_debuglink

create_gcov doesn't seem to find the debug sections from seperate .debug files.
Is support for this planned or can I achieve this without unstripped binaries ?

source_info.h:56:19: error: ‘getBaseDiscriminatorFromDiscriminator’ is not a member of ‘llvm::DILocation’

sys-devel/llvm-3.9.1-r1
sys-devel/gcc-5.4.0-r3

I'm trying to compile the tool under gentoo and facing the following compilation error:

mv -f $depbase.Tpo $depbase.Po
depbase=`echo module_grouper.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT module_grouper.o -MD -MP -MF $depbase.Tpo -c -o module_grouper.o module_grouper.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo profile_creator.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT profile_creator.o -MD -MP -MF $depbase.Tpo -c -o profile_creator.o profile_creator.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo profile_writer.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT profile_writer.o -MD -MP -MF $depbase.Tpo -c -o profile_writer.o profile_writer.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo sample_reader.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT sample_reader.o -MD -MP -MF $depbase.Tpo -c -o sample_reader.o sample_reader.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo source_info.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT source_info.o -MD -MP -MF $depbase.Tpo -c -o source_info.o source_info.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo symbol_map.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT symbol_map.o -MD -MP -MF $depbase.Tpo -c -o symbol_map.o symbol_map.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo profile.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT profile.o -MD -MP -MF $depbase.Tpo -c -o profile.o profile.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo create_gcov.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT create_gcov.o -MD -MP -MF $depbase.Tpo -c -o create_gcov.o create_gcov.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -std=gnu++11 -g -O2 -std=gnu++11   -o create_gcov addr2line.o gcov.o instruction_map.o module_grouper.o profile_creator.o profile_writer.o sample_reader.o source_info.o symbol_map.o profile.o create_gcov.o libquipper.a libglog.a libsymbolize.a libgflags.a -lz -lcrypto 
depbase=`echo profile_reader.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT profile_reader.o -MD -MP -MF $depbase.Tpo -c -o profile_reader.o profile_reader.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo dump_gcov.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT dump_gcov.o -MD -MP -MF $depbase.Tpo -c -o dump_gcov.o dump_gcov.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -std=gnu++11 -g -O2 -std=gnu++11   -o dump_gcov profile_reader.o symbol_map.o module_grouper.o gcov.o dump_gcov.o libglog.a libgflags.a libsymbolize.a -lz -lcrypto 
depbase=`echo sample_merger.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT sample_merger.o -MD -MP -MF $depbase.Tpo -c -o sample_merger.o sample_merger.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -std=gnu++11 -g -O2 -std=gnu++11   -o sample_merger addr2line.o gcov.o instruction_map.o module_grouper.o profile_creator.o profile_writer.o sample_reader.o source_info.o symbol_map.o profile.o sample_merger.o libquipper.a libglog.a libsymbolize.a libgflags.a -lz -lcrypto 
depbase=`echo profile_merger.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT profile_merger.o -MD -MP -MF $depbase.Tpo -c -o profile_merger.o profile_merger.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -std=gnu++11 -g -O2 -std=gnu++11   -o profile_merger addr2line.o gcov.o instruction_map.o module_grouper.o profile_creator.o profile_writer.o sample_reader.o source_info.o symbol_map.o profile.o profile_reader.o profile_merger.o libquipper.a libglog.a libsymbolize.a libgflags.a -lz -lcrypto 
depbase=`echo profile_diff.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT profile_diff.o -MD -MP -MF $depbase.Tpo -c -o profile_diff.o profile_diff.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -std=gnu++11 -g -O2 -std=gnu++11   -o profile_diff addr2line.o gcov.o instruction_map.o module_grouper.o profile_creator.o profile_writer.o sample_reader.o source_info.o symbol_map.o profile.o profile_reader.o profile_diff.o libquipper.a libglog.a libsymbolize.a libgflags.a -lz -lcrypto 
depbase=`echo profile_update.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -std=gnu++11 -g -O2 -std=gnu++11 -MT profile_update.o -MD -MP -MF $depbase.Tpo -c -o profile_update.o profile_update.cc &&\
mv -f $depbase.Tpo $depbase.Po
g++ -std=gnu++11 -g -O2 -std=gnu++11   -o profile_update addr2line.o gcov.o instruction_map.o module_grouper.o profile_creator.o profile_writer.o sample_reader.o source_info.o symbol_map.o profile.o profile_reader.o profile_update.o libquipper.a libglog.a libsymbolize.a libgflags.a -lz -lcrypto 
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-addr2line.o -MD -MP -MF .deps/create_llvm_prof-addr2line.Tpo -c -o create_llvm_prof-addr2line.o `test -f 'addr2line.cc' || echo './'`addr2line.cc
mv -f .deps/create_llvm_prof-addr2line.Tpo .deps/create_llvm_prof-addr2line.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-gcov.o -MD -MP -MF .deps/create_llvm_prof-gcov.Tpo -c -o create_llvm_prof-gcov.o `test -f 'gcov.cc' || echo './'`gcov.cc
mv -f .deps/create_llvm_prof-gcov.Tpo .deps/create_llvm_prof-gcov.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-instruction_map.o -MD -MP -MF .deps/create_llvm_prof-instruction_map.Tpo -c -o create_llvm_prof-instruction_map.o `test -f 'instruction_map.cc' || echo './'`instruction_map.cc
mv -f .deps/create_llvm_prof-instruction_map.Tpo .deps/create_llvm_prof-instruction_map.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-module_grouper.o -MD -MP -MF .deps/create_llvm_prof-module_grouper.Tpo -c -o create_llvm_prof-module_grouper.o `test -f 'module_grouper.cc' || echo './'`module_grouper.cc
mv -f .deps/create_llvm_prof-module_grouper.Tpo .deps/create_llvm_prof-module_grouper.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-profile_creator.o -MD -MP -MF .deps/create_llvm_prof-profile_creator.Tpo -c -o create_llvm_prof-profile_creator.o `test -f 'profile_creator.cc' || echo './'`profile_creator.cc
mv -f .deps/create_llvm_prof-profile_creator.Tpo .deps/create_llvm_prof-profile_creator.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-profile_writer.o -MD -MP -MF .deps/create_llvm_prof-profile_writer.Tpo -c -o create_llvm_prof-profile_writer.o `test -f 'profile_writer.cc' || echo './'`profile_writer.cc
mv -f .deps/create_llvm_prof-profile_writer.Tpo .deps/create_llvm_prof-profile_writer.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-sample_reader.o -MD -MP -MF .deps/create_llvm_prof-sample_reader.Tpo -c -o create_llvm_prof-sample_reader.o `test -f 'sample_reader.cc' || echo './'`sample_reader.cc
mv -f .deps/create_llvm_prof-sample_reader.Tpo .deps/create_llvm_prof-sample_reader.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-source_info.o -MD -MP -MF .deps/create_llvm_prof-source_info.Tpo -c -o create_llvm_prof-source_info.o `test -f 'source_info.cc' || echo './'`source_info.cc
mv -f .deps/create_llvm_prof-source_info.Tpo .deps/create_llvm_prof-source_info.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-symbol_map.o -MD -MP -MF .deps/create_llvm_prof-symbol_map.Tpo -c -o create_llvm_prof-symbol_map.o `test -f 'symbol_map.cc' || echo './'`symbol_map.cc
mv -f .deps/create_llvm_prof-symbol_map.Tpo .deps/create_llvm_prof-symbol_map.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-profile.o -MD -MP -MF .deps/create_llvm_prof-profile.Tpo -c -o create_llvm_prof-profile.o `test -f 'profile.cc' || echo './'`profile.cc
mv -f .deps/create_llvm_prof-profile.Tpo .deps/create_llvm_prof-profile.Po
g++ -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src  -I/usr/include   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -std=gnu++11 -MT create_llvm_prof-llvm_profile_writer.o -MD -MP -MF .deps/create_llvm_prof-llvm_profile_writer.Tpo -c -o create_llvm_prof-llvm_profile_writer.o `test -f 'llvm_profile_writer.cc' || echo './'`llvm_profile_writer.cc
In file included from symbol_map.h:30:0,
                 from profile_writer.h:21,
                 from llvm_profile_writer.h:7,
                 from llvm_profile_writer.cc:30:
source_info.h: In member function ‘uint32 autofdo::SourceInfo::Offset(bool) const’:
source_info.h:56:19: error: ‘getBaseDiscriminatorFromDiscriminator’ is not a member of ‘llvm::DILocation’
                 ? llvm::DILocation::getBaseDiscriminatorFromDiscriminator(
                   ^
source_info.h: In member function ‘uint32 autofdo::SourceInfo::DuplicationFactor() const’:
source_info.h:66:12: error: ‘getDuplicationFactorFromDiscriminator’ is not a member of ‘llvm::DILocation’
     return llvm::DILocation::getDuplicationFactorFromDiscriminator(
            ^
llvm_profile_writer.cc: In member function ‘virtual void autofdo::LLVMProfileBuilder::VisitCallsite(const Callsite&)’:
llvm_profile_writer.cc:100:32: error: no match for ‘operator[]’ (operand types are ‘llvm::sampleprof::FunctionSamples’ and ‘llvm::StringRef’)
           line, discriminator))[CalleeName];
                                ^
make[1]: *** [Makefile:1055: create_llvm_prof-llvm_profile_writer.o] Error 1

Can't build with LLVM support

Build steps:

$ git clone --recursive https://github.com/google/autofdo.git
$ cd autofdo
$ git checkout 0.18 # missing "perf_data.pb.h" on master
$ aclocal -I .; autoheader; autoconf; automake --add-missing -c
$ ./configure --with-llvm=<path-to-llvm6-build>/bin/llvm-config
checking whether we can compile and link with llvm(ProfileData)... no
configure: Using static LLVM libraries.
configure: WARNING: could not detect the LLVM libraries. Support for LLVM profiles disabled.
$ make -j$(nproc)

Runtime error:

ERROR: LLVM support was not enabled in this configuration.
Please configure and rebuild with:

$ ./configure --with-llvm=<path-to-llvm-config>

LLVM config:

$ ./bin/llvm-config --libs profiledata
-lLLVMProfileData -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle

Thanks for looking into it.

Compilation error

Can't compile, this message is displayed when running make:

In file included from chromiumos-wide-profiling/address_mapper.cc:5:0:
./chromiumos-wide-profiling/address_mapper.h:10:16: fatal error: list: No such file or directory
compilation terminated.
Makefile:770: recipe for target 'chromiumos-wide-profiling/address_mapper.o' failed
make: *** [chromiumos-wide-profiling/address_mapper.o] Error 1

create_llvm_prof fails with "Error reading profile"

I've never used FDO before, so it's possible that I'm just doing something wrong. This is what happened:

First I built my program, including the -g option. Then I ran it under perf record -b as specified in the Clang User's Manual, like this:

perf record -F 96 -b /tmp/kaladont_bin/slim_false_clang nomi < ./data/connected_partitioned_word_lists/list_020468_000001.txt

I stopped the program (and perf) after letting it run for about 5 minutes, and I stopped it with a single control-C on the terminal that was running it. (This is the way that the program that I'm trying to optimize is supposed to be stopped, so I hope it's not an issue.) Perf then reported this, which seems OK:

[ perf record: Woken up 117 times to write data ]
[ perf record: Captured and wrote 29.252 MB perf.data (37557 samples) ]

Afterwards I copied create_llvm_prof to a directory in my PATH, and ran it like this:

create_llvm_prof --binary=/tmp/kaladont_bin/slim_false_clang --out=slim_false_clang.prof

Sadly, create_llvm_prof reported several errors, without producing a slim_false_clang.prof file:

[WARNING:../third_party/perf_data_converter/src/quipper/perf_reader.cc:1244] Skipping 104 bytes of metadata: HEADER_CPU_TOPOLOGY
[ERROR:../third_party/perf_data_converter/src/quipper/perf_data_utils.cc:488] Non-zero remaining event size 24 received for event PERF_RECORD_TIME_CONV with no varaible payload event data or sample info data
[ERROR:../third_party/perf_data_converter/src/quipper/perf_reader.cc:1092] Couldn't get variable payload size for event PERF_RECORD_TIME_CONV
[ERROR:../third_party/perf_data_converter/src/quipper/perf_reader.cc:1020] Couldn't read event PERF_RECORD_TIME_CONV
WARNING: Logging before InitGoogleLogging() is written to STDERR
E20210507 21:47:24.928634 243508 profile_creator.cc:176] Error reading profile.

Software versions:

  • autofdo is git cloned a few hours ago: 089a864

  • Clang and LLVM are at 11.1.0

  • Perf is version 5.12

  • Linux is version 5.11.16

Cannot read compilation info from binary under GCC 7.3

Thanks firstly, any help are appreciated.

When I used GCC 7.3 to build mysql, I found this warning

"W0412 18:42:05.956995 92904 module_grouper.cc:60] Cannot read compilation info from binary. Please use -frecord-compilation-info-in-elf when building the binary"

At the same time, create_gcov failed to convert perf.data to gcov. The gcov file is empty.

what's more, I can't find "-frecord-compilation-info-in-elf" in GCC 7.3 manual, I google it and it seems the compile flag can only be used by Google-gcc.

How to resolve the problem?

BTW: But I successfully convert the perf.data to gcov of the sort.c example under GCC 7.3 (https://gcc.gnu.org/wiki/AutoFDO/Tutorial)

profile_merger: Check failed: count_threshold_ == 0

Hi,

I'm trying to use binary profile_merger from your repository. I have collected 3 perf.data profiles for the same executable binary file that was run with 3 different input data. Then for each of them the create_gcov utility was run. Now I'm having 3 gcov data files after that and want to use them all during the compilation. How can it be done?
I have tried to use profile_merger mreger for that:

bash-4.3# /home/abuild/autofdo/build/profile_merger gcov.data.22760 gcov.data.22788 gcov.data.22815 --gcov_version=1
F0529 10:47:36.755566 31752 symbol_map.cc:193] Check failed: count_threshold_ == 0 (140733849599456 vs. 0)
*** Check failure stack trace: ***
@ 0x449cad google::LogMessage::Flush()
@ 0x44c6c9 google::LogMessageFatal::~LogMessageFatal()
@ 0x427f6e autofdo::SymbolMap::CalculateThreshold()
@ 0x408ae9 main
@ 0x7f45ffbfea48 __libc_start_main
@ 0x409e89 _start
@ (nil) (unknown)
Aborted (core dumped)

Is it a bug or I'm doing something wrong?

Thanks...

profile_creator.cc:138] Error reading profile.

when I record a profile via:

➜  llvm-project git:(master) ✗ perf record -g -F3200 ./foo                                        
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.002 MB perf.data (12 samples) ]
➜  llvm-project git:(master) ✗ /android1/autofdo/create_llvm_prof --binary=foo --profile=perf.data --out=perf.proto
E0623 16:13:15.980063 116888 perf_parser.cc:270] Mapped 25% of samples, expected at least 95%

(no problem) but

➜  llvm-project git:(master) ✗ perf record --call-graph dwarf -F32000 ./foo    
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.567 MB perf.data (70 samples) ]
➜  llvm-project git:(master) ✗ /android1/autofdo/create_llvm_prof --binary=foo --profile=perf.data --out=perf.proto
E0623 16:12:40.868065 116624 sample_info_reader.cc:272] PERF_SAMPLE_REGS_USER is not yet supported.
E0623 16:12:40.868284 116624 sample_info_reader.cc:513] Read/skipped 56 bytes, expected 8440 bytes.
E0623 16:12:40.868356 116624 profile_creator.cc:138] Error reading profile.

It seems that --call-graph dwarf doesn't work, but -g does? (DWARF debug info vs frame pointers for unwinding?)

make cannot configure protobuf and fails with aclocal: warning: couldn't open directory 'm4': No such file or directory

Hi all,
I'm on ubuntu bionic. I followed you readme and did:
sudo apt-get -y install autoconf automake git libelf-dev libssl-dev pkg-config
git clone --recursive https://github.com/google/autofdo.git
cd autofdo/
aclocal -I .; autoheader; autoconf; automake --add-missing -c
./configure --with-llvm=$(which llvm-config-5.0)
make

then I get the error:
make all-am
make[1]: Entering directory '/home/harubyy/profiling/autofdo'
echo "[AUTOGEN] Preparing protobuf"
[AUTOGEN] Preparing protobuf
(cd third_party/protobuf ; autoreconf -f -i -Wall,no-obsolete)
aclocal: warning: couldn't open directory 'm4': No such file or directory
configure.ac:30: error: possibly undefined macro: AC_PROG_LIBTOOL
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1
Makefile:1590: recipe for target 'third_party/protobuf/configure' failed
make[1]: *** [third_party/protobuf/configure] Error 1
make[1]: Leaving directory '/home/harubyy/profiling/autofdo'
Makefile:567: recipe for target 'all' failed
make: *** [all] Error 2

I'm adding my config.log
config.log

m4 directory exits both at the top directory and in third_party/protobuf
Thanks!

create_gcov: Check failed: attr_size <= sizeof(perf_event_attr) (104 vs. 96)

create_gcov --binary= --profile=perf.data --gcov=perf.ado

F0427 14:19:20.557157 22428 perf_reader.cc:1614] Check failed: attr_size <= sizeof(perf_event_attr) (104 vs. 96)
*** Check failure stack trace: ***
    @           0x4424f8  google::LogMessage::Flush()
    @           0x445459  google::LogMessageFatal::~LogMessageFatal()
    @           0x4317c0  quipper::PerfReader::ReadEventDescMetadata()
    @           0x432e93  quipper::PerfReader::ReadMetadata()
    @           0x4339b6  quipper::PerfReader::ReadFromPointer()
    @           0x433a9d  quipper::PerfReader::ReadFile()
    @           0x41d8c8  autofdo::PerfDataSampleReader::Append()
    @           0x41ced3  autofdo::SampleReader::ReadAndSetTotalCount()
    @           0x417451  autofdo::ProfileCreator::ReadSample()
    @           0x418572  autofdo::ProfileCreator::CreateProfile()
    @           0x4080c5  main
    @     0x7f6a3ce0e790  __libc_start_main
    @           0x4093e9  _start
    @              (nil)  (unknown)

This is on Linux 4.0 and gcc 5.1.0.

eb596e885b0 breaks build: "'perf_data.pb.h' file not found"

Here's an excerpt from ninja's redirected stdout and stderr:

[3/238] Building CXX object CMakeFiles/create_gcov_lib.dir/sample_reader.cc.o
FAILED: CMakeFiles/create_gcov_lib.dir/sample_reader.cc.o 
/usr/bin/clang++  -I../ -I../third_party/glog/src -I../third_party/abseil -I../third_party/perf_data_converter/src -I../third_party/perf_data_converter/src/quipper -I../util -I. -Ithird_party/glog -g -march=native -O3 -std=gnu++17 -MD -MT CMakeFiles/create_gcov_lib.dir/sample_reader.cc.o -MF CMakeFiles/create_gcov_lib.dir/sample_reader.cc.o.d -o CMakeFiles/create_gcov_lib.dir/sample_reader.cc.o -c ../sample_reader.cc
In file included from ../sample_reader.cc:6:
In file included from ../sample_reader.h:18:
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:
../third_party/perf_data_converter/src/quipper/base/macros.h:8:9: warning: 'DISALLOW_COPY_AND_ASSIGN' macro redefined [-Wmacro-redefined]
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
        ^
../base/macros.h:114:9: note: previous definition is here
#define DISALLOW_COPY_AND_ASSIGN(TypeName) \
        ^
In file included from ../sample_reader.cc:6:
In file included from ../sample_reader.h:18:
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:18:
../third_party/perf_data_converter/src/quipper/base/macros.h:12:9: warning: 'arraysize' macro redefined [-Wmacro-redefined]
#define arraysize(x) (sizeof(x) / sizeof(*x))
        ^
../base/macros.h:162:9: note: previous definition is here
#define arraysize(array) (sizeof(ArraySizeHelper(array)))
        ^
In file included from ../sample_reader.cc:6:
In file included from ../sample_reader.h:18:
In file included from ../third_party/perf_data_converter/src/quipper/perf_parser.h:21:
../third_party/perf_data_converter/src/quipper/compat/proto.h:14:10: fatal error: 'perf_data.pb.h' file not found
#include "perf_data.pb.h"
         ^~~~~~~~~~~~~~~~
2 warnings and 1 error generated.

create_gcov crashes when perf.data file is large in size

create_gcov is able to create the gcov files from perf.data successfully, when perf.data is few gbs in size (tested upto 2 GBs)
But as I run perf commads for a longer time to profile my code, the perf.data size is 9GBs or so, the create_gcov utility crashes, just mentioning killed.

create_gcov --logtostderr --binary=/a/bin/ghost --profile=perf.data --gcov=test.gcov -gcov_version=1
I0807 13:58:17.993311 7642 perf_reader.cc:670] Perf data is in normal format.
I0807 13:58:51.462221 7642 perf_reader.cc:1298] Number of events stored: 22894985
I0807 13:58:51.480528 7642 perf_reader.cc:1370] Unsupported metadata type, skipping: 16
Killed

Unable to read build ID header

I'm useing create_llvm_prof to use with PGO.
I ran my application which is built with -g under perf record -b which generates perf.data
The perf.data is 366 MB and I can run perf report on it which tells me that the file is okay
While reading the perf.data, create_llvm_prof quits with following error:
E0512 19:20:31.186863 8230 data_reader.cc:15] Unable to read build ID header. Requested 8 bytes, 18424226071644201007 bytes remaining.
E0512 19:20:31.205049 8230 profile_creator.cc:78] Error reading profile.

When I debug, perf_reader.cc:1325, section_iter->offset has a value of 0x50000100000001 which is odd section_iter->size is set to 0xffffffff
type is : HEADER_BUILD_ID

when I continue:

In buffer_reader.cc:12, size_ is 0xffffffff since offset_ is set to the huge value above, the if condition is met and false is returned which causes the error.

Check failed: sample_type_ == attr->sample_type (65671 vs. 67975) Event type sample format does not match sample format of other event type.

Hi, I've narrowed down $SUBJECT down to whether --strip is specified in the perf inject command prior to create_gcov invocation.

$ gcc -g -O3 -o simple simple.c
$ perf version  # it's today's upstream acme's perf/urgent branch
perf version 4.13.rc5.g090657c
$ cat ~/.perfconfig 
[intel-pt]
mispred-all = true
$ perf record -e intel_pt//u ./simple
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 0.198 MB perf.data ]
$ perf inject -i perf.data -o inj --itrace=i100usle
$ create_gcov --binary=simple --profile=inj  --gcov=simple.gcov  -gcov_version=1
F0927 16:58:43.668524 29635 perf_reader.cc:1183] Check failed: sample_type_ == attr->sample_type (65671 vs. 67975) Event type sample format does not match sample format of other event type.
*** Check failure stack trace: ***
    @     0x55bf14afef51  google::LogMessage::Flush()
    @     0x55bf14b02109  google::LogMessageFatal::~LogMessageFatal()
    @     0x55bf14ae991a  quipper::PerfReader::ReadEventAttr()
    @     0x55bf14aeb2c9  quipper::PerfReader::ReadEventDescMetadata()
    @     0x55bf14aec840  quipper::PerfReader::ReadMetadata()
    @     0x55bf14aed44f  quipper::PerfReader::ReadFromData()
    @     0x55bf14aed6f1  quipper::PerfReader::ReadFile()
    @     0x55bf14ad796f  autofdo::PerfDataSampleReader::Append()
    @     0x55bf14ad6dc3  autofdo::SampleReader::ReadAndSetTotalCount()
    @     0x55bf14acf4b5  autofdo::ProfileCreator::ReadSample()
    @     0x55bf14ad0412  autofdo::ProfileCreator::CreateProfile()
    @     0x55bf14ac0410  main
    @     0x7fcbaf1403f1  (unknown)
    @     0x55bf14ac1c4a  _start
    @              (nil)  (unknown)
Aborted (core dumped)

The check doesn't fail when --strip is appended to the perf inject command.
stripped-and-not-stripped-perf-data-files.zip

Build failure when targeting LLVM 11.0

MacOS has moved to Clang 11, and for create_llvm_prof to target it we need to build autofdo --with-llvm=$(which llvm-config-11). The configuration phase passes, but building the create_llvm_prof binary fail both with gcc-9.x and clang-11 due to a missing operator[] in llvm_profile_writer.cc.

When building with GCC-9:

g++-9 -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src -I./third_party/perf_data_converter/src/quipper -I./third_party/protobuf/src  -I/usr/lib/llvm-11/include -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DCREATE_LLVM_PROF -g -O2 -MT create_llvm_prof-llvm_profile_writer.o -MD -MP -MF .deps/create_llvm_prof-llvm_profile_writer.Tpo -c -o create_llvm_prof-llvm_profile_writer.o `test -f 'llvm_profile_writer.cc' || echo './'`llvm_profile_writer.cc
llvm_profile_writer.cc: In member function ‘virtual void autofdo::LLVMProfileBuilder::VisitCallsite(const Callsite&)’:
llvm_profile_writer.cc:100:32: error: no match for ‘operator[]’ (operand types are ‘llvm::sampleprof::FunctionSamplesMap’ {aka ‘std::map<std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples, std::less<void> >’} and ‘llvm::StringRef’)
  100 |           line, discriminator))[CalleeName];
      |                                ^
In file included from /usr/include/c++/9/map:61,
                 from llvm_profile_writer.cc:22:
/usr/include/c++/9/bits/stl_map.h:490:7: note: candidate: ‘std::map<_Key, _Tp, _Compare, _Alloc>::mapped_type& std::map<_Key, _Tp, _Compare, _Alloc>::operator[](const key_type&) [with _Key = std::__cxx11::basic_string<char>; _Tp = llvm::sampleprof::FunctionSamples; _Compare = std::less<void>; _Alloc = std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >; std::map<_Key, _Tp, _Compare, _Alloc>::mapped_type = llvm::sampleprof::FunctionSamples; std::map<_Key, _Tp, _Compare, _Alloc>::key_type = std::__cxx11::basic_string<char>]’
  490 |       operator[](const key_type& __k)
      |       ^~~~~~~~
/usr/include/c++/9/bits/stl_map.h:490:34: note:   no known conversion for argument 1 from ‘llvm::StringRef’ to ‘const key_type&’ {aka ‘const std::__cxx11::basic_string<char>&’}
  490 |       operator[](const key_type& __k)
      |                  ~~~~~~~~~~~~~~~~^~~
/usr/include/c++/9/bits/stl_map.h:510:7: note: candidate: ‘std::map<_Key, _Tp, _Compare, _Alloc>::mapped_type& std::map<_Key, _Tp, _Compare, _Alloc>::operator[](std::map<_Key, _Tp, _Compare, _Alloc>::key_type&&) [with _Key = std::__cxx11::basic_string<char>; _Tp = llvm::sampleprof::FunctionSamples; _Compare = std::less<void>; _Alloc = std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> >; std::map<_Key, _Tp, _Compare, _Alloc>::mapped_type = llvm::sampleprof::FunctionSamples; std::map<_Key, _Tp, _Compare, _Alloc>::key_type = std::__cxx11::basic_string<char>]’
  510 |       operator[](key_type&& __k)
      |       ^~~~~~~~
/usr/include/c++/9/bits/stl_map.h:510:29: note:   no known conversion for argument 1 from ‘llvm::StringRef’ to ‘std::map<std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples, std::less<void> >::key_type&&’ {aka ‘std::__cxx11::basic_string<char>&&’}
  510 |       operator[](key_type&& __k)
      |                  ~~~~~~~~~~~^~~
make[1]: *** [Makefile:1234: create_llvm_prof-llvm_profile_writer.o] Error 1

When building with Clang-11:

clang++-11 -DHAVE_CONFIG_H -I. -I./glog/src  -I. -I./glog/src -I./third_party/perf_data_converter/src/quipper -I./third_party/protobuf/src  -I/usr/lib/llvm-11/include -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DCREATE_LLVM_PROF -g -O2 -MT create_llvm_prof-llvm_profile_writer.o -MD -MP -MF .deps/create_llvm_prof-llvm_profile_writer.Tpo -c -o create_llvm_prof-llvm_profile_writer.o `test -f 'llvm_profile_writer.cc' || echo './'`llvm_profile_writer.cc
In file included from llvm_profile_writer.cc:30:
In file included from ./llvm_profile_writer.h:7:
./profile_writer.h:186:8: warning: 'VisitCallsite' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
  void VisitCallsite(const Callsite &callsite) {
       ^
./profile_writer.h:152:16: note: overridden virtual function is here
  virtual void VisitCallsite(const Callsite &offset) {}
               ^
llvm_profile_writer.cc:100:32: error: no viable overloaded operator[] for type 'llvm::sampleprof::FunctionSamplesMap' (aka 'map<basic_string<char>, llvm::sampleprof::FunctionSamples, less<> >')
          line, discriminator))[CalleeName];
          ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_map.h:490:7: note: candidate function not viable: no known conversion from 'llvm::StringRef' to 'const std::map<std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::key_type' (aka 'const std::__cxx11::basic_string<char>') for 1st argument
      operator[](const key_type& __k)
      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_map.h:510:7: note: candidate function not viable: no known conversion from 'llvm::StringRef' to 'std::map<std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples, std::less<void>, std::allocator<std::pair<const std::__cxx11::basic_string<char>, llvm::sampleprof::FunctionSamples> > >::key_type' (aka 'std::__cxx11::basic_string<char>') for 1st argument
      operator[](key_type&& __k)
      ^
1 warning and 1 error generated.

cmake warning regarding gflags

$ cmake -G Ninja  -DCMAKE_INSTALL_PREFIX=. -DLLVM_PATH=/android0/llvm-project/llvm/build ..
...
CMake Warning at third_party/glog/CMakeLists.txt:63 (find_package):
  By not providing "Findgflags.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "gflags", but
  CMake did not find one.

  Could not find a package configuration file provided by "gflags" (requested
  version 2.2.0) with any of the following names:

    gflagsConfig.cmake
    gflags-config.cmake

  Add the installation prefix of "gflags" to CMAKE_PREFIX_PATH or set
  "gflags_DIR" to a directory containing one of the above files.  If "gflags"
  provides a separate development package or SDK, be sure it has been
  installed.

cannot find input file: `Makefile.in'

After sync with the latest code and patch the pull request "249bd5f"

when build protobuffer, it will report "cannot find input file: `Makefile.in'"

checking for PTHREAD_PRIO_INHERIT... yes
checking the location of hash_map... <unordered_map>
checking for library containing sched_yield... none required
checking whether g++ supports C++11 features by default... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: error: cannot find input file: `Makefile.in'

create_gcov crash

Could anyone help to decode this error message when I attempt to run create_gcov

$create_gcov --binary=which athena.py --profile=./perf.data --gcov=basic.gcov -gcov_version=1
E0721 13:56:07.792387 23417 address_mapper.cc:154] Could not find space to map addr=7f8c5b8dc000 with size a000
F0721 13:56:07.793525 23417 perf_parser.cc:151] Check failed: MapMmapEvent(&event.mmap2, i) Unable to map MMAP2 event!
*** Check failure stack trace: ***
@ 0x441224 google::LogMessage::Flush()
@ 0x443fd9 google::LogMessageFatal::~LogMessageFatal()
@ 0x43a4af quipper::PerfParser::ProcessEvents()
@ 0x43aaf4 quipper::PerfParser::ParseRawEvents()
@ 0x41c590 autofdo::PerfDataSampleReader::Append()
@ 0x41baf3 autofdo::SampleReader::ReadAndSetTotalCount()
@ 0x414ef8 autofdo::ProfileCreator::ReadSample()
@ 0x415cf2 autofdo::ProfileCreator::CreateProfile()
@ 0x40816e main
@ 0x7f8464db3b35 __libc_start_main
@ 0x4095d2 (unknown)
@ (nil) (unknown)
Aborted (core dumped)

Unable to read build ID header

I'm useing create_llvm_prof to use with PGO.
I ran my application which is built with -g under perf record -b which generates perf.data
The perf.data is 366 MB and I can run perf report on it which tells me that the file is okay
While reading the perf.data, create_llvm_prof quits with following error:
E0512 19:20:31.186863 8230 data_reader.cc:15] Unable to read build ID header. Requested 8 bytes, 18424226071644201007 bytes remaining.
E0512 19:20:31.205049 8230 profile_creator.cc:78] Error reading profile.

When I debug:

breakpoint 1 : perf_reader.cc:1325 [data->SeekSet(section_iter->offset);]
breakpoint 2 : buffer_reader.cc:12 [if (offset_ + size > size_)]

breakpoint 1
section_iter->offset == 0x50000100000001
section_iter->size == 0xffffffff

breakpoint 2:
size_ == 0x15d7e030
size == 8
offset_ == 0x50000100000001

backtrace at breakpoint 2:
#0 quipper::BufferReader::ReadData (this=0x7fffffffd828, size=8, dest=0x7fffffffd498) at chromiumos-wide-profiling/buffer_reader.cc:12
#1 0x00000000005d917b in quipper::DataReader::ReadDataValue (this=0x7fffffffd828, size=8, , dest=0x7fffffffd498) at chromiumos-wide-profiling/data_reader.cc:13
#2 0x00000000005ab4a0 in quipper::PerfReader::ReadBuildIDMetadata (this=0x7fffffffdae8, data=0x7fffffffd828, size=4294967295) at chromiumos-wide-profiling/perf_reader.cc:1384
#3 0x00000000005a2a24 in quipper::PerfReader::ReadMetadata (this=0x7fffffffdae8, data=0x7fffffffd828) at chromiumos-wide-profiling/perf_reader.cc:1334
#4 0x00000000005a1842 in quipper::PerfReader::ReadFromData (this=0x7fffffffdae8, data=0x7fffffffd828) at chromiumos-wide-profiling/perf_reader.cc:706
#5 0x00000000005a116d in quipper::PerfReader::ReadFromPointer (this=0x7fffffffdae8, data=0x7fffe0907010 "PERFILE2h", size=366469168) at chromiumos-wide-profiling/perf_reader.cc:657
#6 0x00000000005a1121 in quipper::PerfReader::ReadFromVector (this=0x7fffffffdae8, ) at chromiumos-wide-profiling/perf_reader.cc:648
#7 0x00000000005a1092 in quipper::PerfReader::ReadFile (this=0x7fffffffdae8, ) at chromiumos-wide-profiling/perf_reader.cc:644
#8 0x00000000004379f1 in autofdo::PerfDataSampleReader::Append (this=0x9ccbc0, ) at sample_reader.cc:203
#9 0x0000000000436b36 in autofdo::FileSampleReader::Read (this=0x9ccbc0) at sample_reader.cc:89
#10 0x00000000004369ca in autofdo::SampleReader::ReadAndSetTotalCount (this=0x9ccbc0) at sample_reader.cc:72
#11 0x0000000000429fba in autofdo::ProfileCreator::ReadSample (this=0x7fffffffe140, , ) at profile_creator.cc:77
#12 0x000000000042a0f4 in autofdo::ProfileCreator::CreateProfile (this=0x7fffffffe140, , , writer=0x9c4dd0, ) at profile_creator.cc:45
#13 0x000000000045e88c in main (argc=1, argv=0x7fffffffe2d8) at create_llvm_prof.cc:78

F0115 17:59:41.405607 17526 nonoverlapping_range_map.h:122] Check failed: predecessor->first.second <= low

my command is ./create_gcov --binary=./bs.se --profile=./perf.data --gcov=bs.se.gcov -gcov_version=1
F0115 17:59:41.405607 17526 nonoverlapping_range_map.h:122] Check failed: predecessor->first.second <= low
*** Check failure stack trace: ***
@ 0x4484d3 google::LogMessage::SendToLog()
@ 0x446461 google::LogMessage::Flush()
@ 0x448e39 google::LogMessageFatal::~LogMessageFatal()
@ 0x4559f5 autofdo::NonOverlappingRangeMap<>::InsertRange()
@ 0x453767 autofdo::InlineStackHandler::PopulateSubprogramsByAddress()
@ 0x40a2dc autofdo::Google3Addr2line::Prepare()
@ 0x40986e autofdo::Addr2line::CreateWithSampledFunctions()
@ 0x415b78 autofdo::ProfileCreator::ComputeProfile()
@ 0x415fcc autofdo::ProfileCreator::CreateProfileFromSample()
@ 0x408158 main
@ 0x7fb1e9febbd5 __libc_start_main
@ 0x4093dd (unknown)
@ (nil) (unknown)
Aborted

aclocal-1.14: command not found

On my Fedora 24 machine on attempt to run make after configure I'm seeing this:

make
CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh .../missing aclocal-1.14 -I m4
.../missing: line 81: aclocal-1.14: command not found
WARNING: 'aclocal-1.14' is missing on your system.
         You should only need it if you modified 'acinclude.m4' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'aclocal' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>
Makefile:529: recipe for target 'aclocal.m4' failed
make: *** [aclocal.m4] Error 127

Even though I do have aclocal installed but of a bit newer version, see:

aclocal --version
aclocal (GNU automake) 1.15
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Tom Tromey <[email protected]>
       and Alexandre Duret-Lutz <[email protected]>.

Looks like that's because of this line in configure script https://github.com/google/autofdo/blob/master/configure#L2449

am__api_version='1.14'

Solution to this is as simple as re-creation of configure script right on build machine with execution of autoconf.

Maybe it worth adding this information to some documentation of AutoFDO.

Add unittests

When we open-sourced this tool, the tests are omitted because it'll introduce another dependency to gtest. But it's worth the effort to add the test to this package too to track regression.

create_gcov creates 1600 bytes file for any program

https://gcc.gnu.org/ml/gcc/2015-04/msg00174.html
https://gcc.gnu.org/ml/gcc/2015-04/msg00249.html

Does anybody know with which options should the perf be executed so that to collect appropriate data for the autofdo converter?
I obtain the same data for different programs, and it seems to be empty (1600 Bytes).
They have the same md5sum for different programs:

Data for simple program with 30 lines of code:

$ md5sum ytest.gcov
d85481c9154aa606ce4893b64fe109e7 ytest.gcov

Data for program of 3D Delaunay triangulation construction of

1000000 points.
$ md5sum experimentCGAL_convexHullDynamic.gcov
d85481c9154aa606ce4893b64fe109e7 experimentCGAL_convexHullDynamic.gcov

We tried to collect perf data using option --call-graph fp but it does not help: the output gcov data is still the same.
Sometimes create_gcov reports the following error:
E0421 13:10:37.125629 8732 perf_parser.cc:209] Mapped 50% of samples, expected at least 95%

But it does not mean that there are not enough samples collected in the profile, because 99% of samples are mapped in the case of very simple program (with 1 function).
I try to find working case for more than a week but did not suceeded.

Can anybody show me that create_gcov works at least for one case?

Build failure on master: No rule to make target 'third_party/perf_data_converter/src/quipper/address_mapper.cc'

Steps followed:

$ sudo apt-get -y install libtool autoconf automake git libelf-dev libssl-dev pkg-config
$ aclocal -I .; autoheader; autoconf; automake --add-missing -c
$ ./configure
$ make

 *** No rule to make target 'third_party/perf_data_converter/src/quipper/address_mapper.cc', needed by 'third_party/perf_data_converter/src/quipper/address_mapper.o'.  Stop.

Similar error has been reported here: https://aur.archlinux.org/packages/autofdo-git/

example links

Could the README please link to https://gcc.gnu.org/wiki/AutoFDO/Tutorial

Also, I have not found a more complex example, for example, using AutoFDO to produce an optimized Linux kernel. (Is this even possible?). If there is a more complex example published, where source spans multiple directories, that would be good to link to as well.

incorrect assumption that branch record will set either predicted or mispred

In perf-parser.cc:

    // Either predicted or mispredicted, not both. But don't use a CHECK here,
    // just exit gracefully because it's a minor issue.
    if (entry.flags.predicted == entry.flags.mispred) {
      LOG(ERROR) << "Branch stack entry predicted and mispred flags "
                 << "both have value " << entry.flags.mispred;
      return false;
    }

It is valid for neither flag to be set, when we don't know if the branch was predicted. This is the case when decoding Intel PT.

perf has had a workaround for this for some time (by means of a non default command-line option): https://patchwork.kernel.org/patch/7264751/

But it would be cleaner to fix it in AutoFDO.

Need to support hw_idx field in branch stack samples

Linux commit 42bbabed09ce6208026648a71a45b4394c74585a ("perf tools:
Add hw_idx in struct branch_stack") changed the format of branch stacks in perf samples, by adding a new field. When samples use this new format, a new flag PERF_SAMPLE_BRANCH_HW_INDEX is set in branch_sample_type in the event. Anything reading perf.data samples needs to test this flag and be prepared to consume the extra field, something like this:

diff --git a/src/quipper/kernel/perf_event.h b/src/quipper/kernel/perf_event.h
index a28d22d..af472ec 100644
--- a/src/quipper/kernel/perf_event.h
+++ b/src/quipper/kernel/perf_event.h
@@ -165,7 +165,9 @@ enum perf_branch_sample_type {
   PERF_SAMPLE_BRANCH_NO_TX = 1U << 9,      /* not in transaction */
   PERF_SAMPLE_BRANCH_COND = 1U << 10,      /* conditional branches */

-  PERF_SAMPLE_BRANCH_MAX = 1U << 11, /* non-ABI */
+  PERF_SAMPLE_BRANCH_HW_INDEX = 1U << 17,  /* sample contains hw_idx field */
+
+  PERF_SAMPLE_BRANCH_MAX = 1U << 18, /* non-ABI */
 };

 #define PERF_SAMPLE_BRANCH_PLM_ALL \
diff --git a/src/quipper/sample_info_reader.cc b/src/quipper/sample_info_reader.cc
index 42c17f3..dfb25f0 100644
--- a/src/quipper/sample_info_reader.cc
+++ b/src/quipper/sample_info_reader.cc
@@ -113,9 +113,10 @@ void ReadRawData(DataReader* reader, struct perf_sample* sample) {
   reader->SeekSet(reader_offset);
 }

-// Read call chain info from perf data.  Corresponds to sample format type
-// PERF_SAMPLE_CALLCHAIN.
-void ReadBranchStack(DataReader* reader, struct perf_sample* sample) {
+// Read branch stack info from perf data.  Corresponds to sample format type
+// PERF_SAMPLE_BRANCH_STACK.
+void ReadBranchStack(DataReader* reader, uint64_t branch_sample_type,
+                     struct perf_sample* sample) {
   // Make sure there is no existing allocated memory in
   // |sample->branch_stack|.
   CHECK_EQ(static_cast<void*>(NULL), sample->branch_stack);
@@ -124,6 +125,11 @@ void ReadBranchStack(DataReader* reader, struct perf_sample* sample) {
   // branch_entry structs.
   uint64_t branch_stack_size = 0;
   reader->ReadUint64(&branch_stack_size);
+  if (branch_sample_type & PERF_SAMPLE_BRANCH_HW_INDEX) {
+    // The branch stack may contain an extra field, which we ignore.
+    uint64_t hw_idx;
+    reader->ReadUint64(&hw_idx);
+  }

   struct branch_stack* branch_stack = reinterpret_cast<struct branch_stack*>(
       new uint8_t[sizeof(uint64_t) +
@@ -263,7 +269,7 @@ size_t ReadPerfSampleFromData(const event_t& event,
   // { u64                   nr;
   //   { u64 from, to, flags } lbr[nr];} && PERF_SAMPLE_BRANCH_STACK
   if (sample_fields & PERF_SAMPLE_BRANCH_STACK) {
-    ReadBranchStack(&reader, sample);
+    ReadBranchStack(&reader, attr.branch_sample_type, sample);
   }

   // { u64                   abi; # enum perf_sample_regs_abi

"perf_serializer.cc:234] Unknown event type: 19" with latest Linux tree

Running create_llvm_prof on a profile generated on Linux 5.4.0:

perf record -o cycles.perf.data -e cycles -- ...
create_llvm_prof -format=text -binary=./hotcold -profile=cycles.perf.data -out=cycles.llvm.prof

fails with

E0709 15:46:56.726442  4116 perf_serializer.cc:234] Unknown event type: 19
E0709 15:46:56.726524  4116 perf_serializer.cc:234] Unknown event type: 19
E0709 15:46:56.728390  4116 perf_parser.cc:248] Unknown event type: 19

This is PERF_RECORD_CGROUP, which is now generated by the kernel. It shouldn't be a problem to ignore it when consuming a perf.data file. More generally, all create_llvm_prof needs to do is consume the perf data - couldn't it warn about any unknown types and continue?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.