Trying to build pretty much as is shown in https://github.com/google/llvm-propeller/blob/plo-dev/Propeller_RFC.pdf

llvm-propeller/llvm/lib/Passes/StandardInstrumentations.cpp:341:46: error: no matching function for call to ‘llvm::IRChangePrinter::~IRChangePrinter()’
   ChangePrinter<std::string>::~ChangePrinter();

possibly related: https://reviews.llvm.org/D87000?id=302664

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

First 10 Artifacts Found

• clang-bolt-experiment/llvm-bolt
• clang/test/Driver/Inputs/in.so
• clang/test/Driver/Inputs/montavista_i686_tree/usr/lib/gcc/i686-montavista-linux/4.2.0/crtbegin.o
• clang/test/Driver/Inputs/ubuntu_13.04_multiarch_tree/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/crtbegin.o
• lld/test/COFF/Inputs/alias-implib.lib
• lld/test/COFF/Inputs/except_handler3.lib
• lld/test/COFF/Inputs/gnu-weak.o
• lld/test/COFF/Inputs/gnu-weak2.o
• lld/test/COFF/Inputs/id.res.o
• lld/test/COFF/Inputs/imports-mangle.lib
• Run a Scorecards scan to see full list.

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.

Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

BOLT is in the process of being upstreamed into LLVM, therefore you could unify your effort in order to reach a synergy and not duplicate effort.
https://www.phoronix.com/scan.php?page=news_item&px=BOLT-Optimizer-RFC-LLVM

Hi,
I'm not able to observe the performance benefit due to propeller toolchain for the included test program (main.cc, callee.cc). Followed the steps given in Propeller_RFC.pdf.

High level observations:

1. Elapsed time doesn't show any improvement.
2. cycles and instruction, branch mispredicts are almost same
3. overall cache-misses are lower but L1-icache-load-misses are similar

$ time ./a.out.orig.labels 1000000000 2 >& /dev/null
real 0m21.094s
user 0m20.489s
sys 0m0.604s

$ time ./a.out.labels 1000000000 2 >& /dev/null
real 0m20.357s
user 0m19.908s
sys 0m0.448s

Elapsed time varies from 1 to 5%.

Perf data

$ perf stat -e cycles,instructions,cache-misses,L1-icache-load-misses,br_misp_retired.all_branches,br_inst_retired.all_branches,icache_64b.iftag_stall ./a.out.o
rig.labels 1000000000 1> /dev/null

Performance counter stats for './a.out.orig.labels 1000000000':

80,231,347,233      cycles                                                        (66.67%)

243,314,361,618 instructions # 3.03 insn per cycle (83.33%)
22,522 cache-misses (83.33%)
2,644,077 L1-icache-load-misses (83.33%)
20,400,061 br_misp_retired.all_branches (83.33%)
53,442,616,374 br_inst_retired.all_branches (83.34%)
68,554,744 icache_64b.iftag_stall (57.14%)

  21.191516400 seconds time elapsed

Optimized binary

$ perf stat -e cycles,instructions,cache-misses,L1-icache-load-misses,br_misp_retired.all_branches,br_inst_retired.all_branches,icache_64b.iftag_stall ./a.out.l
abels 1000000000 1> /dev/null

Performance counter stats for './a.out.labels 1000000000':

81,446,698,907      cycles                                                        (66.66%)

243,218,220,681 instructions # 2.99 insn per cycle (83.33%)
14,907 cache-misses (83.34%)
2,533,002 L1-icache-load-misses (83.34%)
20,571,010 br_misp_retired.all_branches (83.34%)
53,455,580,211 br_inst_retired.all_branches (83.33%)
68,847,492 icache_64b.iftag_stall (57.14%)

  21.512644234 seconds time elapsed

The referenced paper doesn't mention the benefit for the included test program. What is expected improvement for the included test?

Please see more details (build, runtime steps, etc.) in following gist.
https://gist.github.com/uttampawar/5407f998bc3f02f58c4b83b0b4dc20fe

Any hint is appreciated.

Hi,
I followed the guide to build llvm-propeller, it worked well with small binaries.
But encountered following issues when processing perf.data of a large binary:

E0825 16:17:57.031502 2792597 llvm_propeller_profile_writer.cc:1189] Failed to find mmap entries in 'perf.data' for XXX

Hi,
After reading propeller RFC, I found it is an amazing work which really help optimization a lot.
However, After I cloned this repository with its default branch(bb-clusters) and did some experiments.
I don't found it doing something special just like RFC says...
Also, I found you deleted lots of propeller-depended code in the commit 2429f12.
I'm curious about this project status.
Could you kindly give some explanation or some advices for starting testing this project (start from which branch? and what difference between those branches?)

Hi,
As per the instructions in the RFC, I followed all the steps but final binary built with "propeller-optimize" causes segfault.

The final build also gives warning as,
Final stage build
$ clang++ -O2 main.cc callee.cc -fpropeller-optimize=perf.propeller -fuse-ld=lld
ld.lld: warning: [Propeller]: Propeller skipped 'a.out'.

$ ./a.out
Segmentation fault (core dumped)

More details in the gist.
https://gist.github.com/uttampawar/88643a0f532160e1de77d76b07d1ad4a
Gist has perf.propeller file, symbol table info before and after the optimizations.

I did see a warning about libtinfo.so.6 during "create_llvm_prof" stage as,

$LLVM_DIR/llvm-propeller/create_llvm_prof --format=propeller --binary=./a.out.labels --profile=perf.data --out=perf.propeller
/home/upawar/projects/llvm-propeller/create_llvm_prof: /mnt/sdb1/upawar/tools/lib/libtinfo.so.6: no version information available (required by /home/upawar/projects/llvm-propeller/create_llvm_prof)

I think there is another additional load brought by bolt.I think the bolts also bring another extra load. Need to collect runtime information of the original binary. I want to know whether propeller can perf optimized binary and use this information to guide the optimization of the link? Although it sounds unlikely, I still want to ask.

I used xcode11 to run the .dylib generated by llvm-project's clang tools and failed
The following is the error message:

21 QTPlugin.dylib 0x0000000121da0e30 _GLOBAL__sub_I_CommandLine.cpp + 151300528
22 QTPlugin.dylib 0x0000000121daadfb _GLOBAL__sub_I_CommandLine.cpp + 151341435
23 libdyld.dylib 0x00007fff6e68cd8a dlopen + 171
24 clang 0x0000000104488ab0 llvm::sys::DynamicLibrary::HandleSet::DLOpen(char const*, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >) + 32
25 clang 0x0000000104488e40 llvm::sys::DynamicLibrary::getPermanentLibrary(char const, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >) + 48
26 clang 0x00000001059306ad llvm::sys::DynamicLibrary::LoadLibraryPermanently(char const, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >) + 29
27 clang 0x000000010592f3cd clang::ExecuteCompilerInvocation(clang::CompilerInstance) + 893
28 clang 0x00000001008da7f2 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) + 5490
29 clang 0x000000010090cb96 ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) + 438
30 clang 0x00000001054353f8 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optionalllvm::StringRef >, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, bool) const::$_1::operator()() const + 40
31 clang 0x00000001054353c5 void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optionalllvm::StringRef >, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, bool) const::$_1>(long) + 21
32 clang 0x000000010447c309 llvm::function_ref<void ()>::operator()() const + 25
33 clang 0x000000010447c2a3 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) + 243
34 clang 0x0000000105433543 clang::driver::CC1Command::Execute(llvm::ArrayRef<llvm::Optionalllvm::StringRef >, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, bool) const + 499
35 clang 0x00000001053c6095 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const + 1909
36 clang 0x00000001053c63b1 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::__1::pair<int, clang::driver::Command const*> >&) const + 209
37 clang 0x00000001053f3a55 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::__1::pair<int, clang::driver::Command const*> >&) + 341
38 clang 0x000000010090bf2a main + 5258
39 libdyld.dylib 0x00007fff6e6a1cc9 start + 1
clang: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 11.0.0 (/xxxx/llvm-project.git 7392820)
Target: x86_64-apple-darwin19.5.0-simulator
Thread model: posix
InstalledDir: /Users/xxx/llvm-project/build/Debug/bin
clang: error: unable to execute command: Abort trap: 6
clang: note: diagnostic msg: Error generating preprocessed source(s).

I'm trying to build (as per the build instructions in RFC pdf file) llvm-propeller compiler toolchain but the build step is failing. Following are the details.

Environment:
OS release: Ubuntu 18.04.3 LTS
Kernel: 4.15.0-58-generic
GCC: gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
/usr/bin/ld: GNU ld (GNU Binutils for Ubuntu) 2.30
CMAKE: cmake version 3.10.2
ninja: v1.8.2

$ cd build
$ cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;lld;compiler-rt" $HOME/projects/llvm-propeller/llvm

$ tail build-ninja.log
....
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

But tail .ninja_log (hidden file) doesn't show any errors.

I've created a gist with both outputs at,
https://gist.github.com/uttampawar/072f8986d974726fae7fd8e6a057b4dc

Am I missing something?

Any help is appreciated. TIA.

--Uttam
[email protected]

I followed the build guide, but I am not able to run cmake command

$ git clone https://github.com/llvm/llvm-project.git
Cloning into 'llvm-project'...
remote: Enumerating objects: 19852, done.
remote: Counting objects: 100% (19852/19852), done.
remote: Compressing objects: 100% (2966/2966), done.
remote: Total 3783698 (delta 19443), reused 16893 (delta 16886), pack-reused 3763846
Receiving objects: 100% (3783698/3783698), 826.68 MiB | 1.16 MiB/s, done.
Resolving deltas: 100% (3059690/3059690), done.
Checking out files: 100% (88978/88978), done.
$ cd llvm-project/
$ mkdir build
$ cd build/
$ cmake -G Ninja ../llvm
CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_ASM_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
See also "/home/mahmood/sources/llvm-project/build/CMakeFiles/CMakeOutput.log".

I also tried this command and it fails

$ cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=host -DLLVM_ENABLE_CXX1Y=ON -DLLVM_BUILD_TESTS=OFF -DLLVM_BINUTILS_INCDIR=~/codestitcher/source/binutils-2.30/include -DLLVM_BUILD_TOOLS=OFF -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++
CMake Error: The source directory "/storage/users/mnaderan/sources/llvm-project" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.

Any way to fix that? What did I do wrong?

Hi, I am unable to build based on the instructions

c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Any pointers?

/usr/include/c++/7/bits/stl_tree.h:2452:21:   required from ‘void std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_insert_unique(_II, _II) [with _InputIterator = __gnu_cxx::__normal_iterator<llvm::StringRef*, std::vector<llvm::StringRef> >; _Key = std::__cxx11::basic_string<char>; _Val = std::__cxx11::basic_string<char>; _KeyOfValue = std::_Identity<std::__cxx11::basic_string<char> >; _Compare = std::less<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<std::__cxx11::basic_string<char> >]’
/usr/include/c++/7/bits/stl_set.h:183:4:   required from ‘std::set<_Key, _Compare, _Alloc>::set(_InputIterator, _InputIterator) [with _InputIterator = __gnu_cxx::__normal_iterator<llvm::StringRef*, std::vector<llvm::StringRef> >; _Key = std::__cxx11::basic_string<char>; _Compare = std::less<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<std::__cxx11::basic_string<char> >]’
/home/ubuntu/llvm-propeller/lld/ELF/Propeller/Propeller.cpp:604:63:   required from here
/usr/include/c++/7/bits/stl_tree.h:2201:59: error: no match for call to ‘(std::_Identity<std::__cxx11::basic_string<char> >) (llvm::StringRef&)’
  = _M_get_insert_hint_unique_pos(__position, _KeyOfValue()(__v));
                                              ~~~~~~~~~~~~~^~~~~
In file included from /usr/include/c++/7/string:48:0,
                 from /home/ubuntu/llvm-propeller/llvm/include/llvm/Support/ErrorHandling.h:18,
                 from /home/ubuntu/llvm-propeller/llvm/include/llvm/ADT/Hashing.h:48,
                 from /home/ubuntu/llvm-propeller/lld/include/lld/Common/LLVM.h:19,
                 from /home/ubuntu/llvm-propeller/lld/include/lld/Common/ErrorHandler.h:71,
                 from /home/ubuntu/llvm-propeller/lld/ELF/Propeller/Propeller.h:50,
                 from /home/ubuntu/llvm-propeller/lld/ELF/Propeller/Propeller.cpp:24:
/usr/include/c++/7/bits/stl_function.h:873:7: note: candidate: _Tp& std::_Identity<_Tp>::operator()(_Tp&) const [with _Tp = std::__cxx11::basic_string<char>]
       operator()(_Tp& __x) const
       ^~~~~~~~
/usr/include/c++/7/bits/stl_function.h:873:7: note:   no known conversion for argument 1 from ‘llvm::StringRef’ to ‘std::__cxx11::basic_string<char>&’
/usr/include/c++/7/bits/stl_function.h:877:7: note: candidate: const _Tp& std::_Identity<_Tp>::operator()(const _Tp&) const [with _Tp = std::__cxx11::basic_string<char>]
       operator()(const _Tp& __x) const
       ^~~~~~~~
/usr/include/c++/7/bits/stl_function.h:877:7: note:   no known conversion for argument 1 from ‘llvm::StringRef’ to ‘const std::__cxx11::basic_string<char>&’

Where does the file pgo-vanilla-compiler.yaml(https://github.com/google/llvm-propeller/blob/bb-clusters/clang-bolt-experiment/v2/pgo-vanilla-compiler.yaml) come from?