kobzol / cargo-pgo Goto Github PK

Hi excuse me,
I just use a hello world project(only have one file main.rs) to learning the PGO
under the project when I execute cargo pgo build meet this

errror[E0463]: can't find crate for profiler_builtins
|
= note: the compiler may have been built without the profiler runtime

error: aborting due to previous error

For more information about this error, try rustc --explain E0463.
error: could not compile ww due to 2 previous errors
[2023-12-05T16:47:31Z ERROR cargo_pgo::pgo::instrument] PGO instrumentation build has failed.
Cargo finished with an error (101)

But there is no package named profiler_builtins in crates.io

Bumping the cargo dependency should do the trick.

Currently, cargo-pgo runs cargo, gathers its output and then writes it to the output. However, it would be much nicer if we could instead stream the output of cargo interactively, so that the user can see it immediately and not just at the end.

Use-case: #5 (comment)

In order to get PGO+BOLT working, I needed to

• Switch from fat LTO to thin
• Disable strip = "symbols" in my release profile
• Use trunk LLVM (you already have this in the README, this is just to affirm that it's still needed - I was trying with the 15.0.4 tag previously)

Is it worth documenting those, and/or is there a way to make those settings work in RUSTFLAGS when you build the PGO-optimized binary that you're going to pass to llvm-bolt --instrument? The errors come from cc, which makes them super cryptic and unhelpful; I found the workarounds by just throwing possible fixes at the wall until something stuck 😂

I can retrigger the errors tonight if having a copy of them would be helpful

Looks like cargo pgo [optimize] build ignores rustflags in config.toml file, is there a way to pass it to PGO subcommand?

The README.md mentions the need to build bolt yourself - this is not required any more, at least on Debian-based distros. It is available in the LLVM repositories( https://apt.llvm.org/ ) starting with LLVM 15(maybe even LLVM 14, have not checked).

Example install on Ubuntu(matching the version used by rustc), starting with rustc 1.65:

# `grep "LLVM version"` if we worry that the LLVM version won't be always the last in the list
export LLVM_MAJOR_VERSION="$(rustc -vV | grep "LLVM version" | cut -d ' ' -f3 | cut -d '.' -f1)"
# `tail -1`  if we worry that `LLVM version` won't be English-only in the future
export LLVM_MAJOR_VERSION="$(rustc -vV | tail -1 | cut -d ' ' -f3 | cut -d '.' -f1)"

export UBUNTU_CODENAME="$(lsb_release -c | cut -d ':' -f2 | xargs)"
echo "deb http://apt.llvm.org/${UBUNTU_CODENAME}/ llvm-toolchain-${UBUNTU_CODENAME}-${LLVM_MAJOR_VERSION} main" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://apt.llvm.org/${UBUNTU_CODENAME}/ llvm-toolchain-${UBUNTU_CODENAME}-${LLVM_MAJOR_VERSION} main" | sudo tee -a /etc/apt/sources.list
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -

sudo apt install bolt-${LLVM_MAJOR_VERSION}

dpkg -L bolt-15:

/.
/usr
/usr/bin
/usr/lib
/usr/lib/llvm-15
/usr/lib/llvm-15/bin
/usr/lib/llvm-15/bin/llvm-bolt
/usr/lib/llvm-15/bin/llvm-bolt-heatmap
/usr/lib/llvm-15/bin/merge-fdata
/usr/share
/usr/share/doc
/usr/share/doc/bolt-15
/usr/share/doc/bolt-15/NEWS.Debian.gz
/usr/share/doc/bolt-15/changelog.Debian.gz
/usr/share/doc/bolt-15/copyright
/usr/bin/llvm-bolt-15
/usr/bin/llvm-bolt-heatmap-15
/usr/bin/llvm-boltdiff-15
/usr/bin/merge-fdata-15
/usr/bin/perf2bolt-15
/usr/lib/llvm-15/bin/llvm-boltdiff
/usr/lib/llvm-15/bin/perf2bolt

This might also need an update:

When cargo-pgo is used, the [build.rustflags] of .cargo\config.toml are ignored. Lto in cargo.toml seems to be ignored as well, judging by performance. The only way to pass flags seems to be through the RUSTFLAGS environment variable.

Is it possible to ship the PGO/BOLT profile files with a crate such that a cargo install will use them? That would be quite powerful for rust ecosystem tools where people tend to prefer installing from source.

[lofty@MISO yukari]$ rustc -V
rustc 1.65.0-nightly (2befdefdd 2022-08-06)
[lofty@MISO yukari]$ cargo pgo -V
cargo-pgo 0.1.0
[lofty@MISO yukari]$ llvm-bolt --version
LLVM (http://llvm.org/):
  LLVM version 16.0.0git
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: skylake

BOLT revision 6bb51bf06214af3690af7034f4edeb265732c481
[lofty@MISO yukari]$ cargo pgo bolt build
[2022-08-07T14:04:55Z INFO  cargo_pgo::bolt::instrument] BOLT profile directory will be cleared.
[2022-08-07T14:04:55Z INFO  cargo_pgo::bolt::instrument] BOLT profiles will be stored into /home/lofty/yukari/target/bolt-profiles.
[2022-08-07T14:04:55Z INFO  cargo_pgo::bolt::instrument] Binary yukari built successfully. It will be now instrumented with BOLT.
[2022-08-07T14:04:55Z INFO  cargo_pgo::bolt::instrument] Binary yukari instrumented successfully. Now run /home/lofty/yukari/target/x86_64-unknown-linux-gnu/release/yukari-bolt-instrumented on your workload.
[2022-08-07T14:04:55Z INFO  cargo_pgo::bolt::instrument] BOLT instrumentation build finished successfully.
[lofty@MISO yukari]$ /home/lofty/yukari/target/x86_64-unknown-linux-gnu/release/yukari-bolt-instrumented
-bash: /home/lofty/yukari/target/x86_64-unknown-linux-gnu/release/yukari-bolt-instrumented: No such file or directory
[lofty@MISO yukari]$ find target -name yukari-bolt-instrumented
[lofty@MISO yukari]$

This was while I was working on https://github.com/yukarichess/yukari.

I wish I could provide more to work with, but I don't know where to begin.

So I have a server that is supposed to run forever. How can I gather instrumentation data for such a program?

Not sure if is it an issue with cargo-pgo or not - I'll just put it here to share with other people.

During the PGO optimization for https://github.com/gorules/zen/ project, I found that cargo bench --workspace works fine (builds all required targets successfully and performs the required benchmarks) but during the cargo pgo bench -- --workspace I get the following linking errors: https://gist.github.com/zamazan4ik/176517dd1d5bb7dc07c92e13016dfd8d

I'm not sure what is the reason for the difference in behavior between these two almost the same commands. Maybe some custom build.rs rules are the root cause of the problem.

Currently I was only able to build an optimized benchmark binary with cargo pgo --bench <bench>. But I could find no way to feed it into the benchmark runner. I suspect something like cargo pgo bench and cargo pgo bolt bench would be required to support this. The only workaround I found was to copy the bolt optimized binary to the path of the regular cargo bench generated one, essentially replacing the binary. But that's rather cumbersome.

I'm using cargo-pgo on CentOS 7 following the instruction in README, when I ran cargo pgo build, undefined reference to memfd_create in memfd.rs was thrown.
I googled and it's said that this is due to a too-old glibc, so are there any suggestions for users on these OS with only glibc2.17, or the optimization is based on a newer OS so we have to update the working environment?

Hi!

I am trying to test PGO with https://github.com/cloudflare/boringtun . I have the following errors:

... <lines omitted>
Compiling boringtun v0.5.2 (/Users/zamazan4ik/open_source/boringtun/boringtun)
warning: unused variable: `fwmark`===> ] 170/177: clap, boringtun, clap, criterion
   --> boringtun/src/device/peer.rs:107:9
    |
107 |         fwmark: Option<u32>,
    |         ^^^^^^ help: if this is intentional, prefix it with an underscore: `_fwmark`
    |
    = note: `#[warn(unused_variables)]` on by default

warning: 535 warnings emitted========> ] 172/177: boringtun, criterion

   Compiling boringtun-cli v0.5.2 (/Users/zamazan4ik/open_source/boringtun/boringtun-cli)
warning: unused variable: `fwmark`===> ] 174/177: crypto_benches(bench), boringtun(test), boringtun-cli(bin test)
   --> boringtun/src/device/peer.rs:107:9
    |
107 |         fwmark: Option<u32>,
    |         ^^^^^^ help: if this is intentional, prefix it with an underscore: `_fwmark`
    |
    = note: `#[warn(unused_variables)]` on by default

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boringtun-cli`; 3 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boringtun`; 26 warnings emitted
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boringtun`; 48 warnings emitted
PGO optimized build has failed.
Cargo finished with an error (101)

I did the following steps:

• Run cargo pgo bench. All is fine here.
• Run cargo pgo optimize bench. Here I got the errors.

I have tried to google the error but with no success. I didn't dig into LLVM, however.

If you need additional information - I would be happy to provide it :)

I can work for it. BTW, is it necessary for creating a Discord channel?

I get this error when I run cargo pgo optimize on a project under /home, which is mounted on a separate partition. If I copy the project to e.g. /tmp and run the command on the copy - no problem whatsoever.

I have a WASM server-side project I'd love to try out PGO on. Unfortunately it does not seem to work. E.g. running the command

cargo pgo instrument build -- --bin my-binary--release --target wasm32-wasi

results in the error

error[E0463]: can't find crate for `profiler_builtins`

however, the same command without the --target does not provoke the same error.

I do not necessarily think this is anything cargo-pgo can do anything about - I believe the underlying issue is this one. I figured I'd create an issue anyways, in case I was wrong, or anyone else came looking for the same information.

Hi!

Do you plan to add to cargo-pgo support for the additional PGO modes:

• AutoFDO support (PGO via sampling instead of instrumentation): https://github.com/google/autofdo . rustc already supports it: rust-lang/rust@a17193d AutoFDO could be useful for users who want to gather profiles directly from production env, without building slow instrumentation-only binaries
• Add support for BOLT with perf mode instead of instrumentation (for the same reasons as AutoFDO). Yes, I understand that it's only Linux-only feature but still - we have a lot of Linux-users
• Add support for Propeller (an alternative approach to BOLT but from Google). Now it's a part of AutoFDO. However, I didn't test it yet with Rust binaries

And thank you again for cargo-pgo - it's much easier to apply PGO to Rust-based binaries with it :)

Hello, I wanted to configure a github action to generate pgo optimized releases, but did not find good resources on doing that.
Also would it be reasonable to make a cargo-pgo action so it becomes simpler for more people to apply pgo to their code?

Hi, I'd like to run my program multiple times with different flags, merge their profiles, and optimize the binary based on the result. However, cargo pgo run seems to clear the profiles directory on every invocation. Would it be possible to add a flag to avoid doing this?

OS: Arch Linux

I have all LLVM related packages installed, -- --verbose does not output anything

While I can use --profile=xyz to pass a different profile to Cargo, this conflicts with the --release option that cargo-pgo always supplies. In my case, since other tools like wasm-pack are also inflexible in terms of Cargo profiles, I cannot use --release and have (multiple) separate release profiles for different purposes.

Therefore, it would be nice to have a profile option on cargo-pgo directly, which will override the --release argument for Cargo. The default doesn't have to be changed.

I have a command-line app that I wishes to use PGO and BOLT. It takes a few inputs from stdin at the startup. It works well with cargo run.

cargo pgo seems to be overriding stdin by passing stdio::null().

Is it necessary for cargo pgo functionalities? cargo pgo run obviously works differently than cargo run due to this.

Hi!

Is there a way to calculate PGO coverage of the profiled program? I see now cargo-pgo shows warnings for functions without a profile. I want to see more "advanced" statistics regarding PGO coverage. I guess something like gcov + lcov from C++ world (with different report capabilities, etc).

I guess it could be done somehow with llvm-cov report profile.profdata but I am not sure.

Thanks in advance!

During the PGO optimization, I found an interesting example, when a Rust program has a C/C++ dependency in a critical performance path. As an example - TiKV. TiKV depends on https://github.com/tikv/rust-rocksdb/ that under-the-hood compiles RocksDB (https://github.com/tikv/rust-rocksdb/tree/master/librocksdb_sys).

The problem is that PGO instrumentation flags are not propagated to the non-Rust dependencies, so they are not PGO-optimized during the cargo-pgo optimization.

Honestly, not sure what we can/should do here. Because I understand that implementing PGO flags propagation to the non-Rust dependencies would be difficult like hell: different C compilers with different options, need to think somehow modify build.rs scripts, etc.

Right now I see the only option to remediate it - optimize with BOLT since it works on the resulting binary. Probably we need to document this limitation somewhere in cargo-pgo.