r-prof / jointprof Goto Github PK

Ubuntu 16.10 comes with gperftools 2.4, 17.04 has 2.5. Solved locally by manually downloading debs from https://launchpad.net/ubuntu/+source/google-perftools and installing via dpkg -i. Need to figure out the reason, though:

• compile gperftools 2.4 and 2.5 (at least)
• run against compiled versions
• bisect

The code in README.md only works for gperftools <= 2.4. It seems that gperftools >= 2.5 installs a signal handler once when the library is loaded, while gperftools <= 2.4 install a signal handler when the profiler is started, and remove it afterwards.

I think this is an excellent idea/package.

I just wanted to point out there is a CRAN package with a name conflict for this package.

https://cran.r-project.org/web/packages/gProfileR/index.html

While the case is different, I'm unsure if you want to have 2 packages with the same name.

created by Rcpp::cppFunction().

Error: package or namespace load failed for ‘jointprof’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-jointprof/00new/jointprof/libs/jointprof.so':
dlopen(/Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-jointprof/00new/jointprof/libs/jointprof.so, 6): Symbol not found: __Z24GetStackTraceWithContextPPviiPKv
Referenced from: /Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-jointprof/00new/jointprof/libs/jointprof.so
Expected in: flat namespace
in /Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-jointprof/00new/jointprof/libs/jointprof.so
Error: loading failed
Execution halted
ERROR: loading failed

• removing ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/jointprof’
  Warning message:
  In i.p(...) :
  installation of package ‘/var/folders/mq/303j9hds3sd14mdf7d50tb7r0000gn/T//RtmpuSikjS/file2bd27243e666/jointprof_0.0.1.9001.tar.gz’ had non-zero exit status

• R stack trace: "as is"
• C stack trace: collapse all code from libR.so into one node

by returning early from the filter.

currently requires pprof, a Go program. Need to do one of the following:

1. Parse the textual output of pprof
2. Call Go from R via cgo
3. Rebuild the parser for the profiler output in R
4. Build a reader for the profiler data

A first implementation will focus on the first option and require a compatible version of pprof that can be found by the package.

• distinguish R sources from packages' sources
  • R: usually starts with src/main or src/library
  • packages: usually src/ without subdirectory
• interleaved stacks R -> native -> R -> native ...
  • need to find entry point(s) native -> R

• gprofiler::with_profiler()
• gprofiler::run_pprof() via processx

r-lib/tidyselect#56

Needs proper deinitialization (or not do work on reinitialization):

Loading jointprof
Disabling profiler because signal 27 handler is already in use.

adapted from profvis code, rstudio/profvis#74.

https://github.com/brodieG/treeprof

tidyverse/dplyr#2922 (comment)

• Ease of use
• Output format

Windows is highly unlikely to work with gperftools. Need to:

• Capture stack trace
• Write /proc/self maps (how?)
• Symbolize

Perhaps VerySleepy can be ported?

tidyverse/dplyr#977

Prepare for release:

• Check that description is informative
• Check licensing of included files
• usethis::use_cran_comments()
• devtools::check()
• devtools::check_win_devel()
• rhub::check_for_cran()
• rhub::check(platform = 'ubuntu-rchk')
• rhub::check_with_sanitizers()
• Polish pkgdown reference index
• Draft blog post

Submit to CRAN:

• usethis::use_version('major')
• Update cran-comments.md
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• usethis::use_github_release()
• usethis::use_dev_version()
• Update install instructions in README
• Finish blog post
• Tweet
• Add link to blog post in pkgdown news menu

The jointprof vignette renders a call graph using pprof with the following remark:

The call graph shows a unified view of run time costs for both R and native code. (Click on the image to see the SVG version.)

This link is currently broken.

https://gperftools.github.io/gperftools/heapprofile.html

• link executable (!) against tcmalloc
• sample with HeapProfilerDump()
• analyze
• use Rprof(memory.profiling = TRUE)

needs a better error message.

Continuing r-prof/profile#7 here because jointprof has find_pprof().

seems to require PKG_LIBS = -ltcmalloc -lprofiler (in that order). Need to double-check if there are any adverse side effects: gperftools/gperftools#948 (comment).

Installation:

• pprof via Go
• gperftools via brew
• ...?

Implement Travis test, Travis debugging is already enabled for this repo.

r-dbi/RSQLite#179 (comment)

to make sure that execution time is attributed correctly in case of deferred/lazy evaluation. This seems to require rewriting R's profile recording routine.

https://github.com/r-lib/lobstr#call-stack-trees

Example where this went wrong: patperry/r-utf8#8 (comment).

@atheriel: Would you like to share your experience?

Need gperftools to run profiling.

• Linux
  • installation instructions?
• OS X
  • installation instructions?
• Windows
  • need to package libprofiler?

https://cran.r-project.org/doc/manuals/R-exts.html#oprofile-and-operf

I'm seeing

Disabling profiler because signal 27 handler is already in use.

on Travis CI, still using R 3.4.4 locally.

because the stock implementation rearranges and aggregates samples. Reuse code, luckily stacktrace.h is public.

Shipping a modified version of libprofiler.so in an R package is an alternative, but:

• reimplementing doesn't seem to be too difficult
• the target file format is very simple to write
• we have more control with our own implementation
• we might even be able to support Windows
• we get rid of that message to stderr

Steps:

• write header
• establish signal handler
• record stack trace, write to file immediately (CpuProfiler::prof_handler())
• daisy-chain with R's signal handler
• when finishing, write trailer and dump proc map (DumpProcSelfMaps())

• org website with pointers
  • installation instructions
• split pprof package
  • installation of binary files: src/install.libs.R in R-exts
• brush up documentation
• get rid of check warnings

for Linux and OS X, including instructions for pprof.

I'm seeing a proper call stack in the debugger, but not in the output of gperftools.

• Try different compiler settings or compilers (clang?)
• Look at the internals of gperftools

• allow incomplete files
• merge records read from incomplete files
• incremental reading

What to do in case of a failure?

• multiprocess
  • only profile main process
• multithreaded
  • by definition only R -> C, depends on the capabilities of the profiler library

The package currently contains a write_flat_ds() function (in debug.R) that is not exported, used anywhere else in the package, or used in any tests. In addition, this function is the sole user of the readr dependency, although that could probably be moved to Suggests or replaced with utils::write.csv().

I think there are a few possible ways to deal with this (currently) dead code:

• Remove it.
• Use it in the test suite; or
• Move it to the profile package.

Example call tree:

a
  b
    c
      a
        d
        b
          c
            a
              b

Simplify to three call trees:

a
  b
    c
a
  d
  b
    c
a
  b

Upstream issue: gperftools/gperftools#948.

The necessary patch to gperftools is in upstream master but not yet on Debian: gperftools/gperftools#1187. It will take some time to appear on Ubuntu anyway (where it is optional), and can already be installed on Homebrew (using HEAD) where it is mandatory.

Aiming at implementing compatibility with both versions, driven by the configure script.

that shows how to use the project.

What happens if a substantial amount of time is spent in internal R routines (native code), such as string conversion? Can we use joint profiling to help profiling R itself?

CC @radfordneal.

I'm running Ubuntu 20.04.1 LTS with R 3.6.3 installed. I tried to profile the C code in coxph function of survival using the following commands (I used coxed solely to increase the running time of coxph):

library(survival)
library(jointprof)
library(coxed)

simdata = sim.survdata(N = 10000)

joint_pprof(coxph(Surv(y, failed) ~ X1+X2+X3, data=simdata$data))

The following the graph I got as a result. Before profiling, I recompiled the whole survival package with install.packages("survival", type="source", INSTALL_opts="--with-keep.source") and CFLAGS set to -g -O0 -mtune=native. It's clear that the C code profiling results are missing.

After looking into some intermediate results (the separate pprofile and rprofile files), it is clear that the C code was indeed profiled and the results were stored. However, in the rprofile file:

...
#File 3: /tmp/Rtmp5g7EGI/R.INSTALL235a67daae2bd/survival/R/coxph.fit.R
"aperm.default" "aperm" "apply" 3#55 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
"%in%" 3#55 "FUN" "apply" 3#55 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
"%in%" 3#55 "FUN" "apply" 3#55 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
"%in%" 3#55 "FUN" "apply" 3#55 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
3#59 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
3#59 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
3#59 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
3#59 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
3#59 "coxph.fit" 2#555 "coxph" "comingle_rprof" "comingle_pprof" "joint_pprof" 
...

In line 59 of coxph.fit.R:

    coxfit <- .Call(Ccoxfit6, ...

.Call is called but the function name is not registered in the rprofile result. This is perhaps the reason why the C code profiling results did not show up in the graph, because jointprof need to know where .Call is called inside R to locate the position where the two separate profiling results should be combined.
As I am not very familiar with profiling in general, I could not go any further on this issue.