Saw this message in the travis test logs:

--- PASS: TestWebInterface (0.00s)
	webui_test.go:76: skpping / since dot (graphviz) does not seem to be installed

We should make sure we have graphviz installed so that tests don't skip this. Bonus for fixing the typo "skpping" -> "skipping" :)

After trying gperftools for the first time, and running into issue 791, I decided to try this standalone pprof as the recommend solution to that issue.

However, instead of displaying functions as addresses (without names), the standalone pprof displays only filenames (without function names) despite my best efforts.

For example:

$ LD_PRELOAD=/usr/local/lib/libprofiler.dylib CPUPROFILE=/tmp/prof.out ./_test -e blah
NOTICE:  Running test backupstore in debug mode...
INFO:    Starting server: ../../bin/s3simulator/s3simulator  testfiles/s3simulator.conf
PROFILE: interrupts/evictions/bytes = 0/0/64
...
PASSED
PROFILE: interrupts/evictions/bytes = 334/19/37104

$ pprof -functions -tree ./_test /tmp/prof.out
Local symbolization failed for libsystem_kernel.dylib: unrecognized binary: /usr/lib/system/libsystem_kernel.dylib
Local symbolization failed for libdyld.dylib: unrecognized binary: /usr/lib/system/libdyld.dylib
Local symbolization failed for libsystem_info.dylib: unrecognized binary: /usr/lib/system/libsystem_info.dylib
Local symbolization failed for libsystem_platform.dylib: unrecognized binary: /usr/lib/system/libsystem_platform.dylib
Local symbolization failed for libc++.1.dylib: unrecognized binary: /usr/lib/libc++.1.dylib
Local symbolization failed for libsystem_dnssd.dylib: unrecognized binary: /usr/lib/system/libsystem_dnssd.dylib
Local symbolization failed for libsystem_c.dylib: unrecognized binary: /usr/lib/system/libsystem_c.dylib
Local symbolization failed for libsystem_malloc.dylib: unrecognized binary: /usr/lib/system/libsystem_malloc.dylib
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
File: _test
Type: cpu
Showing nodes accounting for 3.33s, 99.70% of 3.34s total
Dropped 1 node (cum <= 0.02s)
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 
----------------------------------------------------------+-------------
                                             2.15s 69.58% |   <unknown>
                                             0.65s 21.04% |   [_test]
                                             0.01s  0.32% |   [libsystem_info.dylib]
     3.09s 92.51% 92.51%      3.09s 92.51%                | [libsystem_kernel.dylib]
----------------------------------------------------------+-------------
                                             3.05s 99.67% |   [libdyld.dylib]
                                             0.07s  2.29% |   <unknown>
                                             0.06s  1.96% |   [libsystem_dnssd.dylib]
                                             0.05s  1.63% |   [libc++.1.dylib]
                                             0.03s  0.98% |   [libsystem_info.dylib]
     0.11s  3.29% 95.81%      3.06s 91.62%                | [_test]
                                             1.39s 45.42% |   <unknown>
                                             0.96s 31.37% |   [libsystem_info.dylib]
                                             0.65s 21.24% |   [libsystem_kernel.dylib]
                                             0.07s  2.29% |   [libsystem_platform.dylib]
                                             0.06s  1.96% |   [libc++.1.dylib]
                                             0.02s  0.65% |   [libsystem_c.dylib]
                                             0.01s  0.33% |   [libsystem_malloc.dylib]
----------------------------------------------------------+-------------

It doesn't say that symbolisation failed for _test (the main executable) but it almost looks that way, unless I'm doing something wrong.

I hope you can help, because pprof (and therefore gperftools) is completely useless to me without at least displaying the names of the hot functions.

pprof includes the hostname:port as part of the filename for saved profiles, which causes the saved files to use Alternate Data Streams (ADS) when the file exists on an NTFS volume. Technically everything seems to still work, as referencing a saved file using the full name will retrieve the data from the alternate stream, but if those files are moved to another volume the ADS data could be lost. They also appear to be 0 bytes, since most tools only show the size of the primary data stream. Example:

go tool pprof -alloc_space http://hostname:6060/debug/pprof/heap
Fetching profile from http://hostname:6060/debug/pprof/heap
Saved profile in C:\Users\gbray\pprof\pprof.hostname:6060.alloc_objects.alloc_space.001.pb.gz
Entering interactive mode (type "help" for commands)
(pprof) exit
#Run the above twice to save two profiles

C:\Users\gbray> dir .\pprof\


    Directory: C:\Users\gbray\pprof


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       12/21/2016   1:36 PM              0 pprof.hostname

C:\Users\gbray> Get-Item .\pprof\hostname -Stream * | ft FileName,Stream,Length

FileName                               Stream                                      Length
--------                               ------                                      ------
C:\Users\gbray\pprof\pprof.hostname    :$DATA                                      0
C:\Users\gbray\pprof\pprof.hostname    6060.alloc_objects.alloc_space.001.pb.gz    36909
C:\Users\gbray\pprof\pprof.hostname    6060.alloc_objects.alloc_space.002.pb.gz    37513

To prevent using ADS the pprof tool could either replace : with another character, or allow users to specify the filename for the saved profile.

Alternatively, to workaround the ADS issue users can run the following PowerShell script to extract the contents of the ADS and save them as normal files using _ instead of :

$streams = Get-Item ~\pprof\* -Stream * | ?{$_.Stream -ne ':$DATA'}
foreach($stream in $streams){
    $filename = $stream.FileName
    $streamname = $stream.Stream
    $newfilename = "${filename}_${streamname}"
    Write-Output "Starting $newfilename"
    Get-Content $filename -Stream $streamname -Encoding Byte | Set-Content $newfilename -Encoding Byte
    #Remove-Item $filename -Stream $streamname #Remove ADS
}
#Remove any 0 length files
#Get-Item ~\pprof\* | ?{$_.Length -eq 0} | Remove-Item -Force

Attached a profile (zipped gzip is silly, but Github won't let me upload "heapz".)

pprof -raw heapz
...

Samples:
objects/count space/bytes[dflt]
       7438   60936269: 1 2 3 4 5 6
                bytes:[8192] request:[8191]
        238   62466710: 7 8 9 10 5 6
                bytes:[262144] request:[262143]
       3341   54739221: 7 8 9 10 5 6
                bytes:[16384] request:[16383]
       2064   67634517: 1 2 3 4 5 6
                bytes:[32768] request:[32767]
      29710   60847108: 1 2 3 4 5 6
                bytes:[2048] request:[2047]
        396   51920894: 1 2 3 4 5 6
                bytes:[131072] request:[131071]
      63503   65027585: 7 8 9 10 5 6
                bytes:[1024] request:[1023]
       1105   72423082: 7 8 9 10 5 6
                bytes:[65536] request:[65535]
        108   56884975: 11 12 13 3 4 5 6
                bytes:[524288] request:[524287]
      15887   65075220: 7 8 9 10 5 6
                bytes:[4096] request:[4095]
...
pprof -traces heapz
...

-----------+-------------------------------------------------------
   69.07MB   [UNKNOWN]
             [UNKNOWN]
             [UNKNOWN]
             [UNKNOWN]
             __libc_start_main
             [UNKNOWN]
-----------+-------------------------------------------------------
   54.25MB   [UNKNOWN]
             [UNKNOWN]
             [UNKNOWN]
             [UNKNOWN]
             [UNKNOWN]
             __libc_start_main
             [UNKNOWN]
-----------+-------------------------------------------------------
   62.06MB   [UNKNOWN]
             [UNKNOWN]
             [UNKNOWN]
             [UNKNOWN]
             __libc_start_main
             [UNKNOWN]
-----------+-------------------------------------------------------
...etc

the "traces" output (which is much more useful (as it can be filtered, etc)) should contain the bytes/request tags.
heapz.zip

I want a way to combine memory profiles collected by different languages on the same process. (e.g. Go and C++)

profile.Merge provides merging of multiple profile.Profiles, but the merge operation is allowed only when the Profiles contain the same types and the period type.
Go's and C++'s memory profiles have the same period type, but the sample types are different, so profile.Merge is unhappy about it.

• Go
  PeriodType: space bytes
  Period: 524288
  Samples: alloc_objects/count alloc_space/bytes inuse_objects/count inuse_space/bytes

• C++
  PeriodType: space bytes
  Period: 2097152
  Samples: objects/count space/bytes[dflt]

I see two options:

1. Extend or provide a variation of profile.Merge that accepts slightly different profiles, maps sample types, and outputs the intersection.
   a) Must be careful about handling of the different Periods. (may need normalization. I am afraid the current Merge doesn't do anything complex other than choosing max Period and adding up values.)
   b) Handling of Duration/Time should be carefully done.
   c) Also, it's possible the one profile may include symbolization info while another doesn't. Does the final output have to drop all symbols in this case?

2. Extend the http endpoint or profile output format to include multiple profile proto messages, and let the pprof tool visualize and analyze the multiple profiles separately. For example, layout Go and C++ allocation graphs in separate groups, but in the same svg graph.

If the normalization required by 1) can't be done correctly, I'd prefer the option 2).

@rauls5382 @aalexand

The pprof executable supports flags added by parseFlags in internal/driver/cli.go, but not all of those flags are reported in the output of pprof -help. As far as I can tell the following flags are not mentioned in the -help output:

• -inuse_space
• -inuse_objects
• -alloc_space
• -alloc_objects
• -total_delay
• -contentions
• -mean_delay

This was reported in the Go repository at golang/go#20734.

$ go tool pprof x.prof
Type: cpu
Time: Mar 7, 2017 at 10:22am (EST)
Duration: 45.82s, Total samples = 400.50ms ( 0.87%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 399.40ms, 99.73% of 400.50ms total
Dropped 42 nodes (cum <= 2ms)
Showing top 10 nodes out of 24
      flat  flat%   sum%        cum   cum%
  110.70ms 27.64% 27.64%   110.70ms 27.64%  rsc.io/tmp/gammabench.Lgamma17 /Users/rsc/src/rsc.io/tmp/gammabench/lgamma17.s
  107.90ms 26.94% 54.58%   107.90ms 26.94%  rsc.io/tmp/gammabench.LgammaZZZ /Users/rsc/src/rsc.io/tmp/gammabench/lgammazzz.s
   96.70ms 24.14% 78.73%    96.70ms 24.14%  rsc.io/tmp/gammabench.Lgamma16 /Users/rsc/src/rsc.io/tmp/gammabench/lgamma16.s
   60.60ms 15.13% 93.86%    60.60ms 15.13%  runtime.usleep /Users/rsc/go/src/runtime/sys_darwin_amd64.s
    9.60ms  2.40% 96.25%   106.30ms 26.54%  rsc.io/tmp/gammabench.BenchmarkLgamma16 /Users/rsc/src/rsc.io/tmp/gammabench/g_test.go
    5.50ms  1.37% 97.63%   113.40ms 28.31%  rsc.io/tmp/gammabench.BenchmarkLgammaZZZ /Users/rsc/src/rsc.io/tmp/gammabench/g_test.go
    4.20ms  1.05% 98.68%   114.90ms 28.69%  rsc.io/tmp/gammabench.BenchmarkLgamma17 /Users/rsc/src/rsc.io/tmp/gammabench/g_test.go
    4.20ms  1.05% 99.73%     4.20ms  1.05%  runtime.mach_semaphore_signal /Users/rsc/go/src/runtime/sys_darwin_amd64.s
         0     0% 99.73%     3.90ms  0.97%  runtime.goready /Users/rsc/go/src/runtime/proc.go
         0     0% 99.73%     3.90ms  0.97%  runtime.goready.func1 /Users/rsc/go/src/runtime/proc.go
(pprof) 

These file names didn't used to be there, and they make the output pretty wide and noisy. Are they intended to be there? Am I doing something wrong in the profile generation?

Thanks.

When I run go tool pprof myfile.prof on Windows I get an error: failed to access temp dir /pprof: mkdir /pprof: Access is denied. Setting PPROF_TMPDIR to a valid directory works around this.

I suspect what happens is that PPROF_TMPDIR ignores the fact that %HOME% is empty on Windows and still uses "%HOME%/pprof" as the default value of PPROF_TMPDIR. It should probably

1. Use %TEMP% on Windows.
2. Check that $HOME is not empty before using it (everywhere).

(Go version 1.8.3. pprof version unknown, since it's not displayed anywhere that I can find.)

Travis only supports Linux and OSX, let's integrate with AppVeyor for Windows testing to catch things like #148.

See https://github.com/google/benchmark for example integration.

I'm trying to update the vendored pprof in the Go repository.

I've already manually cherrypicked the fix to issue #195 (I'll properly re-vendor later when the patch gets merged), now the nacl trybots are stuck on:

--- FAIL: TestNewListenerAndURL (0.00s)
    --- FAIL: TestNewListenerAndURL/localhost: (0.00s)
    	webui_test.go:228: newListenerAndURL(localhost:) = listen tcp: Protocol not available
FAIL
FAIL	cmd/vendor/github.com/google/pprof/internal/driver	0.681s
2017/08/20 12:28:34 Failed: exit status 1

Link to the full log.

Looks similar to issue #144: the test fails because tcp is not available.

Flame graphs allow you to move in a specific ancestry path from a very compact UI and lets the users to zoom in/out specific paths easily. Flame graphs became one of the defato representation of profiling data in the industry in the recent years, e.g. go-torch is a highly successful tool and often user's entry point to visualize pprof data in Go.

Consider supporting flame graphs in addition to the dot visualization on the new web UI, so we can provide this representation out-of-the-box with minimal setup and external dependencies.

(If you need this feature as well, you can 👍 so the team knows about your interest.)

Right now some amount of code in gperftools is relying on pprof --symbols (invoked via symbolize.cc). And apparently golang version doesn't support this flag. Can this support be added?

Trying to vendor pprof for the next go release (1.9). The nacl trybots do not like the TestHttpsInsecure test.

--- FAIL: TestHttpsInsecure (0.00s)
	fetch_test.go:252: net.Listen: got error listen tcp: Protocol not available, want no error
FAIL
FAIL	cmd/vendor/github.com/google/pprof/internal/driver	0.556s

Full log.

Related to #178: a problem with the current format is that a tag is just a label with a name and either a string or an int value...but the name, while often implying a unit, isn't actually a unit.

It would be good if we could extend the profile.proto format to support tags having both a name and an associated unit scale.

Sorry if this is not the best place to post this question, but I couldn't find any mailing list info.

I have followed the instructions in the Readme, putting defer profile.Start().Stop() at the top of my main function, but the output when examined with pprof doesn't have any information about individual function calls:

➜  ~ go tool pprof -text /tmp/profile344813197/cpu.pprof
15.91s of 15.91s total (  100%)
      flat  flat%   sum%        cum   cum%
    15.91s   100%   100%     15.91s   100%  

This is most certainly a misunderstanding about how the tool is supposed to work but I'd appreciate some clarification.

To help catch things like #146.

See https://docs.travis-ci.com/user/osx-ci-environment for the versions available in Travis. Suggesting using xcode6.4 (OSX 10.10), xcode7.3 (OSX 10.11) and xcode8.1 (OSX 10.12).

https://go-review.googlesource.com/#/c/33157/ changed httpGet to skip certificate verification on HTTP GET, but symbolization makes an extra request using HTTP POST, which silently fails.

https://github.com/google/pprof/blob/6addbe4/internal/symbolizer/symbolizer.go#L82

cc @bdarnell

EDIT: the real error is:

$ pprof https+insecure://127.0.0.1:8080/debug/pprof/profile
Fetching profile over HTTP from https+insecure://127.0.0.1:8080/debug/pprof/profile
pprof: http post https+insecure://127.0.0.1:8080/debug/pprof/symbol: Post https+insecure://127.0.0.1:8080/debug/pprof/symbol: unsupported protocol scheme "https+insecure"

We're getting TestHttpsInsecure failures on some (but not all) of the darwin builders.

OSX 10.11 and 10.12 are fine.
The OSX 10.8 and 10.10 builders fail. The darwin-arm-a1549ios builder is also failing.

This is the OSX error:

--- FAIL: TestHttpsInsecure (10.04s)
	fetch_test.go:305: grabProfile(https+insecure://127.0.0.1:59997/debug/pprof/profile) got PeriodType: cpu nanoseconds
		Period: 10000000
		Time: 2017-06-20 11:36:19.953808169 -0700 PDT
		Duration: 10.0
		Samples:
		samples/count cpu/nanoseconds
		        492 4920000000: 1 2 3 4 5 6 
		Locations
		     1: 0x105efd3 M=1 runtime.kevent /var/folders/6g/8k13h11n2zb8hxx99wk6f6y40000gn/T/workdir/go/src/runtime/sys_darwin_amd64.s:526 s=0
		     2: 0x102a672 M=1 runtime.netpoll /var/folders/6g/8k13h11n2zb8hxx99wk6f6y40000gn/T/workdir/go/src/runtime/netpoll_kqueue.go:79 s=0
		     3: 0x103405a M=1 runtime.findrunnable /var/folders/6g/8k13h11n2zb8hxx99wk6f6y40000gn/T/workdir/go/src/runtime/proc.go:2099 s=0
		     4: 0x10349eb M=1 runtime.schedule /var/folders/6g/8k13h11n2zb8hxx99wk6f6y40000gn/T/workdir/go/src/runtime/proc.go:2237 s=0
		     5: 0x1034d05 M=1 runtime.park_m /var/folders/6g/8k13h11n2zb8hxx99wk6f6y40000gn/T/workdir/go/src/runtime/proc.go:2300 s=0
		     6: 0x105aeda M=1 runtime.mcall /var/folders/6g/8k13h11n2zb8hxx99wk6f6y40000gn/T/workdir/go/src/runtime/asm_amd64.s:286 s=0
		Mappings
		1: 0x0/0x0/0x0   [FN]
		, want function "TestHttpsInsecure"
FAIL
FAIL	cmd/vendor/github.com/google/pprof/internal/driver	10.545s

Full logs: OSX 10.10, OSX 10.8, iphone6. The last one is a little different.

As a Go newbie, I've been configuring both my $GOROOT and $GOPATH as /go. However, pprof refuses to build with this setup.

$ go get github.com/google/pprof
warning: GOPATH set to GOROOT (/go) has no effect
package github.com/google/pprof: cannot download, $GOPATH must not be set to $GOROOT. For more details see: go help gopath

I tried mkdir -p /go-special && export GOPATH=/go-special, but the pprof build/installation process doesn't like that either.

Could the README please be more explicit about the right way to do this?

Since changeset f90721db3d, TestSymbolizationPath is failing on Plan 9.

--- FAIL: TestSymbolizationPath (0.02s)
	fetch_test.go:79: :/prod/path/binary:abcde10001, want /tmp/home918705075/pprof/binaries/abcde10001/binary, got /prod/path/binary
FAIL
FAIL	cmd/vendor/github.com/google/pprof/internal/driver	0.484s

The test is failing because internal/driver/fetch.go has been changed to handle
the correct home directory environment variable on Plan 9 and Windows,
but internal/driver/fetch.go still uses the $HOME environment variable.

On reports, pprof uses B/kB/MB/GB to represent 1/2^10/2^20/2^30, which as at least one user pointed out is ambiguous at best. We should probably remap these to B/KiB/MiB/GiB, which should be more accurate. The only compat concerns would be for people that parse pprof's output, and we offer no guarantees not to break them.

The other place where we use units is on the unit fields in profile.proto and the command line override, where we recognize kb/mb/gb to mean kiB, MiB, GiB, etc... This is also ambiguous at best, and problematic because changing it could cause us to stop parsing existing profiles. In reality I don't know if we have any profile sources that currently generate profiles with anything other than 'b', but if there are we would break them. A quick search over the accessible code shows only profiles generated using 'bytes', so this is probably worthwhile to do in general. It would only affect people that are used to specifying units by hand to pprof. For example, pprof -unit=mb would change from MiB to MB, and instead users would have to say pprof -unit=MiB (or maybe pprof -unit=mib).

(again, silly zipped profile for github naming)

pprof -raw heapz
Samples:
objects/count space/bytes[dflt]
       7438   60936269: 1 2 3 4 5 6
                bytes:[8192] request:[8191]
        238   62466710: 7 8 9 10 5 6
                bytes:[262144] request:[262143]
       3341   54739221: 7 8 9 10 5 6
                bytes:[16384] request:[16383]
       2064   67634517: 1 2 3 4 5 6
                bytes:[32768] request:[32767]
      29710   60847108: 1 2 3 4 5 6
                bytes:[2048] request:[2047]
        396   51920894: 1 2 3 4 5 6
                bytes:[131072] request:[131071]
      63503   65027585: 7 8 9 10 5 6
                bytes:[1024] request:[1023]
       1105   72423082: 7 8 9 10 5 6
                bytes:[65536] request:[65535]
        108   56884975: 11 12 13 3 4 5 6
                bytes:[524288] request:[524287]
      15887   65075220: 7 8 9 10 5 6
                bytes:[4096] request:[4095]

Now let's try to filter to some of these. Each of the below queries should match exactly one entry and do (this is good)

pprof -traces -tagfocus=4096bytes heapz
pprof -traces -tagfocus=4095request heapz
pprof -traces -tagfocus=4kb heapz
pprof -traces -tagfocus=512kb heapz
pprof -traces -tagfocus=524288bytes: heapz

But this matches everything:

pprof -traces -tagfocus=512kb: heapz

Note that 524288 bytes = 512kb. But for some reason when we have a non-empty range that breaks. That's a flat bug. :1kb seems to work, otoh, as does 1kb:4kb.

A slightly different problem is that there's no easy way to specify request ranges without having to write out full numbers. I'd be very happy if I could specify, say:

-tagfocus=/request/256kb:512kb

to mean "I want to see a tag, whose name matches "request", whose value is between 256kb and 512kb.

heapz.zip

I'm trying to update the vendored pprof in the Go repository, and the nacl-386 and nacl-amd64p32 are failing with the following error:

--- FAIL: TestParseData (0.37s)
	fuzz_test.go:14: Problem reading directory testdata/ : open testdata/: No such file or directory
FAIL
FAIL	cmd/vendor/github.com/google/pprof/fuzz	0.443s
2017/08/20 11:28:07 Failed: exit status 1

All the other trybots are fine.

The vendored pprof commit is 2191094.

Full nacl-386 log
Link the Go change

perf report --symfs

pprof looks up binaries in the directory specified by env var PPROF_BINARY_PATH but does lookup based on filename of the binary and full path.

Example: If perf recording has binary /A/B/C.so and symfs path is /X/Y/Z, perf tries to search the binary in /X/Y/Z/A/B/C.so but pprof tries to search the binary in /X/Y/Z/C.so

It would be nice to have a --symfs like option for pprof so that both perf and pprof can be used with the same symfs directory.

Mentioned offhand in #153:

A slightly different problem is that there's no easy way to specify request ranges without having to write out full numbers. I'd be very happy if I could specify, say:

-tagfocus=/request/256kb:512kb

to mean "I want to see a tag, whose name matches "request", whose value is between 256kb and 512kb.

If we can't manage the unit conversion and I have to specify /request/262144:524288, I can work with that for now (though it'd be very nice....)

This is becoming more important as we add more tags to generated profiles, some of which aren't really comparable; they're not tweaked values, they're things with totally different meanings. (think "requested alignment"; I do not want tagfocus=:512 to pick up every unaligned alloc even if it was a 2 MiB request!

Hi,

I'm trying to use pprof and I'm getting strange errors:

[aaron@TC ruby (anon_ic)]$ ~/.local/go/bin/pprof -text ruby y.prof
ruby: parsing profile: unrecognized profile format
fetched 1 profiles out of 2
Local symbolization failed for ruby: Parsing /Users/aaron/git/ruby/./ruby: bad magic number '[207 250 237 254]' in record at byte 0x0
Local symbolization failed for libdyld.dylib: Parsing /usr/lib/system/libdyld.dylib: bad magic number '[202 254 186 190]' in record at byte 0x0
Local symbolization failed for libsystem_kernel.dylib: Parsing /usr/lib/system/libsystem_kernel.dylib: bad magic number '[202 254 186 190]' in record at byte 0x0
Local symbolization failed for libsystem_malloc.dylib: Parsing /usr/lib/system/libsystem_malloc.dylib: bad magic number '[202 254 186 190]' in record at byte 0x0
Some binary filenames not available. Symbolization may be incomplete.
File: stringio.bundle
Type: cpu
Showing nodes accounting for 60ms, 100% of 60ms total
      flat  flat%   sum%        cum   cum%
      30ms 50.00% 50.00%       30ms 50.00%  [libsystem_kernel.dylib]
      20ms 33.33% 83.33%       50ms 83.33%  [ruby]
      10ms 16.67%   100%       10ms 16.67%  [libsystem_malloc.dylib]
         0     0%   100%       50ms 83.33%  <unknown>
         0     0%   100%       50ms 83.33%  [libdyld.dylib]
[aaron@TC ruby (anon_ic)]$

I'm trying to do profiling against my Ruby binary, but I'm getting bad magic number errors. I'm using gperftools version 2.5 on OS X. Any help would be greatly appreciated. Thanks!

Currently pprof tops out at GB (Well, GiB: #136 :P). I have profiles containing quantities much larger than 2^30 bytes.

Please add TiB and PiB.

I am attempting to use pprof inside of the docker container image golang:1.7-alpine.

This is the error I get:

bash-4.3# go tool pprof -svg -output=/pprof.svg http://host:port/debug/pprof/heap
Fetching profile from http://host:port/debug/pprof/heap
Could not save profile: open /root/pprof/pprof.host:port.inuse_objects.inuse_space.001.pb.gz: protocol error
Generating report in /pprof.svg
bash-4.3#

The svg file seems to work fine, but the pb.gz file is not created.

I've tried installing things, including protobuf, graphviz, gzip, etc.
Not sure what is missing or needed, or why this error is occurring.

I don't know if this is an actual bug, a documentation bug, or neither but I hit an interesting corner case trying to collect pprof measurements that I wanted to report. While working with https://github.com/urfave/cli I ran into urfave/cli#551 trying to collect pprof data.

Is pprof collection expected to work on code that calls into a reflected function? If so, I don't believe it does so there may be a real bug. If not it would be helpful if the docs mentioned this limitation.

Thanks.

I believe that this repo is the code that powers go tool pprof. If I'm wrong on that count, disregard everything else I say.

I'm using pprof on macOS 10.12.1 with go 1.7.3. I'm constantly using it in interactive mode, and it would be really nice to be able to call back previous commands with the "up" arrow key. Instead, of course, it prints ^[[A, which I assume means that this isn't supported?

I completely understand if this isn't worth the development team's time, but I thought I would suggest it.

Hi
I am trying to project profile on web. When I use "web" command I get this error.

(pprof) web
(pprof) Cannot open display:
Run '/usr/bin/eog --help' to see a full list of available command line options.
Error: no "view" rule for type "image/svg+xml" passed its test case
(for more information, add "--debug=1" on the command line)
Can't call method "get_value" on an undefined value at /usr/bin/mimeopen line 162.
/usr/bin/xdg-open: line 563: links2: not found
/usr/bin/xdg-open: line 563: links: not found

I'm on the current pprof version.

Open the attached cpu profile (generated by a go test -bench=. -cpuprofile=cpu.prof command on the current tip version of go) with -call_tree activated:

$ pprof -call_tree cpu.prof

Call top:

(pprof) top
panic: TrimTree only works on trees

goroutine 1 [running]:
github.com/google/pprof/internal/graph.(*Graph).TrimTree(0xc420380d00, 0xc4203e96b0)
	/home/alberto/gocode/src/github.com/google/pprof/internal/graph/graph.go:431 +0x6e7
github.com/google/pprof/internal/report.(*Report).newTrimmedGraph(0xc420380c40, 0x7f0240, 0xc420047a00, 0x11, 0xc4200f4d70)
	/home/alberto/gocode/src/github.com/google/pprof/internal/report/report.go:139 +0x136
github.com/google/pprof/internal/report.printText(0xa2db40, 0xc420168150, 0xc420380c40, 0xc420047af0, 0x774e40)
	/home/alberto/gocode/src/github.com/google/pprof/internal/report/report.go:682 +0x43
github.com/google/pprof/internal/report.Generate(0xa2db40, 0xc420168150, 0xc420380c40, 0xa30a40, 0xc42006d5c0, 0x0, 0x0)
	/home/alberto/gocode/src/github.com/google/pprof/internal/report/report.go:94 +0x2e0
github.com/google/pprof/internal/driver.generateReport(0xc4200ff2c0, 0xc42040eba0, 0x1, 0x1, 0xc42040d5c0, 0xc42006d560, 0xc42040d5c0, 0x0)
	/home/alberto/gocode/src/github.com/google/pprof/internal/driver/driver.go:99 +0x249
github.com/google/pprof/internal/driver.interactive(0xc4200feb40, 0xc42006d560, 0xc4200feb40, 0x0)
	/home/alberto/gocode/src/github.com/google/pprof/internal/driver/interactive.go:111 +0x62c
github.com/google/pprof/internal/driver.PProf(0xc4200efe48, 0x0, 0x0)
	/home/alberto/gocode/src/github.com/google/pprof/internal/driver/driver.go:55 +0x154
github.com/google/pprof/driver.PProf(0xc4200eff18, 0xc420138620, 0xc420127c40)
	/home/alberto/gocode/src/github.com/google/pprof/driver/driver.go:32 +0x1c7
main.main()
	/home/alberto/gocode/src/github.com/google/pprof/pprof.go:27 +0x5c

cpu.zip

no heap profile info. only MAPPED_LIBRARIES info.
as follows

It is sometimes desirable to be able to associate a custom comment with a profile. There is "comment" field in the profile proto, so it would be good to have a -comment flag that can be used like:

pprof.go -comment "My comment" localhost:8080

or

pprof.go -comment "My comment" -proto -output new-profile.pb.gz profile.pb.gz

In the former case of remote collection, pprof stores the remote profile in a local file printing its path, the comment should become a part of that profile. Storing the address of the remote source as a comment for the remote collections would also be good.

If a profile has a drop frame matching against an anonymous namespace, the match never happens. I think the culprit is this line (https://github.com/google/pprof/blob/master/profile/prune.go#L39), which checks for brackets before applying the drop frame RE, and only applies the RE to the symbol before the brackets. I'm assuming this is for symbols that still have parameters.

Here's an example [1], where I create a profile with a drop frame RE "Foo::.*::Bar". If I comment that line out, things behave as expected [2].

Since this is already a hack, I'll just special-case "(anonymous namespace)", but there are probably other things with a legit "(" as a part of the function name (std::function objects?).

[1] $ pprof -traces tmp.pb
...
-----------+-------------------------------------------------------
15B
Foo::(anonymous namespace)::Test::Bar
Foo::(anonymous namespace)::Test::Baz
main
__libc_start_main
_start
-----------+-------------------------------------------------------

[2] $ ./pprof -traces tmp.pb
...
-----------+-------------------------------------------------------
15B Foo::(anonymous namespace)::Test::Baz
main
__libc_start_main
_start
-----------+-------------------------------------------------------

I ran pprof -http :9090 on a MacOS host, and when it opened in my browser it just shows "permission denied".

Adding the host to the error message reveals that host is ::1, not [::1] as expected by isLocalhost.

https://github.com/golang/go/tree/master/src/cmd/pprof
My understanding that they are historically related, but no longer share code. A note in README would be useful.

I am using go1.7 to bootstrap go1.8, install from source, since my system CentOS/RHEL 5.x not supported

I think those env-variables are correct since it could be successfully installed. But in my install-golang-script, it will roughly 30% fail, after I run ./all.bash.

All of the fail is in testing packages stage due to

 *** Test killed with quit: ran too long (4m0s).
FAIL	runtime/pprof	240.050s

I tried to set timeout longer, like this:
export GO_TEST_TIMEOUT_SCALE=10,
but the result is it will stuck at this stage, more than 10 minutes.

I also tried to set high resolution timer, but still didn't work.
export CONFIG_HIGH_RES_TIMERS=y

Anyone know how to fix it ?

I use gperftools-2.5 to trace java off-heap memory leak, but I can't find which method called Unsafe_AllocateMemory.

I know that perf can get java stack using Mixed-Mode. I tried that with pprof but without luck. Is it possible to get the java stack frame in pprof?

also @brendangregg

i meet following proc map:

7fd3558d0000-7fd3559b8000 r-xp 00000000 00:00 106546 /usr/lib64/libstdc++.so.6.0.13
7fd3559b8000-7fd355bb8000 ---p 000e8000 00:00 106546 /usr/lib64/libstdc++.so.6.0.13
7fd355bb8000-7fd355bbf000 r-xp 000e8000 00:00 106546 /usr/lib64/libstdc++.so.6.0.13
7fd355bbf000-7fd355bc1000 rwxp 000ef000 00:00 106546 /usr/lib64/libstdc++.so.6.0.13
ParseLibraries() with merge them into:

$libname=/usr/lib64/libstdc++.so.6.0.13
$start=0x00007fd3558d0000
$finish=0x00007fd355bc1000
$offset=0x00000000000ef000
the later AddressSub() will get wrong address because offset 0xef000 is incorrect for first segment.

In addition to title and labels, a DotConfig should allow a url to be specified for the legend box.

Am i able to use svg library from pprof to draw random graphs in svg format? Like object graph?

pprof output currently includes a line of the form:

Duration: 1.60s, Total samples = 14.50ms (  0.9%)

The "Total samples" part is confusing. Milliseconds are continuous, but samples are discrete — they're individual points, so how can you sum them up into a quantity of milliseconds?

I seem to recall that an earlier (pre-GitHub) version of pprof reported the number of samples as an actual number of samples. Can we please go back to that? Otherwise, please update the formatting of the Duration line to give a clearer indication of what a quantity of samples reported in milliseconds actually means.

while running such a command
go tool pprof -callgrind profile.log > profile.callgrind
pprof seems to be hanging.

System: Go 1.8, Xenial 16.04, linux/amd64

As a follow-up from #180, we want to be able to give the users a convenient way to start a new HTTP server at localhost:<random_port>.

There seems to be no good way to do it without a new flag. Empty -http value is conflicting with the positional argument.

$ pprof -http profile.out

I propose us to add a new flag (e.g. browse) that starts a server locally at a random port and opens the page in the browser.

/cc @aalexand

pprof ./m "/tmp/m.46192.main-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv
Using local file ./m.
Using local file /tmp/m.46192.main-end.heap.
gv: Unable to open the display.

already installed gv.

Depending on the behavior of my profiled binary, libprofiler outputs a profiling file (see attached) that makes pprof hangs. I wasn't able to easily strace where it's stuck.

Tested on linux. Profiled executable built with clang 3.7+

profile.txt (.txt for the sake of github upload)

These should either not be generated, or generated in a test directory and cleaned up.

I.e.

1. get latest gperftools release
2. ./configure && make -j8 check
3. one of tests is sampling_test.sh which saves /tmp/sampling_test_dir/out.growth
4. perl pprof handles this file while golang version doesn't

We cannot drop perl version until this is fixed.

The issue was noticed in #111.

Test server:

package main

import (
  "log"
  "net/http"
  _ "net/http/pprof"
)

func run() {
  for {
  }
}

func main() {
  go func() {
    log.Println(http.ListenAndServe("localhost:6060", nil))
  }()
  run()
}

Running it as go run server.go and profiling as "pprof -symbolize=remote localhost:6060/debug/pprof/profile?seconds=5" produces unknown symbol on OSX with Go 1.8. Doing the same with Go 1.7.5 produces correctly symbolized output.

Looking at "-raw" output for the Go 1.8 vs. 1.7.5 profiles, there is a difference in the mapping section:

Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile?seconds=5
Saved profile in /Users/aalexand/pprof/pprof.samples.cpu.011.pb.gz
Type: cpu
Time: Mar 15, 2017 at 1:08pm (PDT)
Duration: 5s, Total samples = 4.29s (85.74%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) raw
PeriodType: cpu nanoseconds
Period: 10000000
Time: 2017-03-15 13:08:32.787861506 -0700 PDT
Duration: 5.00
Samples:
samples/count cpu/nanoseconds
         83  830000000: 1 2 3 4 
          1   10000000: 5 6 7 3 4 
        345 3450000000: 8 9 10 11 
Locations
     1: 0x105adc6 
     2: 0x1037612 
     3: 0x10309cd 
     4: 0x10308a3 
     5: 0x105abc6 
     6: 0x105abf4 
     7: 0x1037726 
     8: 0x1311630 
     9: 0x1311679 
    10: 0x102dad9 
    11: 0x1059af0 
Mappings

with Go 1.8 vs.

Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile?seconds=5
Saved profile in /Users/aalexand/pprof/pprof.samples.cpu.012.pb.gz
Type: cpu
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) raw
PeriodType: cpu nanoseconds
Period: 10000000
Samples:
samples/count cpu/nanoseconds
        388 3880000000: 1 2 3 4 
         37  370000000: 5 6 7 8 
Locations
     1: 0x2040 M=1 main.run :0 s=0
     2: 0x2089 M=1 main.main :0 s=0
     3: 0x2ddd3 M=1 runtime.main :0 s=0
     4: 0x5d930 M=1 runtime.goexit :0 s=0
     5: 0x5e8b6 M=1 runtime.usleep :0 s=0
     6: 0x377d1 M=1 runtime.sysmon :0 s=0
     7: 0x30c1d M=1 runtime.mstart1 :0 s=0
     8: 0x30af3 M=1 runtime.mstart :0 s=0
Mappings
1: 0x0/0xffffffffffffffff/0x0   [FN]

with Go 1.7.5.

Doing the same on Linux, produces

Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile?seconds=5
Saved profile in /usr/local/google/home/aalexand/pprof/pprof.server.samples.cpu.005.pb.gz
File: server
Type: cpu
Time: Mar 15, 2017 at 1:16pm (PDT)
Duration: 5s, Total samples = 4.99s (99.79%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) raw
PeriodType: cpu nanoseconds
Period: 10000000
Time: 2017-03-15 13:16:34.995513521 -0700 PDT
Duration: 5.00
Samples:
samples/count cpu/nanoseconds
        499 4990000000: 1 2 3 4 
Locations
     1: 0x70fb00 M=1 main.run :0 s=0
     2: 0x70fb49 M=1 main.main :0 s=0
     3: 0x42dab9 M=1 runtime.main :0 s=0
     4: 0x45a4f0 M=1 runtime.goexit :0 s=0
Mappings
1: 0x400000/0x710000/0x0 /tmp/go-build929989772/command-line-arguments/_obj/exe/server  [FN]

and

Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile?seconds=5
Saved profile in /usr/local/google/home/aalexand/pprof/pprof.server.samples.cpu.004.pb.gz
File: server
Type: cpu
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) raw
PeriodType: cpu nanoseconds
Period: 10000000
Samples:
samples/count cpu/nanoseconds
        500 5000000000: 1 2 
Locations
     1: 0x401000 M=1 main.run :0 s=0
     2: 0x45ddc0 M=1 runtime.goexit :0 s=0
Mappings
1: 0x400000/0x699000/0x0 /tmp/go-build092539107/command-line-arguments/_obj/exe/server  [FN]

respectively that are both correctly symbolized profiles.