Alright, well, let’s ignore temporarily the difference between caller and callee call-graphs, mostly because when I compare the results between these two options on my machine, I only see effects inside the kernel.kallsyms DSO for reasons I don’t understand — relatively new to this myself.
I found that for your example, it’s a little easier to read the whole tree. So, using --stdio, let’s look at the whole tree for __sin_sse2:
# Overhead Command Shared Object Symbol
# ........ ......... ................. ......................
#
94.72% perf-test libm-2.19.so [.] __sin_sse2
|
--- __sin_sse2
|
|--44.20%-- foo
| |
| --100.00%-- main
| __libc_start_main
| _start
| 0x0
|
|--27.95%-- baz
| |
| |--51.78%-- bar
| | foo
| | main
| | __libc_start_main
| | _start
| | 0x0
| |
| --48.22%-- foo
| main
| __libc_start_main
| _start
| 0x0
|
--27.84%-- bar
|
--100.00%-- foo
main
__libc_start_main
_start
0x0
So, the way I read this is: 44% of the time, sin is called from foo; 27% of the time it’s called from baz, and 27% from bar.
The documentation for -g is instructive:
-g [type,min[,limit],order[,key]], --call-graph
Display call chains using type, min percent threshold, optional print limit and order. type can be either:
· flat: single column, linear exposure of call chains.
· graph: use a graph tree, displaying absolute overhead rates.
· fractal: like graph, but displays relative rates. Each branch of the tree is considered as a new profiled object.
order can be either:
- callee: callee based call graph.
- caller: inverted caller based call graph.
key can be:
- function: compare on functions
- address: compare on individual code addresses
Default: fractal,0.5,callee,function.
The important piece here is that the default is fractal, and in fractal mode, each branch is a new object.
So, you can see that 50% of the time that baz is called, it’s called from bar, and the other 50% it’s called from foo.
This isn’t always the most useful measure, so it’s instructive to look at the results using -g graph:
94.72% perf-test libm-2.19.so [.] __sin_sse2
|
--- __sin_sse2
|
|--41.87%-- foo
| |
| --41.48%-- main
| __libc_start_main
| _start
| 0x0
|
|--26.48%-- baz
| |
| |--13.50%-- bar
| | foo
| | main
| | __libc_start_main
| | _start
| | 0x0
| |
| --12.57%-- foo
| main
| __libc_start_main
| _start
| 0x0
|
--26.38%-- bar
|
--26.17%-- foo
main
__libc_start_main
_start
0x0
This changes to using absolute percentages, where each percentage of time is reported for that call chain: So foo->bar is 26% of the total ticks (which in turn calls baz), and foo->baz (direct) is 12% of the total ticks.
I still have no idea why I don’t see any differences between callee and caller graphs though, from the perspective of __sin_sse2.
Update
One thing I did change from your command line is how the callgraphs were gathered. Linux perf by default uses the frame pointer method of reconstructing callstacks. This can be a problem when the compiler uses -fomit-frame-pointer as a default. So I used
perf record --call-graph dwarf ./perf-test