Showing preview only (545K chars total). Download the full file or copy to clipboard to get everything.
Repository: brendangregg/perf-tools
Branch: master
Commit: cb42511b963a
Files: 60
Total size: 523.1 KB
Directory structure:
gitextract_cv8l2gg4/
├── LICENSE
├── README.md
├── deprecated/
│ ├── README.md
│ ├── execsnoop-proc
│ ├── execsnoop-proc.8
│ └── execsnoop-proc_example.txt
├── disk/
│ └── bitesize
├── examples/
│ ├── bitesize_example.txt
│ ├── cachestat_example.txt
│ ├── execsnoop_example.txt
│ ├── funccount_example.txt
│ ├── funcgraph_example.txt
│ ├── funcslower_example.txt
│ ├── functrace_example.txt
│ ├── iolatency_example.txt
│ ├── iosnoop_example.txt
│ ├── killsnoop_example.txt
│ ├── kprobe_example.txt
│ ├── opensnoop_example.txt
│ ├── perf-stat-hist_example.txt
│ ├── reset-ftrace_example.txt
│ ├── syscount_example.txt
│ ├── tcpretrans_example.txt
│ ├── tpoint_example.txt
│ └── uprobe_example.txt
├── execsnoop
├── fs/
│ └── cachestat
├── iolatency
├── iosnoop
├── kernel/
│ ├── funccount
│ ├── funcgraph
│ ├── funcslower
│ ├── functrace
│ └── kprobe
├── killsnoop
├── man/
│ └── man8/
│ ├── bitesize.8
│ ├── cachestat.8
│ ├── execsnoop.8
│ ├── funccount.8
│ ├── funcgraph.8
│ ├── funcslower.8
│ ├── functrace.8
│ ├── iolatency.8
│ ├── iosnoop.8
│ ├── killsnoop.8
│ ├── kprobe.8
│ ├── opensnoop.8
│ ├── perf-stat-hist.8
│ ├── reset-ftrace.8
│ ├── syscount.8
│ ├── tcpretrans.8
│ ├── tpoint.8
│ └── uprobe.8
├── misc/
│ └── perf-stat-hist
├── net/
│ └── tcpretrans
├── opensnoop
├── syscount
├── system/
│ └── tpoint
├── tools/
│ └── reset-ftrace
└── user/
└── uprobe
================================================
FILE CONTENTS
================================================
================================================
FILE: LICENSE
================================================
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc., <http://fsf.org/>
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
{description}
Copyright (C) {year} {fullname}
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
{signature of Ty Coon}, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.
================================================
FILE: README.md
================================================
perf-tools
==========
A miscellaneous collection of in-development and unsupported performance analysis tools for Linux ftrace and perf_events (aka the "perf" command). Both ftrace and perf are core Linux tracing tools, included in the kernel source. Your system probably has ftrace already, and perf is often just a package add (see Prerequisites).
These tools are designed to be easy to install (fewest dependencies), provide advanced performance observability, and be simple to use: do one thing and do it well. This collection was created by Brendan Gregg (author of the DTraceToolkit).
Many of these tools employ workarounds so that functionality is possible on existing Linux kernels. Because of this, many tools have caveats (see man pages), and their implementation should be considered a placeholder until future kernel features, or new tracing subsystems, are added.
These are intended for Linux 3.2 and newer kernels. For Linux 2.6.x, see Warnings.
## Presentation
These tools were introduced in the USENIX LISA 2014 presentation: Linux Performance Analysis: New Tools and Old Secrets
- slides: http://www.slideshare.net/brendangregg/linux-performance-analysis-new-tools-and-old-secrets
- video: https://www.usenix.org/conference/lisa14/conference-program/presentation/gregg
## Contents
<center><a href="images/perf-tools_2016.png"><img src="images/perf-tools_2016.png" border=0 width=700></a></center>
Using ftrace:
- [iosnoop](iosnoop): trace disk I/O with details including latency. [Examples](examples/iosnoop_example.txt).
- [iolatency](iolatency): summarize disk I/O latency as a histogram. [Examples](examples/iolatency_example.txt).
- [execsnoop](execsnoop): trace process exec() with command line argument details. [Examples](examples/execsnoop_example.txt).
- [opensnoop](opensnoop): trace open() syscalls showing filenames. [Examples](examples/opensnoop_example.txt).
- [killsnoop](killsnoop): trace kill() signals showing process and signal details. [Examples](examples/killsnoop_example.txt).
- fs/[cachestat](fs/cachestat): basic cache hit/miss statistics for the Linux page cache. [Examples](examples/cachestat_example.txt).
- net/[tcpretrans](net/tcpretrans): show TCP retransmits, with address and other details. [Examples](examples/tcpretrans_example.txt).
- system/[tpoint](system/tpoint): trace a given tracepoint. [Examples](examples/tpoint_example.txt).
- kernel/[funccount](kernel/funccount): count kernel function calls, matching a string with wildcards. [Examples](examples/funccount_example.txt).
- kernel/[functrace](kernel/functrace): trace kernel function calls, matching a string with wildcards. [Examples](examples/functrace_example.txt).
- kernel/[funcslower](kernel/funcslower): trace kernel functions slower than a threshold. [Examples](examples/funcslower_example.txt).
- kernel/[funcgraph](kernel/funcgraph): trace a graph of kernel function calls, showing children and times. [Examples](examples/funcgraph_example.txt).
- kernel/[kprobe](kernel/kprobe): dynamically trace a kernel function call or its return, with variables. [Examples](examples/kprobe_example.txt).
- user/[uprobe](user/uprobe): dynamically trace a user-level function call or its return, with variables. [Examples](examples/uprobe_example.txt).
- tools/[reset-ftrace](tools/reset-ftrace): reset ftrace state if needed. [Examples](examples/reset-ftrace_example.txt).
Using perf_events:
- misc/[perf-stat-hist](misc/perf-stat-hist): power-of aggregations for tracepoint variables. [Examples](examples/perf-stat-hist_example.txt).
- [syscount](syscount): count syscalls by syscall or process. [Examples](examples/syscount_example.txt).
- disk/[bitesize](disk/bitesize): histogram summary of disk I/O size. [Examples](examples/bitesize_example.txt).
Using eBPF:
- As a preview of things to come, see the bcc tracing [Tools section](https://github.com/iovisor/bcc/blob/master/README.md#tracing). These use [bcc](https://github.com/iovisor/bcc), a front end for using [eBPF](http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html). bcc+eBPF will allow some of these tools to be rewritten and improved, and additional tools to be created.
## Screenshots
Showing new processes and arguments:
<pre># <b>./execsnoop</b>
Tracing exec()s. Ctrl-C to end.
PID PPID ARGS
22898 22004 man ls
22905 22898 preconv -e UTF-8
22908 22898 pager -s
22907 22898 nroff -mandoc -rLL=164n -rLT=164n -Tutf8
22906 22898 tbl
22911 22910 locale charmap
22912 22907 groff -mtty-char -Tutf8 -mandoc -rLL=164n -rLT=164n
22913 22912 troff -mtty-char -mandoc -rLL=164n -rLT=164n -Tutf8
22914 22912 grotty
</pre>
Measuring block device I/O latency from queue insert to completion:
<pre># <b>./iolatency -Q</b>
Tracing block I/O. Output every 1 seconds. Ctrl-C to end.
>=(ms) .. <(ms) : I/O |Distribution |
0 -> 1 : 1913 |######################################|
1 -> 2 : 438 |######### |
2 -> 4 : 100 |## |
4 -> 8 : 145 |### |
8 -> 16 : 43 |# |
16 -> 32 : 43 |# |
32 -> 64 : 1 |# |
[...]
</pre>
Tracing the block:block_rq_insert tracepoint, with kernel stack traces, and only for reads:
<pre># <b>./tpoint -s block:block_rq_insert 'rwbs ~ "*R*"'</b>
cksum-11908 [000] d... 7269839.919098: block_rq_insert: 202,1 R 0 () 736560 + 136 [cksum]
cksum-11908 [000] d... 7269839.919107: <stack trace>
=> __elv_add_request
=> blk_flush_plug_list
=> blk_finish_plug
=> __do_page_cache_readahead
=> ondemand_readahead
=> page_cache_async_readahead
=> generic_file_read_iter
=> new_sync_read
=> vfs_read
=> SyS_read
=> system_call_fastpath
[...]
</pre>
Count kernel function calls beginning with "bio_", summarize every second:
<pre># <b>./funccount -i 1 'bio_*'</b>
Tracing "bio_*"... Ctrl-C to end.
FUNC COUNT
bio_attempt_back_merge 26
bio_get_nr_vecs 361
bio_alloc 536
bio_alloc_bioset 536
bio_endio 536
bio_free 536
bio_fs_destructor 536
bio_init 536
bio_integrity_enabled 536
bio_put 729
bio_add_page 1004
[...]
</pre>
There are many more examples in the [examples](examples) directory. Also see the [man pages](man/man8).
## Prerequisites
The intent is as few as possible. Eg, a Linux 3.2 server without debuginfo. See the tool man page for specifics.
### ftrace
FTRACE configured in the kernel. You may already have this configured and available in your kernel version, as FTRACE was first added in 2.6.27. This requires CONFIG_FTRACE and other FTRACE options depending on the tool. Some tools (eg, funccount) require CONFIG_FUNCTION_PROFILER.
### perf_events
Requires the "perf" command to be installed. This is in the linux-tools-common package. After installing that, perf may tell you to install an additional linux-tools package (linux-tools-_kernel_version_). perf can also be built under tools/perf in the kernel source. See [perf_events Prerequisites](http://www.brendangregg.com/perf.html#Prerequisites) for more details about getting perf_events to work fully.
### debugfs
Requires a kernel with CONFIG_DEBUG_FS option enabled. As with FTRACE, this may already be enabled (debugfs was added in 2.6.10-rc3). The debugfs also needs to be mounted:
```
# mount -t debugfs none /sys/kernel/debug
```
### awk
Many of there scripts use awk, and will try to use either mawk or gawk depending on the desired behavior: mawk for buffered output (because of its speed), and gawk for synchronous output (as fflush() works, allowing more efficient grouping of writes).
## Install
These are just scripts. Either grab everything:
```
git clone --depth 1 https://github.com/brendangregg/perf-tools
```
Or use the raw links on github to download individual scripts. Eg:
```
wget https://raw.githubusercontent.com/brendangregg/perf-tools/master/iosnoop
```
This preserves tabs (which copy-n-paste can mess up).
## Warnings
Ftrace was first added to Linux 2.6.27, and perf_events to Linux 2.6.31. These early versions had kernel bugs, and lockups and panics have been reported on 2.6.32 series kernels. This includes CentOS 6.x. If you must analyze older kernels, these tools may only be useful in a fault-tolerant environment, such as a lab with simulated issues. These tools have been primarily developed on Linux 3.2 and later kernels.
Depending on the tool, there may also be overhead incurred. See the next section.
## Internals and Overhead
perf_events is evolving. This collection began development circa Linux 3.16, with Linux 3.2 servers as the main target, at a time when perf_events lacks certain programmatic capabilities (eg, custom in-kernel aggregations). It's possible these will be added in a forthcoming kernel release. Until then, many of these tools employ workarounds, tricks, and hacks in order to work. Some of these tools pass event data to user space for post-processing, which costs much higher overhead than in-kernel aggregations. The overhead of each tool is described in its man page.
__WARNING__: In _extreme_ cases, your target application may run 5x slower when using these tools. Depending on the tool and kernel version, there may also be the risk of kernel panics. Read the program header for warnings, and test before use.
If the overhead is a problem, these tools can be improved. If a tool doesn't already, it could be rewritten in C to use perf_events_open() and mmap() for the trace buffer. It could also implement frequency counts in C, and operate on mmap() directly, rather than using awk/Perl/Python. Additional improvements are possible for ftrace-based tools, such as use of snapshots and per-instance buffers.
Some of these tools are intended as short-term workarounds until more kernel capabilities exist, at which point they can be substantially rewritten. Older versions of these tools will be kept in this repository, for older kernel versions.
As my main target is a fleet of Linux 3.2 servers that do not have debuginfo, these tools try not to require it. At times, this makes the tool more brittle than it needs to be, as I'm employing workarounds (that may be kernel version and platform specific) instead of using debuginfo information (which can be generic). See the man page for detailed prerequisites for each tool.
I've tried to use perf_events ("perf") where possible, since that interface has been developed for multi-user use. For various reasons I've often needed to use ftrace instead. ftrace is surprisingly powerful (thanks Steven Rostedt!), and not all of its features are exposed via perf, or in common usage. This tool collection is in some ways a demonstration of hidden Linux features using ftrace.
Since things are changing, it's very possible you may find some tools don't work on your Linux kernel version. Some expertise and assembly will be required to fix them.
## Links
A case study and summary:
- 13 Aug 2014: http://lwn.net/Articles/608497 Ftrace: The hidden light switch
Related articles:
- 28 Jun 2015: http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html
- 31 Dec 2014: http://www.brendangregg.com/blog/2014-12-31/linux-page-cache-hit-ratio.html
- 06 Sep 2014: http://www.brendangregg.com/blog/2014-09-06/linux-ftrace-tcp-retransmit-tracing.html
- 28 Jul 2014: http://www.brendangregg.com/blog/2014-07-28/execsnoop-for-linux.html
- 25 Jul 2014: http://www.brendangregg.com/blog/2014-07-25/opensnoop-for-linux.html
- 23 Jul 2014: http://www.brendangregg.com/blog/2014-07-23/linux-iosnoop-latency-heat-maps.html
- 16 Jul 2014: http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html
- 10 Jul 2014: http://www.brendangregg.com/blog/2014-07-10/perf-hacktogram.html
================================================
FILE: deprecated/README.md
================================================
Deprecated versions of tools.
================================================
FILE: deprecated/execsnoop-proc
================================================
#!/usr/bin/perl
#
# execsnoop - trace process exec() with arguments. /proc version.
# Written using Linux ftrace.
#
# This shows the execution of new processes, especially short-lived ones that
# can be missed by sampling tools such as top(1).
#
# USAGE: ./execsnoop [-h] [-n name]
#
# REQUIREMENTS: FTRACE CONFIG, sched:sched_process_exec tracepoint (you may
# already have these on recent kernels), and Perl.
#
# This traces exec() from the fork()->exec() sequence, which means it won't
# catch new processes that only fork(), and, it will catch processes that
# re-exec. This instruments sched:sched_process_exec without buffering, and then
# in user-space (this program) reads PPID and process arguments asynchronously
# from /proc.
#
# If the process traced is very short-lived, this program may miss reading
# arguments and PPID details. In that case, "<?>" and "?" will be printed
# respectively. This program is best-effort, and should be improved in the
# future when other kernel capabilities are made available. If you need a
# more reliable tool now, then consider other tracing alternatives (eg,
# SystemTap). This tool is really a proof of concept to see what ftrace can
# currently do.
#
# From perf-tools: https://github.com/brendangregg/perf-tools
#
# See the execsnoop(8) man page (in perf-tools) for more info.
#
# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
# (http://www.gnu.org/copyleft/gpl.html)
#
# 07-Jul-2014 Brendan Gregg Created this.
use strict;
use warnings;
use POSIX qw(strftime);
use Getopt::Long;
my $tracing = "/sys/kernel/debug/tracing";
my $flock = "/var/tmp/.ftrace-lock";
my $tpdir = "sched/sched_process_exec";
my $tptext = $tpdir; $tptext =~ s/\//:/;
local $SIG{INT} = \&cleanup;
local $SIG{QUIT} = \&cleanup;
local $SIG{TERM} = \&cleanup;
local $SIG{PIPE} = \&cleanup;
local $SIG{HUP} = \&cleanup;
$| = 1;
### options
my ($name, $help);
GetOptions("name=s" => \$name,
"help" => \$help)
or usage();
usage() if $help;
sub usage {
print STDERR "USAGE: execsnoop [-h] [-n name]\n";
print STDERR " eg,\n";
print STDERR " execsnoop -n ls # show \"ls\" cmds only.\n";
exit;
}
sub ldie {
unlink $flock;
die @_;
}
sub writeto {
my ($string, $file) = @_;
open FILE, ">$file" or return 0;
print FILE $string or return 0;
close FILE or return 0;
}
### check permissions
chdir "$tracing" or ldie "ERROR: accessing tracing. Root? Kernel has FTRACE?" .
"\ndebugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)";
### ftrace lock
if (-e $flock) {
open FLOCK, $flock; my $fpid = <FLOCK>; chomp $fpid; close FLOCK;
die "ERROR: ftrace may be in use by PID $fpid ($flock)";
}
writeto "$$", $flock or die "ERROR: unable to write $flock.";
### setup and begin tracing
writeto "nop", "current_tracer" or ldie "ERROR: disabling current_tracer.";
writeto "1", "events/$tpdir/enable" or ldie "ERROR: enabling tracepoint " .
"\"$tptext\" (tracepoint missing in this kernel version?)";
open TPIPE, "trace_pipe" or warn "ERROR: opening trace_pipe.";
printf "%-8s %6s %6s %s\n", "TIME", "PID", "PPID", "ARGS";
while (<TPIPE>) {
my ($taskpid, $rest) = split;
my ($task, $pid) = $taskpid =~ /(.*)-(\d+)/;
next if (defined $name and $name ne $task);
my $args = "$task <?>";
if (open CMDLINE, "/proc/$pid/cmdline") {
my $arglist = <CMDLINE>;
if (defined $arglist) {
$arglist =~ s/\000/ /g;
$args = $arglist;
}
close CMDLINE;
}
my $ppid = "?";
if (open STAT, "/proc/$pid/stat") {
my $fields = <STAT>;
if (defined $fields) {
$ppid = (split ' ', $fields)[3];
}
close STAT;
}
my $now = strftime "%H:%M:%S", localtime;
printf "%-8s %6s %6s %s\n", $now, $pid, $ppid, $args;
}
### end tracing
cleanup();
sub cleanup {
print "\nEnding tracing...\n";
close TPIPE;
writeto "0", "events/$tpdir/enable" or
ldie "ERROR: disabling \"$tptext\"";
writeto "", "trace";
unlink $flock;
exit;
}
================================================
FILE: deprecated/execsnoop-proc.8
================================================
.TH execsnoop\-proc 8 "2014-07-07" "USER COMMANDS"
.SH NAME
execsnoop\-proc \- trace process exec() with arguments. Uses Linux ftrace. /proc version.
.SH SYNOPSIS
.B execsnoop\-proc
[\-h] [\-n name]
.SH DESCRIPTION
execsnoop\-proc traces process execution, showing PID, PPID, and argument details
if possible.
This traces exec() from the fork()->exec() sequence, which means it won't
catch new processes that only fork(), and, it will catch processes that
re-exec. This instruments sched:sched_process_exec without buffering, and then
in user-space (this program) reads PPID and process arguments asynchronously
from /proc.
If the process traced is very short-lived, this program may miss reading
arguments and PPID details. In that case, "<?>" and "?" will be printed
respectively.
This program is best-effort (a hack), and should be improved in the future when
other kernel capabilities are made available. It may be useful in the meantime.
If you need a more reliable tool now, consider other tracing alternates (eg,
SystemTap). This tool is really a proof of concept to see what ftrace can
currently do.
See execsnoop(8) for another version that reads arguments from registers
instead of /proc.
Since this uses ftrace, only the root user can use this tool.
.SH REQUIREMENTS
FTRACE CONFIG and the sched:sched_process_exec tracepoint, which you may already
have enabled and available on recent kernels, and Perl.
.SH OPTIONS
\-n name
Only show processes that match this name. This is filtered in user space.
.TP
\-h
Print usage message.
.SH EXAMPLES
.TP
Trace all new processes and arguments (if possible):
.B execsnoop\-proc
.TP
Trace all new processes with process name "sed":
.B execsnoop\-proc -n sed
.SH FIELDS
.TP
TIME
Time of process exec(): HH:MM:SS.
.TP
PID
Process ID.
.TP
PPID
Parent process ID, if this was able to be read (may be missed for short-lived
processes). If it is unable to be read, "?" is printed.
.TP
ARGS
Command line arguments, if these were able to be read in time (may be missed
for short-lived processes). If they are unable to be read, "<?>" is printed.
.SH OVERHEAD
This reads and processes exec() events in user space as they occur. Since the
rate of exec() is expected to be low (< 500/s), the overhead is expected to
be small or negligible.
.SH SOURCE
This is from the perf-tools collection.
.IP
https://github.com/brendangregg/perf-tools
.PP
Also look under the examples directory for a text file containing example
usage, output, and commentary for this tool.
.SH OS
Linux
.SH STABILITY
Unstable - in development.
.SH AUTHOR
Brendan Gregg
.SH SEE ALSO
execsnoop(8), top(1)
================================================
FILE: deprecated/execsnoop-proc_example.txt
================================================
Demonstrations of execsnoop-proc, the Linux ftrace version.
Here's execsnoop showing what's really executed by "man ls":
# ./execsnoop
TIME PID PPID ARGS
17:52:37 22406 25781 man ls
17:52:37 22413 22406 preconv -e UTF-8
17:52:37 22416 22406 pager -s
17:52:37 22415 22406 /bin/sh /usr/bin/nroff -mandoc -rLL=162n -rLT=162n -Tutf8
17:52:37 22414 22406 tbl
17:52:37 22419 22418 locale charmap
17:52:37 22420 22415 groff -mtty-char -Tutf8 -mandoc -rLL=162n -rLT=162n
17:52:37 22421 22420 troff -mtty-char -mandoc -rLL=162n -rLT=162n -Tutf8
17:52:37 22422 22420 grotty
These are short-lived processes, where the argument and PPID details are often
missed by execsnoop:
# ./execsnoop
TIME PID PPID ARGS
18:00:33 26750 1961 multilog <?>
18:00:33 26749 1972 multilog <?>
18:00:33 26749 1972 multilog <?>
18:00:33 26751 ? mkdir <?>
18:00:33 26749 1972 multilog <?>
18:00:33 26752 ? chown <?>
18:00:33 26750 1961 multilog <?>
18:00:33 26750 1961 multilog <?>
18:00:34 26753 1961 multilog <?>
18:00:34 26754 1972 multilog <?>
[...]
This will be fixed in a later version, but likely requires some kernel or
tracer changes first (fetching cmdline as the probe fires).
The previous examples were on Linux 3.14 and 3.16 kernels. Here's a 3.2 system
I'm running:
# ./execsnoop
ERROR: enabling tracepoint "sched:sched_process_exec" (tracepoint missing in this kernel version?) at ./execsnoop line 78.
This kernel version is missing the sched_process_exec probe, which is pretty
annoying.
================================================
FILE: disk/bitesize
================================================
#!/bin/bash
#
# bitesize - show disk I/O size as a histogram.
# Written using Linux perf_events (aka "perf").
#
# This can be used to characterize the distribution of block device I/O
# sizes. To study I/O in more detail, see iosnoop(8).
#
# USAGE: bitesize [-h] [-b buckets] [seconds]
# eg,
# ./bitesize 10
#
# Run "bitesize -h" for full usage.
#
# REQUIREMENTS: perf_events and block:block_rq_issue tracepoint, which you may
# already have on recent kernels.
#
# This uses multiple counting tracepoints with different filters, one for each
# histogram bucket. While this is summarized in-kernel, the use of multiple
# tracepoints does add addiitonal overhead, which is more evident if you add
# more buckets. In the future this functionality will be available in an
# efficient way in the kernel, and this tool can be rewritten.
#
# From perf-tools: https://github.com/brendangregg/perf-tools
#
# COPYRIGHT: Copyright (c) 2014 Brendan Gregg.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
# (http://www.gnu.org/copyleft/gpl.html)
#
# 22-Jul-2014 Brendan Gregg Created this.
duration=0
buckets=(1 8 64 128)
secsz=512
trap ':' INT QUIT TERM PIPE HUP
function usage {
cat <<-END >&2
USAGE: bitesize [-h] [-b buckets] [seconds]
-b buckets # specify histogram buckets (Kbytes)
-h # this usage message
eg,
bitesize # trace I/O size until Ctrl-C
bitesize 10 # trace I/O size for 10 seconds
bitesize -b "8 16 32" # specify custom bucket points
END
exit
}
function die {
echo >&2 "$@"
exit 1
}
### process options
while getopts b:h opt
do
case $opt in
b) buckets=($OPTARG) ;;
h|?) usage ;;
esac
done
shift $(( $OPTIND - 1 ))
tpoint=block:block_rq_issue
var=nr_sector
duration=$1
### convert buckets (Kbytes) to disk sectors
i=0
sectors=(${buckets[*]})
((max_i = ${#buckets[*]} - 1))
while (( i <= max_i )); do
(( sectors[$i] = ${sectors[$i]} * 1024 / $secsz ))
# avoid negative array index errors for old version bash
if (( i > 0 ));then
if (( ${sectors[$i]} <= ${sectors[$i - 1]} )); then
die "ERROR: bucket list must increase in size."
fi
fi
(( i++ ))
done
### build list of tracepoints and filters for each histogram bucket
max_b=${buckets[$max_i]}
max_s=${sectors[$max_i]}
tpoints="-e $tpoint --filter \"$var < ${sectors[0]}\""
awkarray=
i=0
while (( i < max_i )); do
tpoints="$tpoints -e $tpoint --filter \"$var >= ${sectors[$i]} && "
tpoints="$tpoints $var < ${sectors[$i + 1]}\""
awkarray="$awkarray buckets[$i]=${buckets[$i]};"
(( i++ ))
done
awkarray="$awkarray buckets[$max_i]=${buckets[$max_i]};"
tpoints="$tpoints -e $tpoint --filter \"$var >= ${sectors[$max_i]}\""
### prepare to run
if (( duration )); then
etext="for $duration seconds"
cmd="sleep $duration"
else
etext="until Ctrl-C"
cmd="sleep 999999"
fi
echo "Tracing block I/O size (bytes), $etext..."
### run perf
out="-o /dev/stdout" # a workaround needed in linux 3.2; not by 3.4.15
stat=$(eval perf stat $tpoints -a $out $cmd 2>&1)
if (( $? != 0 )); then
echo >&2 "ERROR running perf:"
echo >&2 "$stat"
exit
fi
### find max value for ASCII histogram
most=$(echo "$stat" | awk -v tpoint=$tpoint '
$2 == tpoint { gsub(/,/, ""); if ($1 > m) { m = $1 } }
END { print m }'
)
### process output
echo
echo "$stat" | awk -v tpoint=$tpoint -v max_i=$max_i -v most=$most '
function star(sval, smax, swidth) {
stars = ""
# using int could avoid error on gawk
if (int(smax) == 0) return ""
for (si = 0; si < (swidth * sval / smax); si++) {
stars = stars "#"
}
return stars
}
BEGIN {
'"$awkarray"'
printf(" %-15s: %-8s %s\n", "Kbytes", "I/O",
"Distribution")
}
/Performance counter stats/ { i = -1 }
# reverse order of rule set is important
{ ok = 0 }
$2 == tpoint { num = $1; gsub(/,/, "", num); ok = 1 }
ok && i >= max_i {
printf(" %10.1f -> %-10s: %-8s |%-38s|\n",
buckets[i], "", num, star(num, most, 38))
next
}
ok && i >= 0 && i < max_i {
printf(" %10.1f -> %-10.1f: %-8s |%-38s|\n",
buckets[i], buckets[i+1] - 0.1, num,
star(num, most, 38))
i++
next
}
ok && i == -1 {
printf(" %10s -> %-10.1f: %-8s |%-38s|\n", "",
buckets[0] - 0.1, num, star(num, most, 38))
i++
}
'
================================================
FILE: examples/bitesize_example.txt
================================================
Demonstrations of bitesize, the Linux perf_events version.
bitesize traces block I/O issued, and reports a histogram of I/O size. By
default five buckets are used to gather statistics on common I/O sizes:
# ./bitesize
Tracing block I/O size (bytes), until Ctrl-C...
^C
Kbytes : I/O Distribution
-> 0.9 : 0 | |
1.0 -> 7.9 : 38 |# |
8.0 -> 63.9 : 10108 |######################################|
64.0 -> 127.9 : 13 |# |
128.0 -> : 1 |# |
In this case, most of the I/O was between 8 and 63.9 Kbytes. The "63.9"
really means "less than 64".
Specifying custom buckets to examine the I/O size in more detail:
# ./bitesize -b "8 16 24 32"
Tracing block I/O size (bytes), until Ctrl-C...
^C
Kbytes : I/O Distribution
-> 7.9 : 89 |# |
8.0 -> 15.9 : 14665 |######################################|
16.0 -> 23.9 : 657 |## |
24.0 -> 31.9 : 661 |## |
32.0 -> : 376 |# |
The I/O is mostly between 8 and 15.9 Kbytes
It's probably 8 Kbytes. Checking:
# ./bitesize -b "8 9"
Tracing block I/O size (bytes), until Ctrl-C...
^C
Kbytes : I/O Distribution
-> 7.9 : 62 |# |
8.0 -> 8.9 : 11719 |######################################|
9.0 -> : 1358 |##### |
It is.
The overhead of this tool is relative to the number of buckets used, hence only
using what is necessary.
To study this I/O in more detail, I can use iosnoop(8) and capture it to a file
for post-processing.
Use -h to print the USAGE message:
# ./bitesize -h
USAGE: bitesize [-h] [-b buckets] [seconds]
-b buckets # specify histogram buckets (Kbytes)
-h # this usage message
eg,
bitesize # trace I/O size until Ctrl-C
bitesize 10 # trace I/O size for 10 seconds
bitesize -b "8 16 32" # specify custom bucket points
================================================
FILE: examples/cachestat_example.txt
================================================
Demonstrations of cachestat, the Linux ftrace version.
Here is some sample output showing file system cache statistics, followed by
the workload that caused it:
# ./cachestat -t
Counting cache functions... Output every 1 seconds.
TIME HITS MISSES DIRTIES RATIO BUFFERS_MB CACHE_MB
08:28:57 415 0 0 100.0% 1 191
08:28:58 411 0 0 100.0% 1 191
08:28:59 362 97 0 78.9% 0 8
08:29:00 411 0 0 100.0% 0 9
08:29:01 775 20489 0 3.6% 0 89
08:29:02 411 0 0 100.0% 0 89
08:29:03 6069 0 0 100.0% 0 89
08:29:04 15249 0 0 100.0% 0 89
08:29:05 411 0 0 100.0% 0 89
08:29:06 411 0 0 100.0% 0 89
08:29:07 411 0 3 100.0% 0 89
[...]
I used the -t option to include the TIME column, to make describing the output
easier.
The workload was:
# echo 1 > /proc/sys/vm/drop_caches; sleep 2; cksum 80m; sleep 2; cksum 80m
At 8:28:58, the page cache was dropped by the first command, which can be seen
by the drop in size for "CACHE_MB" (page cache size) from 191 Mbytes to 8.
After a 2 second sleep, a cksum command was issued at 8:29:01, for an 80 Mbyte
file (called "80m"), which caused a total of ~20,400 misses ("MISSES" column),
and the page cache size to grow by 80 Mbytes. The hit ratio during this dropped
to 3.6%. Finally, after another 2 second sleep, at 8:29:03 the cksum command
was run a second time, this time hitting entirely from cache.
Instrumenting all file system cache accesses does cost some overhead, and this
tool might slow your target system by 2% or so. Test before use if this is a
concern.
This tool also uses dynamic tracing, and is tied to Linux kernel implementation
details. If it doesn't work for you, it probably needs fixing.
Use -h to print the USAGE message:
# ./cachestat -h
USAGE: cachestat [-Dht] [interval]
-D # print debug counters
-h # this usage message
-t # include timestamp
interval # output interval in secs (default 1)
eg,
cachestat # show stats every second
cachestat 5 # show stats every 5 seconds
See the man page and example file for more info.
================================================
FILE: examples/execsnoop_example.txt
================================================
Demonstrations of execsnoop, the Linux ftrace version.
Here's execsnoop showing what's really executed by "man ls":
# ./execsnoop
Tracing exec()s. Ctrl-C to end.
PID PPID ARGS
22898 22004 man ls
22905 22898 preconv -e UTF-8
22908 22898 pager -s
22907 22898 nroff -mandoc -rLL=164n -rLT=164n -Tutf8
22906 22898 tbl
22911 22910 locale charmap
22912 22907 groff -mtty-char -Tutf8 -mandoc -rLL=164n -rLT=164n
22913 22912 troff -mtty-char -mandoc -rLL=164n -rLT=164n -Tutf8
22914 22912 grotty
Many commands. This is particularly useful for understanding application
startup.
Another use for execsnoop is identifying short-lived processes. Eg, with the -t
option to see timestamps:
# ./execsnoop -t
Tracing exec()s. Ctrl-C to end.
TIMEs PID PPID ARGS
7419756.154031 8185 8181 mawk -W interactive -v o=1 -v opt_name=0 -v name= [...]
7419756.154131 8186 8184 cat -v trace_pipe
7419756.245264 8188 1698 ./run
7419756.245691 8189 1696 ./run
7419756.246212 8187 1689 ./run
7419756.278993 8190 1693 ./run
7419756.278996 8191 1692 ./run
7419756.288430 8192 1695 ./run
7419756.290115 8193 1691 ./run
7419756.292406 8194 1699 ./run
7419756.293986 8195 1690 ./run
7419756.294149 8196 1686 ./run
7419756.296527 8197 1687 ./run
7419756.296973 8198 1697 ./run
7419756.298356 8200 1685 ./run
7419756.298683 8199 1688 ./run
7419757.269883 8201 1696 ./run
[...]
So we're running many "run" commands every second. The PPID is included, so I
can debug this further (they are "supervise" processes).
Short-lived processes can consume CPU and not be visible from top(1), and can
be the source of hidden performance issues.
Here's another example: I noticed CPU usage was high in top(1), but couldn't
see the responsible process:
$ top
top - 00:04:32 up 78 days, 15:41, 3 users, load average: 0.85, 0.29, 0.14
Tasks: 123 total, 1 running, 121 sleeping, 0 stopped, 1 zombie
Cpu(s): 15.7%us, 34.9%sy, 0.0%ni, 49.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.2%st
Mem: 7629464k total, 7537216k used, 92248k free, 1376492k buffers
Swap: 0k total, 0k used, 0k free, 5432356k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7225 bgregg-t 20 0 29480 6196 2128 S 3 0.1 0:02.64 ec2rotatelogs
1 root 20 0 24320 2256 1340 S 0 0.0 0:01.23 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 1:19.61 ksoftirqd/0
4 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0
5 root 20 0 0 0 0 S 0 0.0 0:00.01 kworker/u:0
6 root RT 0 0 0 0 S 0 0.0 0:16.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:17.29 watchdog/0
8 root RT 0 0 0 0 S 0 0.0 0:15.85 migration/1
9 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:0
[...]
See the line starting with "Cpu(s):". So there's about 50% CPU utilized (this
is a two CPU server, so that's equivalent to one full CPU), but this CPU usage
isn't visible from the process listing.
vmstat agreed, showing the same average CPU usage statistics:
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 0 92816 1376476 5432188 0 0 0 3 2 1 0 1 99 0
1 0 0 92676 1376484 5432264 0 0 0 24 6573 6130 12 38 49 0
1 0 0 91964 1376484 5432272 0 0 0 0 6529 6097 16 35 49 0
1 0 0 92692 1376484 5432272 0 0 0 0 6192 5775 17 35 49 0
1 0 0 92692 1376484 5432272 0 0 0 0 6554 6121 14 36 50 0
1 0 0 91940 1376484 5432272 0 0 0 12 6546 6101 13 38 49 0
1 0 0 92560 1376484 5432272 0 0 0 0 6201 5769 15 35 49 0
1 0 0 92676 1376484 5432272 0 0 0 0 6524 6123 17 34 49 0
1 0 0 91932 1376484 5432272 0 0 0 0 6546 6107 10 40 49 0
1 0 0 92832 1376484 5432272 0 0 0 0 6057 5710 13 38 49 0
1 0 0 92248 1376484 5432272 0 0 84 28 6592 6183 16 36 48 1
1 0 0 91504 1376492 5432348 0 0 0 12 6540 6098 18 33 49 1
[...]
So this could be caused by short-lived processes, who vanish before they are
seen by top(1). Do I have my execsnoop handy? Yes:
# ~/perf-tools/bin/execsnoop
Tracing exec()s. Ctrl-C to end.
PID PPID ARGS
10239 10229 gawk -v o=0 -v opt_name=0 -v name= -v opt_duration=0 [...]
10240 10238 cat -v trace_pipe
10242 7225 sh [?]
10243 10242 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.201201.3122.txt
10245 7225 sh [?]
10246 10245 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.202201.3122.txt
10248 7225 sh [?]
10249 10248 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.203201.3122.txt
10251 7225 sh [?]
10252 10251 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.204201.3122.txt
10254 7225 sh [?]
10255 10254 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.205201.3122.txt
10257 7225 sh [?]
10258 10257 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.210201.3122.txt
10260 7225 sh [?]
10261 10260 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.211201.3122.txt
10263 7225 sh [?]
10264 10263 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.212201.3122.txt
10266 7225 sh [?]
10267 10266 /usr/sbin/lsof -X /logs/tomcat/cores/threaddump.20141215.213201.3122.txt
[...]
The output scrolled quickly, showing that many shell and lsof processes were
being launched. If you check the PID and PPID columns carefully, you can see that
these are ultimately all from PID 7225. We saw that earlier in the top output:
ec2rotatelogs, at 3% CPU. I now know the culprit.
I should have used "-t" to show the timestamps with this example.
Run -h to print the USAGE message:
# ./execsnoop -h
USAGE: execsnoop [-hrt] [-a argc] [-d secs] [name]
-d seconds # trace duration, and use buffers
-a argc # max args to show (default 8)
-r # include re-execs
-t # include time (seconds)
-h # this usage message
name # process name to match (REs allowed)
eg,
execsnoop # watch exec()s live (unbuffered)
execsnoop -d 1 # trace 1 sec (buffered)
execsnoop grep # trace process names containing grep
execsnoop 'log$' # filenames ending in "log"
See the man page and example file for more info.
================================================
FILE: examples/funccount_example.txt
================================================
Demonstrations of funccount, the Linux ftrace version.
Tracing all kernel functions that start with "bio_" (which would be block
interface functions), and counting how many times they were executed until
Ctrl-C is hit:
# ./funccount 'bio_*'
Tracing "bio_*"... Ctrl-C to end.
^C
FUNC COUNT
bio_attempt_back_merge 26
bio_get_nr_vecs 361
bio_alloc 536
bio_alloc_bioset 536
bio_endio 536
bio_free 536
bio_fs_destructor 536
bio_init 536
bio_integrity_enabled 536
bio_put 729
bio_add_page 1004
Note that these counts are performed in-kernel context, using the ftrace
function profiler, which means this is a (relatively) low overhead technique.
Test yourself to quantify overhead.
As was demonstrated here, wildcards can be used. Individual functions can also
be specified. For example, all of the following are valid arguments:
bio_init
bio_*
*init
*bio*
A "*" within a string (eg, "bio*init") is not supported.
The full list of what can be traced is in:
/sys/kernel/debug/tracing/available_filter_functions, which can be grep'd to
check what is there. Note that grep uses regular expressions, whereas
funccount uses globbing for wildcards.
Counting all "tcp_" kernel functions, and printing a summary every one second:
# ./funccount -i 1 -t 5 'tcp_*'
Tracing "tcp_*". Top 5 only... Ctrl-C to end.
FUNC COUNT
tcp_cleanup_rbuf 386
tcp_service_net_dma 386
tcp_established_options 549
tcp_v4_md5_lookup 560
tcp_v4_md5_do_lookup 890
FUNC COUNT
tcp_service_net_dma 498
tcp_cleanup_rbuf 499
tcp_established_options 664
tcp_v4_md5_lookup 672
tcp_v4_md5_do_lookup 1071
[...]
Neat.
Tracing all "ext4*" kernel functions for 10 seconds, and printing the top 25:
# ./funccount -t 25 -d 10 'ext4*'
Tracing "ext4*" for 10 seconds. Top 25 only...
FUNC COUNT
ext4_inode_bitmap 840
ext4_meta_trans_blocks 840
ext4_ext_drop_refs 843
ext4_find_entry 845
ext4_discard_preallocations 1008
ext4_free_inodes_count 1120
ext4_group_desc_csum 1120
ext4_group_desc_csum_set 1120
ext4_getblk 1128
ext4_es_free_extent 1328
ext4_map_blocks 1471
ext4_es_lookup_extent 1751
ext4_mb_check_limits 1873
ext4_es_lru_add 2031
ext4_data_block_valid 2312
ext4_journal_check_start 3080
ext4_mark_inode_dirty 5320
ext4_get_inode_flags 5955
ext4_get_inode_loc 5955
ext4_mark_iloc_dirty 5955
ext4_reserve_inode_write 5955
ext4_inode_table 7076
ext4_get_group_desc 8476
ext4_has_inline_data 9492
ext4_inode_touch_time_cmp 38980
Ending tracing...
So ext4_inode_touch_time_cmp() was called the most frequently, at 38,980 times.
This may be normal, this may not. The purpose of this tool is to give you one
view of how one or many kernel functions are executed. Previously I had little
idea what ext4 was doing internally. Now I know the top 25 functions, and their
rate, and can begin researching them from the source code.
Use -h to print the USAGE message:
# ./funccount -h
USAGE: funccount [-hT] [-i secs] [-d secs] [-t top] funcstring
-d seconds # total duration of trace
-h # this usage message
-i seconds # interval summary
-t top # show top num entries only
-T # include timestamp (for -i)
eg,
funccount 'vfs*' # trace all funcs that match "vfs*"
funccount -d 5 'tcp*' # trace "tcp*" funcs for 5 seconds
funccount -t 10 'ext3*' # show top 10 "ext3*" funcs
funccount -i 1 'ext3*' # summary every 1 second
funccount -i 1 -d 5 'ext3*' # 5 x 1 second summaries
See the man page and example file for more info.
================================================
FILE: examples/funcgraph_example.txt
================================================
Demonstrations of funcgraph, the Linux ftrace version.
I'll start by showing do_nanosleep(), since it's usually a low frequency
function that can be easily triggered (run "vmstat 1"):
# ./funcgraph do_nanosleep
Tracing "do_nanosleep"... Ctrl-C to end.
0) | do_nanosleep() {
0) | hrtimer_start_range_ns() {
0) | __hrtimer_start_range_ns() {
0) | lock_hrtimer_base.isra.24() {
0) 0.198 us | _raw_spin_lock_irqsave();
0) 0.908 us | }
0) 0.061 us | idle_cpu();
0) 0.117 us | ktime_get();
0) 0.371 us | enqueue_hrtimer();
0) 0.075 us | _raw_spin_unlock_irqrestore();
0) 3.447 us | }
0) 3.998 us | }
0) | schedule() {
0) | __schedule() {
0) 0.050 us | rcu_note_context_switch();
0) 0.055 us | _raw_spin_lock_irq();
0) | deactivate_task() {
0) | dequeue_task() {
0) 0.142 us | update_rq_clock();
0) | dequeue_task_fair() {
0) | dequeue_entity() {
0) | update_curr() {
0) 0.086 us | cpuacct_charge();
0) 0.757 us | }
0) 0.052 us | clear_buddies();
0) 0.103 us | update_cfs_load();
0) | update_cfs_shares() {
0) | reweight_entity() {
0) 0.077 us | update_curr();
0) 0.438 us | }
0) 0.794 us | }
0) 3.067 us | }
0) 0.064 us | set_next_buddy();
0) 0.066 us | update_cfs_load();
0) 0.085 us | update_cfs_shares();
0) | hrtick_update() {
0) 0.063 us | hrtick_start_fair();
0) 0.367 us | }
0) 5.188 us | }
0) 5.923 us | }
0) 6.228 us | }
0) | put_prev_task_fair() {
0) 0.078 us | put_prev_entity();
0) | put_prev_entity() {
0) 0.070 us | update_curr();
0) 0.074 us | __enqueue_entity();
0) 0.737 us | }
0) 1.367 us | }
0) | pick_next_task_fair() {
0) | pick_next_entity() {
0) 0.052 us | wakeup_preempt_entity.isra.95();
0) 0.070 us | clear_buddies();
0) 0.676 us | }
0) | set_next_entity() {
0) 0.052 us | update_stats_wait_end();
0) 0.435 us | }
0) | pick_next_entity() {
0) 0.065 us | clear_buddies();
0) 0.376 us | }
0) | set_next_entity() {
0) 0.067 us | update_stats_wait_end();
0) 0.374 us | }
0) 0.051 us | hrtick_start_fair();
0) 3.879 us | }
0) 0.057 us | paravirt_start_context_switch();
0) | xen_load_sp0() {
0) 0.050 us | paravirt_get_lazy_mode();
0) 0.057 us | __xen_mc_entry();
0) 0.056 us | paravirt_get_lazy_mode();
0) 1.441 us | }
0) | xen_load_tls() {
0) 0.049 us | paravirt_get_lazy_mode();
0) 0.051 us | paravirt_get_lazy_mode();
0) | load_TLS_descriptor() {
0) | arbitrary_virt_to_machine() {
0) 0.081 us | __virt_addr_valid();
0) 0.052 us | __phys_addr();
0) 0.084 us | get_phys_to_machine();
0) 1.115 us | }
0) 0.053 us | __xen_mc_entry();
0) 1.744 us | }
0) | load_TLS_descriptor() {
0) | arbitrary_virt_to_machine() {
0) 0.053 us | __virt_addr_valid();
0) 0.056 us | __phys_addr();
0) 0.057 us | get_phys_to_machine();
0) 0.990 us | }
0) 0.053 us | __xen_mc_entry();
0) 1.583 us | } /* load_TLS_descriptor */
0) | load_TLS_descriptor() {
0) | arbitrary_virt_to_machine() {
0) 0.057 us | __virt_addr_valid();
0) 0.051 us | __phys_addr();
0) 0.053 us | get_phys_to_machine();
0) 0.978 us | }
0) 0.052 us | __xen_mc_entry();
0) 1.586 us | }
0) 0.052 us | paravirt_get_lazy_mode();
0) 6.630 us | }
0) | xen_end_context_switch() {
0) 0.666 us | xen_mc_flush();
0) 0.050 us | paravirt_end_context_switch();
0) 1.286 us | }
0) 0.172 us | xen_write_msr_safe();
------------------------------------------
0) platfor-3210 => vmstat-2854
------------------------------------------
0) | do_nanosleep() {
0) | hrtimer_start_range_ns() {
0) | __hrtimer_start_range_ns() {
0) | lock_hrtimer_base.isra.24() {
0) 0.217 us | _raw_spin_lock_irqsave();
0) 0.831 us | }
0) 0.066 us | idle_cpu();
0) 0.123 us | ktime_get();
0) 1.172 us | enqueue_hrtimer();
0) 0.089 us | _raw_spin_unlock_irqrestore();
0) 4.050 us | }
0) 4.523 us | }
[...]
The default output shows the function call graph, including all child kernel
functions, along with the function duration times. These times are printed on
either the return line for the function ("}"), or for leaf functions, on the
same line.
The format of this output is documented in the function graph section of the
kernel source file Documentation/trace/ftrace.txt.
This particular example shows the workings of do_nanosleep, in the first dozen
lines, and then schedule() is called to sleep this thread and run another. The
inner workings of schedule() is included in the output.
This output is great for determining the behavior of a certain kernel function,
and to identify functions that can be studied in more details using other, lower
overhead, tools (eg, funccount(8), functrace(8), kprobe(8)). The overheads
of funcgraph are moderate, since all kernel functions are traced in case
they are executed, then included in the output if they are.
Now, if you want to start understanding the general behavior of the kernel,
without a certain kernel function in mind, you may be better to begin with
CPU stack profiling using perf and generating a flame graph. Such an approach
has low overhead, as you are in control of the frequency of event collection
(eg, gathering CPU stacks at 99 Hertz). For instructions, see:
http://www.brendangregg.com/perf.html#FlameGraphs
On newer Linux kernels, you can use the -m option to limit the function
depth. Eg, 3 levels only:
# ./funcgraph -m 3 do_nanosleep
Tracing "do_nanosleep"... Ctrl-C to end.
1) | do_nanosleep() {
1) | hrtimer_start_range_ns() {
1) 1.115 us | __hrtimer_start_range_ns();
1) 1.919 us | }
1) | schedule() {
1) | __schedule() {
1) 1000131 us | }
1) 11.006 us | xen_evtchn_do_upcall();
1) 1000149 us | }
1) | hrtimer_cancel() {
1) 0.212 us | hrtimer_try_to_cancel();
1) 0.699 us | }
1) 1000154 us | }
Neat.
Now do_sys_open() to 3 levels:
0) | do_sys_open() {
0) | getname() {
0) 0.296 us | getname_flags();
0) 0.768 us | }
0) | get_unused_fd_flags() {
0) 0.397 us | __alloc_fd();
0) 0.827 us | }
0) | do_filp_open() {
0) 4.166 us | path_openat();
0) 4.617 us | }
0) | __fsnotify_parent() {
0) 0.083 us | dget_parent();
0) 0.063 us | dput();
0) 0.883 us | }
0) 0.058 us | fsnotify();
0) | fd_install() {
0) 0.133 us | __fd_install();
0) 0.525 us | }
0) | putname() {
0) 0.198 us | final_putname();
0) 0.512 us | }
0) 10.777 us | }
[...]
I can then pick the highest latency child function, then run funcgraph again
using it as the target.
Without timestamps (-D to elide duration):
# ./funcgraph -Dm 3 do_sys_open
Tracing "do_sys_open"... Ctrl-C to end.
1) do_sys_open() {
1) getname() {
1) getname_flags();
1) }
1) get_unused_fd_flags() {
1) __alloc_fd();
1) }
1) do_filp_open() {
1) path_openat();
1) }
1) __fsnotify_parent();
1) fsnotify();
1) fd_install() {
1) __fd_install();
1) }
1) putname() {
1) final_putname();
1) }
1) }
Beautiful.
I could elide the CPU column as well, but I want to leave it: if it changes
half-way through some output, you know the CPU buffer has switched, and the
output may be shuffled.
For this example, I trace vfs_read() calls by process ID 5363: which is a bash
shell. I also include headers (-H) and absolute timestamps (-t). While
tracing, in that bash shell, I typed the word "hello":
# ./funcgraph -Htp 5363 vfs_read
Tracing "vfs_read" for PID 5363... Ctrl-C to end.
# tracer: function_graph
#
# TIME CPU DURATION FUNCTION CALLS
# | | | | | | | |
7238523.638008 | 0) | finish_task_switch() {
7238523.638012 | 0) | xen_evtchn_do_upcall() {
7238523.638012 | 0) | irq_enter() {
7238523.638013 | 0) 0.153 us | rcu_irq_enter();
7238523.638014 | 0) 1.144 us | }
7238523.638014 | 0) 0.056 us | exit_idle();
7238523.638014 | 0) | __xen_evtchn_do_upcall() {
7238523.638015 | 0) | evtchn_2l_handle_events() {
7238523.638015 | 0) 0.057 us | irq_from_virq();
7238523.638015 | 0) | evtchn_from_irq() {
7238523.638015 | 0) | irq_get_irq_data() {
7238523.638016 | 0) 0.058 us | irq_to_desc();
7238523.638016 | 0) 0.565 us | }
7238523.638016 | 0) 0.966 us | }
7238523.638016 | 0) | get_evtchn_to_irq() {
7238523.638017 | 0) 0.050 us | evtchn_2l_max_channels();
7238523.638017 | 0) 0.386 us | }
7238523.638017 | 0) | generic_handle_irq() {
7238523.638017 | 0) 0.058 us | irq_to_desc();
7238523.638018 | 0) | handle_percpu_irq() {
7238523.638018 | 0) | ack_dynirq() {
7238523.638018 | 0) | evtchn_from_irq() {
7238523.638018 | 0) | irq_get_irq_data() {
7238523.638019 | 0) 0.049 us | irq_to_desc();
7238523.638019 | 0) 0.441 us | }
7238523.638019 | 0) 0.772 us | }
7238523.638019 | 0) 0.049 us | irq_move_irq();
7238523.638020 | 0) 0.060 us | evtchn_2l_clear_pending();
7238523.638020 | 0) 1.810 us | }
7238523.638020 | 0) | handle_irq_event_percpu() {
7238523.638020 | 0) | xen_irq_work_interrupt() {
7238523.638021 | 0) | irq_enter() {
7238523.638021 | 0) 0.056 us | rcu_irq_enter();
7238523.638021 | 0) 0.384 us | }
7238523.638021 | 0) | __wake_up() {
7238523.638022 | 0) 0.059 us | _raw_spin_lock_irqsave();
7238523.638022 | 0) | __wake_up_common() {
7238523.638022 | 0) | autoremove_wake_function() {
7238523.638023 | 0) | default_wake_function() {
7238523.638023 | 0) | try_to_wake_up() {
7238523.638023 | 0) 0.220 us | _raw_spin_lock_irqsave();
7238523.638024 | 0) 0.270 us | task_waking_fair();
7238523.638024 | 0) | select_task_rq_fair() {
7238523.638025 | 0) 0.055 us | source_load();
7238523.638025 | 0) 0.056 us | target_load();
7238523.638025 | 0) 0.060 us | idle_cpu();
7238523.638026 | 0) 0.054 us | cpus_share_cache();
7238523.638026 | 0) 0.083 us | idle_cpu();
7238523.638026 | 0) 2.060 us | }
7238523.638027 | 0) 0.051 us | _raw_spin_lock();
7238523.638027 | 0) | ttwu_do_activate.constprop.124() {
7238523.638027 | 0) | activate_task() {
7238523.638027 | 0) | enqueue_task() {
7238523.638028 | 0) 0.120 us | update_rq_clock();
7238523.638028 | 0) | enqueue_task_fair() {
7238523.638028 | 0) | enqueue_entity() {
7238523.638028 | 0) 0.147 us | update_curr();
7238523.638029 | 0) 0.055 us | __compute_runnable_contrib.part.51();
7238523.638029 | 0) 0.066 us | __update_entity_load_avg_contrib();
7238523.638029 | 0) 0.141 us | update_cfs_rq_blocked_load();
7238523.638030 | 0) 0.068 us | account_entity_enqueue();
7238523.638030 | 0) 0.351 us | update_cfs_shares();
7238523.638031 | 0) 0.053 us | place_entity();
7238523.638031 | 0) 0.082 us | __enqueue_entity();
7238523.638032 | 0) 0.050 us | update_cfs_rq_blocked_load();
7238523.638032 | 0) 3.922 us | }
7238523.638032 | 0) | enqueue_entity() {
7238523.638033 | 0) 0.058 us | update_curr();
7238523.638033 | 0) 0.056 us | __compute_runnable_contrib.part.51();
7238523.638033 | 0) 0.078 us | __update_entity_load_avg_contrib();
7238523.638034 | 0) 0.055 us | update_cfs_rq_blocked_load();
7238523.638034 | 0) 0.064 us | account_entity_enqueue();
7238523.638034 | 0) 0.059 us | update_cfs_shares();
7238523.638035 | 0) 0.050 us | place_entity();
7238523.638036 | 0) 0.057 us | __enqueue_entity();
7238523.638036 | 0) 3.829 us | }
7238523.638037 | 0) 0.057 us | hrtick_update();
7238523.638037 | 0) 8.876 us | }
7238523.638037 | 0) 9.698 us | }
7238523.638037 | 0) 10.113 us | }
7238523.638038 | 0) | ttwu_do_wakeup() {
7238523.638038 | 0) | check_preempt_curr() {
7238523.638038 | 0) | resched_task() {
7238523.638038 | 0) | xen_smp_send_reschedule() {
7238523.638038 | 0) | xen_send_IPI_one() {
7238523.638039 | 0) | notify_remote_via_irq() {
7238523.638039 | 0) | evtchn_from_irq() {
7238523.638039 | 0) | irq_get_irq_data() {
7238523.638039 | 0) 0.051 us | irq_to_desc();
7238523.638039 | 0) 0.518 us | }
7238523.638040 | 0) 0.955 us | }
7238523.638041 | 0) 2.001 us | }
7238523.638041 | 0) 2.391 us | }
7238523.638041 | 0) 2.745 us | }
7238523.638041 | 0) 3.183 us | }
7238523.638042 | 0) 3.663 us | }
7238523.638042 | 0) 4.621 us | }
7238523.638043 | 0) 15.443 us | }
7238523.638043 | 0) 0.067 us | _raw_spin_unlock();
7238523.638043 | 0) 0.167 us | ttwu_stat();
7238523.638044 | 0) 0.087 us | _raw_spin_unlock_irqrestore();
7238523.638044 | 0) 21.447 us | }
7238523.638045 | 0) 21.940 us | }
7238523.638045 | 0) 22.406 us | }
7238523.638045 | 0) 23.071 us | }
7238523.638045 | 0) 0.073 us | _raw_spin_unlock_irqrestore();
7238523.638046 | 0) 24.382 us | }
7238523.638046 | 0) | irq_exit() {
7238523.638047 | 0) 0.085 us | idle_cpu();
7238523.638047 | 0) 0.093 us | rcu_irq_exit();
7238523.638048 | 0) 1.242 us | }
7238523.638048 | 0) 27.410 us | }
7238523.638049 | 0) 0.139 us | add_interrupt_randomness();
7238523.638049 | 0) 0.089 us | note_interrupt();
7238523.638050 | 0) 29.582 us | }
7238523.638050 | 0) 32.112 us | }
7238523.638050 | 0) 32.951 us | }
7238523.638051 | 0) 35.765 us | }
7238523.638051 | 0) 36.170 us | }
7238523.638051 | 0) | irq_exit() {
7238523.638051 | 0) 0.082 us | idle_cpu();
7238523.638052 | 0) 0.071 us | rcu_irq_exit();
7238523.638053 | 0) 1.328 us | }
7238523.638053 | 0) 40.563 us | }
7238523.638054 | 0) | __mmdrop() {
7238523.638054 | 0) | pgd_free() {
7238523.638055 | 0) 0.151 us | _raw_spin_lock();
7238523.638055 | 0) 0.069 us | _raw_spin_unlock();
7238523.638056 | 0) | xen_pgd_free() {
7238523.638056 | 0) 0.067 us | xen_get_user_pgd();
7238523.638057 | 0) | free_pages() {
7238523.638057 | 0) | __free_pages() {
7238523.638057 | 0) | free_hot_cold_page() {
7238523.638058 | 0) 0.080 us | free_pages_prepare();
7238523.638058 | 0) 0.363 us | get_pfnblock_flags_mask();
7238523.638059 | 0) 1.626 us | }
7238523.638059 | 0) 2.317 us | }
7238523.638060 | 0) 2.847 us | }
7238523.638060 | 0) 3.908 us | }
7238523.638060 | 0) | free_pages() {
7238523.638060 | 0) | __free_pages() {
7238523.638061 | 0) | free_hot_cold_page() {
7238523.638061 | 0) 0.083 us | free_pages_prepare();
7238523.638061 | 0) 0.139 us | get_pfnblock_flags_mask();
7238523.638062 | 0) 1.062 us | }
7238523.638062 | 0) 1.534 us | }
7238523.638062 | 0) 2.038 us | }
7238523.638063 | 0) 8.268 us | }
7238523.638064 | 0) 0.160 us | destroy_context();
7238523.638065 | 0) 0.384 us | kmem_cache_free();
7238523.638066 | 0) 11.433 us | }
7238523.638066 | 0) 54.448 us | }
7238523.638066 | 0) 19354026 us | } /* __schedule */
7238523.638067 | 0) 19354026 us | } /* schedule */
7238523.638067 | 0) 19354027 us | } /* schedule_timeout */
7238523.638067 | 0) 0.121 us | down_read();
7238523.638068 | 0) | copy_from_read_buf() {
7238523.638069 | 0) | tty_audit_add_data() {
7238523.638070 | 0) 0.220 us | _raw_spin_lock_irqsave();
7238523.638071 | 0) 0.097 us | _raw_spin_unlock_irqrestore();
7238523.638071 | 0) 0.078 us | _raw_spin_lock_irqsave();
7238523.638072 | 0) 0.077 us | _raw_spin_unlock_irqrestore();
7238523.638072 | 0) 2.795 us | }
7238523.638073 | 0) 4.183 us | }
7238523.638073 | 0) 0.084 us | copy_from_read_buf();
7238523.638074 | 0) 0.078 us | n_tty_set_room();
7238523.638074 | 0) 0.082 us | n_tty_write_wakeup();
7238523.638075 | 0) | __wake_up() {
7238523.638075 | 0) 0.084 us | _raw_spin_lock_irqsave();
7238523.638076 | 0) | __wake_up_common() {
7238523.638076 | 0) 0.095 us | pollwake();
7238523.638077 | 0) 0.819 us | }
7238523.638077 | 0) 0.074 us | _raw_spin_unlock_irqrestore();
7238523.638078 | 0) 2.463 us | }
7238523.638078 | 0) 0.071 us | n_tty_set_room();
7238523.638078 | 0) 0.082 us | up_read();
7238523.638079 | 0) | remove_wait_queue() {
7238523.638079 | 0) 0.082 us | _raw_spin_lock_irqsave();
7238523.638080 | 0) 0.086 us | _raw_spin_unlock_irqrestore();
7238523.638080 | 0) 1.239 us | }
7238523.638081 | 0) 0.142 us | mutex_unlock();
7238523.638081 | 0) 19354047 us | } /* n_tty_read */
7238523.638082 | 0) | tty_ldisc_deref() {
7238523.638082 | 0) 0.064 us | ldsem_up_read();
7238523.638082 | 0) 0.554 us | }
7238523.638083 | 0) 0.074 us | get_seconds();
7238523.638083 | 0) 19354052 us | } /* tty_read */
7238523.638084 | 0) 0.352 us | __fsnotify_parent();
7238523.638085 | 0) 0.178 us | fsnotify();
7238523.638085 | 0) 19354058 us | } /* vfs_read */
7238523.638156 | 0) | vfs_read() {
7238523.638157 | 0) | rw_verify_area() {
7238523.638157 | 0) | security_file_permission() {
7238523.638158 | 0) | apparmor_file_permission() {
7238523.638158 | 0) 0.183 us | common_file_perm();
7238523.638159 | 0) 0.778 us | }
7238523.638159 | 0) 0.081 us | __fsnotify_parent();
7238523.638160 | 0) 0.104 us | fsnotify();
7238523.638160 | 0) 2.662 us | }
7238523.638161 | 0) 3.337 us | }
7238523.638161 | 0) | tty_read() {
7238523.638161 | 0) 0.067 us | tty_paranoia_check();
7238523.638162 | 0) | tty_ldisc_ref_wait() {
7238523.638162 | 0) 0.080 us | } /* ldsem_down_read */
7238523.638163 | 0) 0.637 us | }
7238523.638163 | 0) | n_tty_read() {
7238523.638164 | 0) 0.078 us | _raw_spin_lock_irq();
7238523.638164 | 0) 0.090 us | mutex_lock_interruptible();
7238523.638165 | 0) 0.078 us | down_read();
7238523.638165 | 0) | add_wait_queue() {
7238523.638166 | 0) 0.070 us | _raw_spin_lock_irqsave();
7238523.638166 | 0) 0.084 us | _raw_spin_unlock_irqrestore();
7238523.638167 | 0) 1.111 us | }
7238523.638167 | 0) 0.083 us | tty_hung_up_p();
7238523.638168 | 0) 0.080 us | n_tty_set_room();
7238523.638169 | 0) 0.068 us | up_read();
7238523.638169 | 0) | schedule_timeout() {
7238523.638170 | 0) | schedule() {
7238523.638170 | 0) | __schedule() {
7238523.638171 | 0) 0.078 us | rcu_note_context_switch();
7238523.638171 | 0) 0.081 us | _raw_spin_lock_irq();
7238523.638172 | 0) | deactivate_task() {
7238523.638172 | 0) | dequeue_task() {
7238523.638172 | 0) 0.181 us | update_rq_clock();
7238523.638173 | 0) | dequeue_task_fair() {
7238523.638174 | 0) | dequeue_entity() {
7238523.638174 | 0) | update_curr() {
7238523.638174 | 0) 0.257 us | cpuacct_charge();
7238523.638175 | 0) 0.982 us | }
7238523.638175 | 0) 0.079 us | update_cfs_rq_blocked_load();
7238523.638176 | 0) 0.080 us | clear_buddies();
7238523.638177 | 0) 0.096 us | account_entity_dequeue();
7238523.638177 | 0) | update_cfs_shares() {
7238523.638178 | 0) 0.113 us | update_curr();
7238523.638178 | 0) 0.087 us | account_entity_dequeue();
7238523.638179 | 0) 0.073 us | account_entity_enqueue();
7238523.638179 | 0) 1.948 us | }
7238523.638180 | 0) 5.913 us | }
7238523.638180 | 0) | dequeue_entity() {
7238523.638180 | 0) 0.086 us | update_curr();
7238523.638181 | 0) 0.079 us | update_cfs_rq_blocked_load();
7238523.638182 | 0) 0.076 us | clear_buddies();
7238523.638182 | 0) 0.076 us | account_entity_dequeue();
7238523.638183 | 0) 0.104 us | update_cfs_shares();
7238523.638183 | 0) 3.171 us | }
7238523.638184 | 0) 0.076 us | hrtick_update();
7238523.638184 | 0) 10.785 us | }
7238523.638184 | 0) 12.057 us | }
7238523.638185 | 0) 12.704 us | }
7238523.638185 | 0) | pick_next_task_fair() {
7238523.638185 | 0) 0.074 us | check_cfs_rq_runtime();
7238523.638186 | 0) | pick_next_entity() {
7238523.638186 | 0) 0.067 us | clear_buddies();
7238523.638187 | 0) 0.544 us | }
7238523.638187 | 0) | put_prev_entity() {
7238523.638187 | 0) 0.079 us | check_cfs_rq_runtime();
7238523.638188 | 0) 0.612 us | }
7238523.638188 | 0) | put_prev_entity() {
7238523.638188 | 0) 0.076 us | check_cfs_rq_runtime();
7238523.638189 | 0) 0.618 us | }
7238523.638189 | 0) | set_next_entity() {
7238523.638190 | 0) 0.078 us | update_stats_wait_end();
7238523.638190 | 0) 0.712 us | }
7238523.638190 | 0) 5.023 us | }
7238523.638191 | 0) 0.086 us | paravirt_start_context_switch();
7238523.638192 | 0) 0.070 us | xen_read_cr0();
7238523.638193 | 0) | xen_write_cr0() {
7238523.638193 | 0) 0.085 us | paravirt_get_lazy_mode();
7238523.638194 | 0) 0.085 us | __xen_mc_entry();
7238523.638194 | 0) 0.077 us | paravirt_get_lazy_mode();
7238523.638195 | 0) 1.822 us | }
7238523.638195 | 0) | xen_load_sp0() {
7238523.638195 | 0) 0.074 us | paravirt_get_lazy_mode();
7238523.638196 | 0) 0.085 us | __xen_mc_entry();
7238523.638196 | 0) 0.078 us | paravirt_get_lazy_mode();
7238523.638197 | 0) 1.754 us | }
7238523.638197 | 0) | xen_load_tls() {
7238523.638198 | 0) 0.069 us | paravirt_get_lazy_mode();
7238523.638198 | 0) 0.082 us | paravirt_get_lazy_mode();
7238523.638199 | 0) 0.127 us | load_TLS_descriptor();
7238523.638199 | 0) 0.080 us | load_TLS_descriptor();
7238523.638200 | 0) 0.094 us | load_TLS_descriptor();
7238523.638201 | 0) 0.081 us | paravirt_get_lazy_mode();
7238523.638202 | 0) 4.155 us | }
7238523.638202 | 0) | xen_end_context_switch() {
7238523.638202 | 0) 0.699 us | xen_mc_flush();
7238523.638204 | 0) 0.089 us | paravirt_end_context_switch();
7238523.638204 | 0) 1.915 us | }
7238523.797630 | 0) | finish_task_switch() {
7238523.797634 | 0) | xen_evtchn_do_upcall() {
7238523.797634 | 0) | irq_enter() {
7238523.797634 | 0) 0.134 us | rcu_irq_enter();
7238523.797635 | 0) 0.688 us | }
7238523.797635 | 0) 0.055 us | exit_idle();
7238523.797635 | 0) | __xen_evtchn_do_upcall() {
7238523.797636 | 0) | evtchn_2l_handle_events() {
7238523.797636 | 0) 0.048 us | irq_from_virq();
7238523.797636 | 0) | evtchn_from_irq() {
7238523.797636 | 0) | irq_get_irq_data() {
7238523.797637 | 0) 0.061 us | irq_to_desc();
7238523.797637 | 0) 0.564 us | }
7238523.797637 | 0) 0.954 us | }
7238523.797638 | 0) | get_evtchn_to_irq() {
7238523.797638 | 0) 0.057 us | evtchn_2l_max_channels();
7238523.797638 | 0) 0.409 us | }
7238523.797638 | 0) | generic_handle_irq() {
7238523.797638 | 0) 0.052 us | irq_to_desc();
7238523.797639 | 0) | handle_percpu_irq() {
7238523.797639 | 0) | ack_dynirq() {
7238523.797639 | 0) | evtchn_from_irq() {
7238523.797639 | 0) | irq_get_irq_data() {
7238523.797640 | 0) 0.057 us | irq_to_desc();
7238523.797640 | 0) 0.440 us | }
7238523.797640 | 0) 0.746 us | }
7238523.797640 | 0) 0.056 us | irq_move_irq();
7238523.797641 | 0) 0.058 us | evtchn_2l_clear_pending();
7238523.797641 | 0) 1.729 us | }
7238523.797641 | 0) | handle_irq_event_percpu() {
7238523.797641 | 0) | xen_irq_work_interrupt() {
7238523.797642 | 0) | irq_enter() {
7238523.797642 | 0) 0.053 us | rcu_irq_enter();
7238523.797642 | 0) 0.396 us | }
7238523.797642 | 0) | __wake_up() {
7238523.797643 | 0) 0.053 us | _raw_spin_lock_irqsave();
7238523.797643 | 0) | __wake_up_common() {
7238523.797643 | 0) | autoremove_wake_function() {
7238523.797644 | 0) | default_wake_function() {
7238523.797644 | 0) | try_to_wake_up() {
7238523.797644 | 0) 0.228 us | _raw_spin_lock_irqsave();
7238523.797645 | 0) 0.194 us | task_waking_fair();
7238523.797645 | 0) | select_task_rq_fair() {
7238523.797645 | 0) 0.051 us | source_load();
7238523.797646 | 0) 0.050 us | target_load();
7238523.797646 | 0) 0.067 us | idle_cpu();
7238523.797647 | 0) 0.050 us | cpus_share_cache();
7238523.797647 | 0) 0.068 us | idle_cpu();
7238523.797647 | 0) 1.983 us | }
7238523.797648 | 0) 0.051 us | _raw_spin_lock();
7238523.797648 | 0) | ttwu_do_activate.constprop.124() {
7238523.797648 | 0) | activate_task() {
7238523.797648 | 0) | enqueue_task() {
7238523.797648 | 0) 0.135 us | update_rq_clock();
7238523.797649 | 0) | enqueue_task_fair() {
7238523.797649 | 0) | enqueue_entity() {
7238523.797649 | 0) 0.059 us | update_curr();
7238523.797650 | 0) 0.073 us | __compute_runnable_contrib.part.51();
7238523.797650 | 0) 0.066 us | __update_entity_load_avg_contrib();
7238523.797650 | 0) 0.059 us | update_cfs_rq_blocked_load();
7238523.797651 | 0) 0.064 us | account_entity_enqueue();
7238523.797651 | 0) 0.137 us | update_cfs_shares();
7238523.797651 | 0) 0.054 us | place_entity();
7238523.797652 | 0) 0.074 us | __enqueue_entity();
7238523.797652 | 0) 3.085 us | }
7238523.797652 | 0) | enqueue_entity() {
7238523.797653 | 0) 0.058 us | update_curr();
7238523.797654 | 0) 0.049 us | update_cfs_rq_blocked_load();
7238523.797654 | 0) 0.057 us | account_entity_enqueue();
7238523.797655 | 0) 0.066 us | update_cfs_shares();
7238523.797655 | 0) 0.049 us | place_entity();
7238523.797655 | 0) 0.051 us | __enqueue_entity();
7238523.797656 | 0) 3.432 us | }
7238523.797656 | 0) 0.049 us | hrtick_update();
7238523.797657 | 0) 7.552 us | }
7238523.797657 | 0) 8.414 us | }
7238523.797657 | 0) 8.753 us | }
7238523.797657 | 0) | ttwu_do_wakeup() {
7238523.797657 | 0) | check_preempt_curr() {
7238523.797657 | 0) | resched_task() {
7238523.797658 | 0) | xen_smp_send_reschedule() {
7238523.797658 | 0) | xen_send_IPI_one() {
7238523.797658 | 0) | notify_remote_via_irq() {
7238523.797658 | 0) | evtchn_from_irq() {
7238523.797658 | 0) | irq_get_irq_data() {
7238523.797659 | 0) 0.069 us | irq_to_desc();
7238523.797659 | 0) 0.504 us | }
7238523.797659 | 0) 0.869 us | }
7238523.797660 | 0) 1.940 us | } /* notify_remote_via_irq */
7238523.797660 | 0) 2.319 us | }
7238523.797660 | 0) 2.712 us | }
7238523.797661 | 0) 3.147 us | }
7238523.797661 | 0) 3.625 us | }
7238523.797662 | 0) 4.525 us | }
7238523.797662 | 0) 13.961 us | }
7238523.797662 | 0) 0.069 us | _raw_spin_unlock();
7238523.797663 | 0) 0.168 us | ttwu_stat();
7238523.797663 | 0) 0.076 us | _raw_spin_unlock_irqrestore();
7238523.797664 | 0) 19.821 us | }
7238523.797664 | 0) 20.301 us | }
7238523.797664 | 0) 20.796 us | }
7238523.797664 | 0) 21.367 us | }
7238523.797665 | 0) 0.071 us | _raw_spin_unlock_irqrestore();
7238523.797665 | 0) 22.621 us | }
7238523.797666 | 0) | irq_exit() {
7238523.797666 | 0) 0.085 us | idle_cpu();
7238523.797666 | 0) 0.106 us | rcu_irq_exit();
7238523.797667 | 0) 1.220 us | }
7238523.797667 | 0) 25.712 us | }
7238523.797668 | 0) 0.138 us | add_interrupt_randomness();
7238523.797668 | 0) 0.092 us | note_interrupt();
7238523.797669 | 0) 27.713 us | }
7238523.797669 | 0) 30.163 us | }
7238523.797669 | 0) 31.017 us | }
7238523.797670 | 0) 33.953 us | }
7238523.797670 | 0) 34.384 us | }
7238523.797670 | 0) | irq_exit() {
7238523.797671 | 0) 0.079 us | idle_cpu();
7238523.797671 | 0) 0.072 us | rcu_irq_exit();
7238523.797672 | 0) 1.023 us | }
7238523.797672 | 0) 37.789 us | }
7238523.797672 | 0) 39.298 us | }
7238523.797673 | 0) 159502.1 us | }
7238523.797673 | 0) 159502.8 us | }
7238523.797673 | 0) 159503.5 us | }
7238523.797674 | 0) 0.112 us | down_read();
7238523.797675 | 0) | copy_from_read_buf() {
7238523.797676 | 0) | tty_audit_add_data() {
7238523.797676 | 0) 0.226 us | _raw_spin_lock_irqsave();
7238523.797677 | 0) 0.075 us | _raw_spin_unlock_irqrestore();
7238523.797677 | 0) 0.101 us | _raw_spin_lock_irqsave();
7238523.797678 | 0) 0.068 us | _raw_spin_unlock_irqrestore();
7238523.797679 | 0) 2.656 us | }
7238523.797679 | 0) 3.762 us | }
7238523.797679 | 0) 0.145 us | copy_from_read_buf();
7238523.797680 | 0) 0.068 us | n_tty_set_room();
7238523.797680 | 0) 0.058 us | n_tty_write_wakeup();
7238523.797681 | 0) | __wake_up() {
7238523.797682 | 0) 0.060 us | _raw_spin_lock_irqsave();
7238523.797682 | 0) | __wake_up_common() {
7238523.797683 | 0) 0.083 us | pollwake();
7238523.797683 | 0) 0.739 us | }
7238523.797683 | 0) 0.069 us | _raw_spin_unlock_irqrestore();
7238523.797684 | 0) 2.745 us | }
7238523.797684 | 0) 0.061 us | n_tty_set_room();
7238523.797685 | 0) 0.074 us | up_read();
7238523.797685 | 0) | remove_wait_queue() {
7238523.797685 | 0) 0.075 us | _raw_spin_lock_irqsave();
7238523.797686 | 0) 0.070 us | _raw_spin_unlock_irqrestore();
7238523.797686 | 0) 1.110 us | }
7238523.797687 | 0) 0.146 us | mutex_unlock();
7238523.797687 | 0) 159524.0 us | }
7238523.797688 | 0) | tty_ldisc_deref() {
7238523.797688 | 0) 0.070 us | ldsem_up_read();
7238523.797689 | 0) 0.739 us | }
7238523.797689 | 0) 0.066 us | get_seconds();
7238523.797690 | 0) 159528.3 us | }
7238523.797690 | 0) 0.298 us | __fsnotify_parent();
7238523.797691 | 0) 0.179 us | fsnotify();
7238523.797692 | 0) 159534.6 us | }
7238523.797762 | 0) | vfs_read() {
7238523.797763 | 0) | rw_verify_area() {
7238523.797763 | 0) | security_file_permission() {
7238523.797764 | 0) | apparmor_file_permission() {
7238523.797764 | 0) 0.165 us | common_file_perm();
7238523.797765 | 0) 0.732 us | }
7238523.797765 | 0) 0.081 us | __fsnotify_parent();
7238523.797766 | 0) 0.094 us | fsnotify();
7238523.797766 | 0) 2.711 us | }
7238523.797767 | 0) 3.386 us | }
7238523.797767 | 0) | tty_read() {
7238523.797767 | 0) 0.077 us | tty_paranoia_check();
7238523.797768 | 0) | tty_ldisc_ref_wait() {
7238523.797768 | 0) 0.083 us | ldsem_down_read();
7238523.797769 | 0) 0.686 us | }
7238523.797769 | 0) | n_tty_read() {
7238523.797770 | 0) 0.071 us | _raw_spin_lock_irq();
7238523.797770 | 0) 0.111 us | mutex_lock_interruptible();
7238523.797771 | 0) 0.072 us | down_read();
7238523.797771 | 0) | add_wait_queue() {
7238523.797772 | 0) 0.083 us | _raw_spin_lock_irqsave();
7238523.797772 | 0) 0.085 us | _raw_spin_unlock_irqrestore();
7238523.797773 | 0) 1.124 us | }
7238523.797773 | 0) 0.066 us | tty_hung_up_p();
7238523.797774 | 0) 0.090 us | n_tty_set_room();
7238523.797774 | 0) 0.064 us | up_read();
7238523.797775 | 0) | schedule_timeout() {
7238523.797775 | 0) | schedule() {
7238523.797775 | 0) | __schedule() {
7238523.797776 | 0) 0.083 us | rcu_note_context_switch();
7238523.797776 | 0) 0.078 us | _raw_spin_lock_irq();
7238523.797777 | 0) | deactivate_task() {
7238523.797777 | 0) | dequeue_task() {
7238523.797777 | 0) 0.191 us | update_rq_clock();
7238523.797778 | 0) | dequeue_task_fair() {
7238523.797778 | 0) | dequeue_entity() {
7238523.797779 | 0) | update_curr() {
7238523.797779 | 0) 0.179 us | cpuacct_charge();
7238523.797780 | 0) 0.902 us | }
7238523.797780 | 0) 0.070 us | __update_entity_load_avg_contrib();
7238523.797781 | 0) 0.152 us | update_cfs_rq_blocked_load();
7238523.797781 | 0) 0.073 us | clear_buddies();
7238523.797782 | 0) 0.074 us | account_entity_dequeue();
7238523.797783 | 0) | update_cfs_shares() {
7238523.797783 | 0) 0.111 us | update_curr();
7238523.797783 | 0) 0.082 us | account_entity_dequeue();
7238523.797784 | 0) 0.081 us | account_entity_enqueue();
7238523.797785 | 0) 2.330 us | }
7238523.797785 | 0) 6.633 us | } /* dequeue_entity */
7238523.797786 | 0) | dequeue_entity() {
7238523.797786 | 0) 0.078 us | update_curr();
7238523.797787 | 0) 0.086 us | update_cfs_rq_blocked_load();
7238523.797787 | 0) 0.076 us | clear_buddies();
7238523.797788 | 0) 0.079 us | account_entity_dequeue();
7238523.797789 | 0) 0.074 us | update_cfs_shares();
7238523.797789 | 0) 3.287 us | }
7238523.797789 | 0) 0.074 us | hrtick_update();
7238523.797790 | 0) 11.606 us | }
7238523.797790 | 0) 12.879 us | }
7238523.797790 | 0) 13.406 us | }
7238523.797791 | 0) | pick_next_task_fair() {
7238523.797791 | 0) 0.073 us | check_cfs_rq_runtime();
7238523.797792 | 0) | pick_next_entity() {
7238523.797792 | 0) 0.076 us | clear_buddies();
7238523.797793 | 0) 0.663 us | }
7238523.797793 | 0) | put_prev_entity() {
7238523.797793 | 0) 0.076 us | check_cfs_rq_runtime();
7238523.797794 | 0) 0.598 us | }
7238523.797794 | 0) | put_prev_entity() {
7238523.797794 | 0) 0.078 us | check_cfs_rq_runtime();
7238523.797795 | 0) 0.618 us | }
7238523.797795 | 0) | set_next_entity() {
7238523.797795 | 0) 0.096 us | update_stats_wait_end();
7238523.797796 | 0) 0.738 us | }
7238523.797796 | 0) 5.222 us | }
7238523.797797 | 0) 0.078 us | paravirt_start_context_switch();
7238523.797798 | 0) 0.071 us | xen_read_cr0();
7238523.797799 | 0) | xen_write_cr0() {
7238523.797799 | 0) 0.078 us | paravirt_get_lazy_mode();
7238523.797800 | 0) 0.084 us | __xen_mc_entry();
7238523.797800 | 0) 0.076 us | paravirt_get_lazy_mode();
7238523.797801 | 0) 1.798 us | }
7238523.797801 | 0) | xen_load_sp0() {
7238523.797801 | 0) 0.080 us | paravirt_get_lazy_mode();
7238523.797802 | 0) 0.076 us | __xen_mc_entry();
7238523.797802 | 0) 0.073 us | paravirt_get_lazy_mode();
7238523.797803 | 0) 1.623 us | }
7238523.797803 | 0) | xen_load_tls() {
7238523.797803 | 0) 0.082 us | paravirt_get_lazy_mode();
7238523.797804 | 0) 0.084 us | paravirt_get_lazy_mode();
7238523.797804 | 0) 0.136 us | load_TLS_descriptor();
7238523.797805 | 0) 0.072 us | load_TLS_descriptor();
7238523.797806 | 0) 0.080 us | load_TLS_descriptor();
7238523.797806 | 0) 0.088 us | paravirt_get_lazy_mode();
7238523.797807 | 0) 3.360 us | }
7238523.797807 | 0) | xen_end_context_switch() {
7238523.797807 | 0) 0.601 us | xen_mc_flush();
7238523.797808 | 0) 0.098 us | paravirt_end_context_switch();
7238523.797809 | 0) 1.902 us | }
7238524.005649 | 0) | finish_task_switch() {
7238524.005653 | 0) | xen_evtchn_do_upcall() {
7238524.005653 | 0) | irq_enter() {
7238524.005653 | 0) 0.138 us | rcu_irq_enter();
7238524.005654 | 0) 0.753 us | }
7238524.005654 | 0) 0.056 us | exit_idle();
7238524.005655 | 0) | __xen_evtchn_do_upcall() {
7238524.005655 | 0) | evtchn_2l_handle_events() {
7238524.005655 | 0) 0.057 us | irq_from_virq();
7238524.005656 | 0) | evtchn_from_irq() {
7238524.005656 | 0) | irq_get_irq_data() {
7238524.005656 | 0) 0.050 us | irq_to_desc();
7238524.005656 | 0) 0.499 us | }
7238524.005657 | 0) 0.958 us | }
7238524.005657 | 0) | get_evtchn_to_irq() {
7238524.005657 | 0) 0.057 us | evtchn_2l_max_channels();
7238524.005658 | 0) 0.400 us | }
7238524.005659 | 0) | generic_handle_irq() {
7238524.005659 | 0) 0.052 us | irq_to_desc();
7238524.005659 | 0) | handle_percpu_irq() {
7238524.005659 | 0) | ack_dynirq() {
7238524.005659 | 0) | evtchn_from_irq() {
7238524.005660 | 0) | irq_get_irq_data() {
7238524.005660 | 0) 0.056 us | irq_to_desc();
7238524.005660 | 0) 0.439 us | }
7238524.005660 | 0) 0.739 us | }
7238524.005661 | 0) 0.051 us | irq_move_irq();
7238524.005661 | 0) 0.051 us | evtchn_2l_clear_pending();
7238524.005661 | 0) 1.963 us | }
7238524.005662 | 0) | handle_irq_event_percpu() {
7238524.005662 | 0) | xen_irq_work_interrupt() {
7238524.005662 | 0) | irq_enter() {
7238524.005662 | 0) 0.053 us | rcu_irq_enter();
7238524.005663 | 0) 0.392 us | }
7238524.005663 | 0) | __wake_up() {
7238524.005663 | 0) 0.058 us | _raw_spin_lock_irqsave();
7238524.005664 | 0) | __wake_up_common() {
7238524.005664 | 0) | autoremove_wake_function() {
7238524.005664 | 0) | default_wake_function() {
7238524.005665 | 0) | try_to_wake_up() {
7238524.005665 | 0) 0.226 us | _raw_spin_lock_irqsave();
7238524.005665 | 0) 0.392 us | task_waking_fair();
7238524.005666 | 0) | select_task_rq_fair() {
7238524.005666 | 0) 0.067 us | source_load();
7238524.005667 | 0) 0.057 us | target_load();
7238524.005667 | 0) 0.065 us | idle_cpu();
7238524.005668 | 0) 0.050 us | cpus_share_cache();
7238524.005668 | 0) 0.080 us | idle_cpu();
7238524.005668 | 0) 2.053 us | }
7238524.005669 | 0) 0.051 us | _raw_spin_lock();
7238524.005669 | 0) | ttwu_do_activate.constprop.124() {
7238524.005669 | 0) | activate_task() {
7238524.005669 | 0) | enqueue_task() {
7238524.005669 | 0) 0.165 us | update_rq_clock();
7238524.005670 | 0) | enqueue_task_fair() {
7238524.005670 | 0) | enqueue_entity() {
7238524.005670 | 0) 0.065 us | update_curr();
7238524.005671 | 0) 0.078 us | __compute_runnable_contrib.part.51();
7238524.005671 | 0) 0.070 us | __update_entity_load_avg_contrib();
7238524.005671 | 0) 0.051 us | update_cfs_rq_blocked_load();
7238524.005672 | 0) 0.069 us | account_entity_enqueue();
7238524.005672 | 0) 0.132 us | update_cfs_shares();
7238524.005673 | 0) 0.054 us | place_entity();
7238524.005673 | 0) 0.081 us | __enqueue_entity();
7238524.005673 | 0) 3.111 us | }
7238524.005673 | 0) | enqueue_entity() {
7238524.005674 | 0) 0.059 us | update_curr();
7238524.005674 | 0) 0.057 us | update_cfs_rq_blocked_load();
7238524.005674 | 0) 0.067 us | account_entity_enqueue();
7238524.005675 | 0) 0.082 us | update_cfs_shares();
7238524.005675 | 0) 0.120 us | place_entity();
7238524.005675 | 0) 0.051 us | __enqueue_entity();
7238524.005676 | 0) 2.075 us | }
7238524.005676 | 0) 0.049 us | hrtick_update();
7238524.005676 | 0) 6.167 us | }
7238524.005676 | 0) 6.979 us | }
7238524.005676 | 0) 7.317 us | }
7238524.005677 | 0) | ttwu_do_wakeup() {
7238524.005677 | 0) | check_preempt_curr() {
7238524.005677 | 0) | resched_task() {
7238524.005677 | 0) | xen_smp_send_reschedule() {
7238524.005677 | 0) | xen_send_IPI_one() {
7238524.005678 | 0) | notify_remote_via_irq() {
7238524.005678 | 0) | evtchn_from_irq() {
7238524.005678 | 0) | irq_get_irq_data() {
7238524.005678 | 0) 0.051 us | irq_to_desc();
7238524.005679 | 0) 0.545 us | }
7238524.005679 | 0) 0.910 us | }
7238524.005680 | 0) 1.962 us | } /* notify_remote_via_irq */
7238524.005680 | 0) 2.332 us | }
7238524.005680 | 0) 2.684 us | }
7238524.005681 | 0) 3.606 us | }
7238524.005681 | 0) 4.064 us | }
7238524.005682 | 0) 5.129 us | }
7238524.005682 | 0) 13.194 us | }
7238524.005683 | 0) 0.066 us | _raw_spin_unlock();
7238524.005683 | 0) 0.165 us | ttwu_stat();
7238524.005684 | 0) 0.070 us | _raw_spin_unlock_irqrestore();
7238524.005684 | 0) 19.634 us | }
7238524.005685 | 0) 20.080 us | }
7238524.005685 | 0) 20.608 us | }
7238524.005685 | 0) 21.348 us | }
7238524.005685 | 0) 0.084 us | _raw_spin_unlock_irqrestore();
7238524.005686 | 0) 22.728 us | }
7238524.005686 | 0) | irq_exit() {
7238524.005687 | 0) 0.077 us | idle_cpu();
7238524.005687 | 0) 0.093 us | rcu_irq_exit();
7238524.005688 | 0) 1.101 us | }
7238524.005688 | 0) 25.644 us | }
7238524.005688 | 0) 0.138 us | add_interrupt_randomness();
7238524.005689 | 0) 0.083 us | note_interrupt();
7238524.005689 | 0) 27.672 us | }
7238524.005690 | 0) 30.410 us | }
7238524.005690 | 0) 31.458 us | }
7238524.005690 | 0) 35.276 us | }
7238524.005691 | 0) 35.797 us | }
7238524.005691 | 0) | irq_exit() {
7238524.005691 | 0) 0.066 us | idle_cpu();
7238524.005692 | 0) 0.080 us | rcu_irq_exit();
7238524.005692 | 0) 1.110 us | }
7238524.005693 | 0) 39.440 us | }
7238524.005693 | 0) 41.142 us | }
7238524.005694 | 0) 207918.1 us | }
7238524.005694 | 0) 207918.7 us | }
7238524.005694 | 0) 207919.4 us | }
7238524.005695 | 0) 0.068 us | down_read();
7238524.005696 | 0) | copy_from_read_buf() {
7238524.005697 | 0) | tty_audit_add_data() {
7238524.005697 | 0) 0.233 us | _raw_spin_lock_irqsave();
7238524.005698 | 0) 0.076 us | _raw_spin_unlock_irqrestore();
7238524.005699 | 0) 0.076 us | _raw_spin_lock_irqsave();
7238524.005699 | 0) 0.078 us | _raw_spin_unlock_irqrestore();
7238524.005700 | 0) 2.696 us | }
7238524.005700 | 0) 4.335 us | }
7238524.005701 | 0) 0.086 us | copy_from_read_buf();
7238524.005701 | 0) 0.074 us | n_tty_set_room();
7238524.005702 | 0) 0.085 us | n_tty_write_wakeup();
7238524.005702 | 0) | __wake_up() {
7238524.005703 | 0) 0.061 us | _raw_spin_lock_irqsave();
7238524.005703 | 0) | __wake_up_common() {
7238524.005703 | 0) 0.080 us | pollwake();
7238524.005704 | 0) 0.687 us | }
7238524.005704 | 0) 0.063 us | _raw_spin_unlock_irqrestore();
7238524.005705 | 0) 2.040 us | }
7238524.005705 | 0) 0.071 us | n_tty_set_room();
7238524.005706 | 0) 0.074 us | up_read();
7238524.005706 | 0) | remove_wait_queue() {
7238524.005706 | 0) 0.074 us | _raw_spin_lock_irqsave();
7238524.005707 | 0) 0.069 us | _raw_spin_unlock_irqrestore();
7238524.005707 | 0) 1.076 us | }
7238524.005708 | 0) 0.139 us | mutex_unlock();
7238524.005708 | 0) 207939.0 us | }
7238524.005709 | 0) | tty_ldisc_deref() {
7238524.005709 | 0) 0.077 us | ldsem_up_read();
7238524.005710 | 0) 0.702 us | }
7238524.005710 | 0) 0.068 us | get_seconds();
7238524.005711 | 0) 207943.4 us | }
7238524.005712 | 0) 0.301 us | __fsnotify_parent();
7238524.005713 | 0) 0.157 us | fsnotify();
7238524.005713 | 0) 207950.3 us | }
7238524.005783 | 0) | vfs_read() {
7238524.005784 | 0) | rw_verify_area() {
7238524.005784 | 0) | security_file_permission() {
7238524.005785 | 0) | apparmor_file_permission() {
7238524.005785 | 0) 0.164 us | common_file_perm();
7238524.005786 | 0) 0.790 us | }
7238524.005786 | 0) 0.080 us | __fsnotify_parent();
7238524.005787 | 0) 0.094 us | fsnotify();
7238524.005787 | 0) 2.683 us | }
7238524.005788 | 0) 3.313 us | }
7238524.005788 | 0) | tty_read() {
7238524.005788 | 0) 0.087 us | tty_paranoia_check();
7238524.005789 | 0) | tty_ldisc_ref_wait() {
7238524.005789 | 0) 0.080 us | ldsem_down_read();
7238524.005790 | 0) 0.683 us | }
7238524.005790 | 0) | n_tty_read() {
7238524.005791 | 0) 0.080 us | _raw_spin_lock_irq();
7238524.005791 | 0) 0.104 us | mutex_lock_interruptible();
7238524.005792 | 0) 0.070 us | down_read();
7238524.005792 | 0) | add_wait_queue() {
7238524.005793 | 0) 0.079 us | _raw_spin_lock_irqsave();
7238524.005793 | 0) 0.087 us | _raw_spin_unlock_irqrestore();
7238524.005794 | 0) 1.147 us | }
7238524.005794 | 0) 0.078 us | tty_hung_up_p();
7238524.005795 | 0) 0.071 us | n_tty_set_room();
7238524.005795 | 0) 0.077 us | up_read();
7238524.005796 | 0) | schedule_timeout() {
7238524.005796 | 0) | schedule() {
7238524.005796 | 0) | __schedule() {
7238524.005797 | 0) 0.087 us | rcu_note_context_switch();
7238524.005797 | 0) 0.075 us | _raw_spin_lock_irq();
7238524.005798 | 0) | deactivate_task() {
7238524.005798 | 0) | dequeue_task() {
7238524.005798 | 0) 0.177 us | update_rq_clock();
7238524.005799 | 0) | dequeue_task_fair() {
7238524.005799 | 0) | dequeue_entity() {
7238524.005800 | 0) | update_curr() {
7238524.005800 | 0) 0.334 us | cpuacct_charge();
7238524.005801 | 0) 1.199 us | }
7238524.005802 | 0) 0.081 us | __update_entity_load_avg_contrib();
7238524.005802 | 0) 0.064 us | update_cfs_rq_blocked_load();
7238524.005803 | 0) 0.076 us | clear_buddies();
7238524.005803 | 0) 0.079 us | account_entity_dequeue();
7238524.005804 | 0) | update_cfs_shares() {
7238524.005804 | 0) 0.108 us | update_curr();
7238524.005805 | 0) 0.083 us | account_entity_dequeue();
7238524.005805 | 0) 0.084 us | account_entity_enqueue();
7238524.005806 | 0) 1.869 us | }
7238524.005806 | 0) 6.530 us | } /* dequeue_entity */
7238524.005807 | 0) | dequeue_entity() {
7238524.005807 | 0) 0.104 us | update_curr();
7238524.005808 | 0) 0.115 us | __update_entity_load_avg_contrib();
7238524.005808 | 0) 0.069 us | update_cfs_rq_blocked_load();
7238524.005809 | 0) 0.066 us | clear_buddies();
7238524.005809 | 0) 0.086 us | account_entity_dequeue();
7238524.005810 | 0) 0.102 us | update_cfs_shares();
7238524.005811 | 0) 3.907 us | }
7238524.005811 | 0) 0.071 us | hrtick_update();
7238524.005812 | 0) 12.301 us | }
7238524.005812 | 0) 13.546 us | }
7238524.005812 | 0) 14.105 us | }
7238524.005812 | 0) | pick_next_task_fair() {
7238524.005813 | 0) 0.078 us | check_cfs_rq_runtime();
7238524.005813 | 0) | pick_next_entity() {
7238524.005814 | 0) 0.071 us | clear_buddies();
7238524.005815 | 0) 0.585 us | }
7238524.005815 | 0) | put_prev_entity() {
7238524.005815 | 0) 0.073 us | check_cfs_rq_runtime();
7238524.005816 | 0) 0.717 us | }
7238524.005816 | 0) | put_prev_entity() {
7238524.005817 | 0) 0.080 us | check_cfs_rq_runtime();
7238524.005817 | 0) 0.687 us | }
7238524.005817 | 0) | set_next_entity() {
7238524.005818 | 0) 0.091 us | update_stats_wait_end();
7238524.005818 | 0) 0.786 us | }
7238524.005819 | 0) 6.135 us | }
7238524.005820 | 0) 0.091 us | paravirt_start_context_switch();
7238524.005821 | 0) 0.089 us | xen_read_cr0();
7238524.005821 | 0) | xen_write_cr0() {
7238524.005821 | 0) 0.078 us | paravirt_get_lazy_mode();
7238524.005822 | 0) 0.083 us | __xen_mc_entry();
7238524.005823 | 0) 0.074 us | paravirt_get_lazy_mode();
7238524.005823 | 0) 1.657 us | }
7238524.005823 | 0) | xen_load_sp0() {
7238524.005824 | 0) 0.074 us | paravirt_get_lazy_mode();
7238524.005824 | 0) 0.083 us | __xen_mc_entry();
7238524.005825 | 0) 0.087 us | paravirt_get_lazy_mode();
7238524.005825 | 0) 1.764 us | }
7238524.005826 | 0) | xen_load_tls() {
7238524.005826 | 0) 0.077 us | paravirt_get_lazy_mode();
7238524.005826 | 0) 0.084 us | paravirt_get_lazy_mode();
7238524.005827 | 0) 0.150 us | load_TLS_descriptor();
7238524.005828 | 0) 0.082 us | load_TLS_descriptor();
7238524.005828 | 0) 0.084 us | load_TLS_descriptor();
7238524.005829 | 0) 0.076 us | paravirt_get_lazy_mode();
7238524.005829 | 0) 3.388 us | }
7238524.005829 | 0) | xen_end_context_switch() {
7238524.005830 | 0) 0.731 us | xen_mc_flush();
7238524.005831 | 0) 0.093 us | paravirt_end_context_switch();
7238524.005831 | 0) 1.836 us | }
7238524.141853 | 0) | finish_task_switch() {
7238524.141857 | 0) | xen_evtchn_do_upcall() {
7238524.141858 | 0) | irq_enter() {
7238524.141858 | 0) 0.133 us | rcu_irq_enter();
7238524.141859 | 0) 0.766 us | }
7238524.141859 | 0) 0.056 us | exit_idle();
7238524.141859 | 0) | __xen_evtchn_do_upcall() {
7238524.141859 | 0) | evtchn_2l_handle_events() {
7238524.141860 | 0) 0.049 us | irq_from_virq();
7238524.141860 | 0) | evtchn_from_irq() {
7238524.141860 | 0) | irq_get_irq_data() {
7238524.141860 | 0) 0.058 us | irq_to_desc();
7238524.141861 | 0) 0.498 us | }
7238524.141861 | 0) 0.897 us | }
7238524.141861 | 0) | get_evtchn_to_irq() {
7238524.141861 | 0) 0.049 us | evtchn_2l_max_channels();
7238524.141862 | 0) 0.392 us | }
7238524.141862 | 0) | generic_handle_irq() {
7238524.141862 | 0) 0.061 us | irq_to_desc();
7238524.141862 | 0) | handle_percpu_irq() {
7238524.141863 | 0) | ack_dynirq() {
7238524.141863 | 0) | evtchn_from_irq() {
7238524.141863 | 0) | irq_get_irq_data() {
7238524.141863 | 0) 0.051 us | irq_to_desc();
7238524.141863 | 0) 0.439 us | }
7238524.141864 | 0) 0.745 us | }
7238524.141864 | 0) 0.049 us | irq_move_irq();
7238524.141864 | 0) 0.060 us | evtchn_2l_clear_pending();
7238524.141864 | 0) 1.714 us | }
7238524.141865 | 0) | handle_irq_event_percpu() {
7238524.141865 | 0) | xen_irq_work_interrupt() {
7238524.141865 | 0) | irq_enter() {
7238524.141865 | 0) 0.053 us | rcu_irq_enter();
7238524.141866 | 0) 0.371 us | }
7238524.141866 | 0) | __wake_up() {
7238524.141866 | 0) 0.051 us | _raw_spin_lock_irqsave();
7238524.141867 | 0) | __wake_up_common() {
7238524.141867 | 0) | autoremove_wake_function() {
7238524.141867 | 0) | default_wake_function() {
7238524.141867 | 0) | try_to_wake_up() {
7238524.141868 | 0) 0.213 us | _raw_spin_lock_irqsave();
7238524.141869 | 0) 0.196 us | task_waking_fair();
7238524.141870 | 0) | select_task_rq_fair() {
7238524.141870 | 0) 0.051 us | source_load();
7238524.141870 | 0) 0.049 us | target_load();
7238524.141871 | 0) 0.065 us | idle_cpu();
7238524.141871 | 0) 0.051 us | cpus_share_cache();
7238524.141872 | 0) 0.078 us | idle_cpu();
7238524.141872 | 0) 2.427 us | }
7238524.141873 | 0) 0.050 us | _raw_spin_lock();
7238524.141873 | 0) | ttwu_do_activate.constprop.124() {
7238524.141873 | 0) | activate_task() {
7238524.141873 | 0) | enqueue_task() {
7238524.141873 | 0) 0.170 us | update_rq_clock();
7238524.141874 | 0) | enqueue_task_fair() {
7238524.141874 | 0) | enqueue_entity() {
7238524.141874 | 0) 0.058 us | update_curr();
7238524.141875 | 0) 0.076 us | __compute_runnable_contrib.part.51();
7238524.141875 | 0) 0.059 us | __update_entity_load_avg_contrib();
7238524.141875 | 0) 0.060 us | update_cfs_rq_blocked_load();
7238524.141876 | 0) 0.064 us | account_entity_enqueue();
7238524.141876 | 0) 0.123 us | update_cfs_shares();
7238524.141876 | 0) 0.055 us | place_entity();
7238524.141877 | 0) 0.078 us | __enqueue_entity();
7238524.141877 | 0) 3.039 us | }
7238524.141877 | 0) | enqueue_entity() {
7238524.141877 | 0) 0.065 us | update_curr();
7238524.141878 | 0) 0.050 us | update_cfs_rq_blocked_load();
7238524.141878 | 0) 0.049 us | account_entity_enqueue();
7238524.141878 | 0) 0.081 us | update_cfs_shares();
7238524.141879 | 0) 0.049 us | place_entity();
7238524.141879 | 0) 0.051 us | __enqueue_entity();
7238524.141879 | 0) 2.021 us | }
7238524.141880 | 0) 0.049 us | hrtick_update();
7238524.141880 | 0) 6.040 us | }
7238524.141880 | 0) 6.874 us | }
7238524.141880 | 0) 7.212 us | }
7238524.141880 | 0) | ttwu_do_wakeup() {
7238524.141881 | 0) | check_preempt_curr() {
7238524.141881 | 0) | resched_task() {
7238524.141881 | 0) | xen_smp_send_reschedule() {
7238524.141881 | 0) | xen_send_IPI_one() {
7238524.141881 | 0) | notify_remote_via_irq() {
7238524.141881 | 0) | evtchn_from_irq() {
7238524.141882 | 0) | irq_get_irq_data() {
7238524.141882 | 0) 0.049 us | irq_to_desc();
7238524.141882 | 0) 0.497 us | }
7238524.141882 | 0) 0.860 us | }
7238524.141883 | 0) 1.882 us | } /* notify_remote_via_irq */
7238524.141884 | 0) 2.257 us | }
7238524.141884 | 0) 2.619 us | }
7238524.141884 | 0) 3.079 us | }
7238524.141884 | 0) 3.526 us | }
7238524.141885 | 0) 4.485 us | }
7238524.141885 | 0) 12.333 us | }
7238524.141885 | 0) 0.062 us | _raw_spin_unlock();
7238524.141886 | 0) 0.169 us | ttwu_stat();
7238524.141887 | 0) 0.076 us | _raw_spin_unlock_irqrestore();
7238524.141887 | 0) 19.434 us | }
7238524.141887 | 0) 19.909 us | }
7238524.141888 | 0) 20.377 us | }
7238524.141888 | 0) 21.020 us | }
7238524.141888 | 0) 0.075 us | _raw_spin_unlock_irqrestore();
7238524.141888 | 0) 22.268 us | }
7238524.141889 | 0) | irq_exit() {
7238524.141889 | 0) 0.087 us | idle_cpu();
7238524.141889 | 0) 0.101 us | rcu_irq_exit();
7238524.141890 | 0) 1.127 us | }
7238524.141890 | 0) 25.163 us | }
7238524.141891 | 0) 0.133 us | add_interrupt_randomness();
7238524.141892 | 0) 0.083 us | note_interrupt();
7238524.141892 | 0) 27.453 us | }
7238524.141893 | 0) 30.024 us | }
7238524.141893 | 0) 30.898 us | }
7238524.141893 | 0) 33.683 us | }
7238524.141893 | 0) 34.097 us | }
7238524.141894 | 0) | irq_exit() {
7238524.141894 | 0) 0.065 us | idle_cpu();
7238524.141895 | 0) 0.076 us | rcu_irq_exit();
7238524.141895 | 0) 1.135 us | }
7238524.141895 | 0) 37.746 us | }
7238524.141896 | 0) 39.634 us | }
7238524.141897 | 0) 136100.0 us | }
7238524.141897 | 0) 136100.6 us | }
7238524.141897 | 0) 136101.4 us | }
7238524.141898 | 0) 0.093 us | down_read();
7238524.141899 | 0) | copy_from_read_buf() {
7238524.141900 | 0) | tty_audit_add_data() {
7238524.141900 | 0) 0.238 us | _raw_spin_lock_irqsave();
7238524.141901 | 0) 0.069 us | _raw_spin_unlock_irqrestore();
7238524.141901 | 0) 0.090 us | _raw_spin_lock_irqsave();
7238524.141902 | 0) 0.077 us | _raw_spin_unlock_irqrestore();
7238524.141902 | 0) 2.513 us | }
7238524.141903 | 0) 3.632 us | }
7238524.141903 | 0) 0.085 us | copy_from_read_buf();
7238524.141904 | 0) 0.066 us | n_tty_set_room();
7238524.141905 | 0) 0.067 us | n_tty_write_wakeup();
7238524.141905 | 0) | __wake_up() {
7238524.141906 | 0) 0.070 us | _raw_spin_lock_irqsave();
7238524.141906 | 0) | __wake_up_common() {
7238524.141906 | 0) 0.086 us | pollwake();
7238524.141907 | 0) 0.620 us | }
7238524.141907 | 0) 0.064 us | _raw_spin_unlock_irqrestore();
7238524.141907 | 0) 1.980 us | }
7238524.141908 | 0) 0.059 us | n_tty_set_room();
7238524.141909 | 0) 0.071 us | up_read();
7238524.141909 | 0) | remove_wait_queue() {
7238524.141909 | 0) 0.079 us | _raw_spin_lock_irqsave();
7238524.141910 | 0) 0.082 us | _raw_spin_unlock_irqrestore();
7238524.141910 | 0) 1.164 us | }
7238524.141910 | 0) 0.142 us | mutex_unlock();
7238524.141911 | 0) 136120.9 us | }
7238524.141911 | 0) | tty_ldisc_deref() {
7238524.141912 | 0) 0.062 us | ldsem_up_read();
7238524.141912 | 0) 0.593 us | }
7238524.141912 | 0) 0.079 us | get_seconds();
7238524.141913 | 0) 136125.1 us | }
7238524.141914 | 0) 0.280 us | __fsnotify_parent();
7238524.141915 | 0) 0.187 us | fsnotify();
7238524.141915 | 0) 136131.2 us | }
7238524.141988 | 0) | vfs_read() {
7238524.141989 | 0) | rw_verify_area() {
7238524.141989 | 0) | security_file_permission() {
7238524.141989 | 0) | apparmor_file_permission() {
7238524.141990 | 0) 0.149 us | common_file_perm();
7238524.141990 | 0) 0.774 us | }
7238524.141991 | 0) 0.079 us | __fsnotify_parent();
7238524.141991 | 0) 0.095 us | fsnotify();
7238524.141992 | 0) 2.558 us | }
7238524.141992 | 0) 3.300 us | }
7238524.141993 | 0) | tty_read() {
7238524.141993 | 0) 0.076 us | tty_paranoia_check();
7238524.141994 | 0) | tty_ldisc_ref_wait() {
7238524.141994 | 0) 0.081 us | ldsem_down_read();
7238524.141995 | 0) 0.689 us | }
7238524.141995 | 0) | n_tty_read() {
7238524.141995 | 0) 0.073 us | _raw_spin_lock_irq();
7238524.141996 | 0) 0.110 us | mutex_lock_interruptible();
7238524.141997 | 0) 0.069 us | down_read();
7238524.141998 | 0) | add_wait_queue() {
7238524.141998 | 0) 0.079 us | _raw_spin_lock_irqsave();
7238524.141999 | 0) 0.078 us | _raw_spin_unlock_irqrestore();
7238524.141999 | 0) 1.201 us | }
7238524.142000 | 0) 0.067 us | tty_hung_up_p();
7238524.142000 | 0) 0.078 us | n_tty_set_room();
7238524.142001 | 0) 0.079 us | up_read();
7238524.142001 | 0) | schedule_timeout() {
7238524.142002 | 0) | schedule() {
7238524.142002 | 0) | __schedule() {
7238524.142002 | 0) 0.076 us | rcu_note_context_switch();
7238524.142003 | 0) 0.080 us | _raw_spin_lock_irq();
7238524.142004 | 0) | deactivate_task() {
7238524.142004 | 0) | dequeue_task() {
7238524.142004 | 0) 0.178 us | update_rq_clock();
7238524.142005 | 0) | dequeue_task_fair() {
7238524.142005 | 0) | dequeue_entity() {
7238524.142005 | 0) | update_curr() {
7238524.142006 | 0) 0.263 us | cpuacct_charge();
7238524.142007 | 0) 0.965 us | }
7238524.142007 | 0) 0.075 us | update_cfs_rq_blocked_load();
7238524.142008 | 0) 0.065 us | clear_buddies();
7238524.142008 | 0) 0.084 us | account_entity_dequeue();
7238524.142009 | 0) | update_cfs_shares() {
7238524.142009 | 0) 0.115 us | update_curr();
7238524.142010 | 0) 0.084 us | account_entity_dequeue();
7238524.142010 | 0) 0.068 us | account_entity_enqueue();
7238524.142011 | 0) 1.754 us | }
7238524.142011 | 0) 5.580 us | }
7238524.142012 | 0) | dequeue_entity() {
7238524.142012 | 0) 0.089 us | update_curr();
7238524.142012 | 0) 0.101 us | update_cfs_rq_blocked_load();
7238524.142013 | 0) 0.076 us | clear_buddies();
7238524.142013 | 0) 0.078 us | account_entity_dequeue();
7238524.142014 | 0) 0.076 us | update_cfs_shares();
7238524.142015 | 0) 3.071 us | }
7238524.142015 | 0) 0.078 us | hrtick_update();
7238524.142016 | 0) 10.525 us | }
7238524.142016 | 0) 11.803 us | }
7238524.142016 | 0) 12.447 us | }
7238524.142017 | 0) | pick_next_task_fair() {
7238524.142017 | 0) 0.069 us | check_cfs_rq_runtime();
7238524.142017 | 0) | pick_next_entity() {
7238524.142018 | 0) 0.061 us | clear_buddies();
7238524.142018 | 0) 0.601 us | }
7238524.142019 | 0) | put_prev_entity() {
7238524.142019 | 0) 0.069 us | check_cfs_rq_runtime();
7238524.142019 | 0) 0.605 us | }
7238524.142020 | 0) | put_prev_entity() {
7238524.142020 | 0) 0.076 us | check_cfs_rq_runtime();
7238524.142020 | 0) 0.609 us | }
7238524.142021 | 0) | set_next_entity() {
7238524.142021 | 0) 0.088 us | update_stats_wait_end();
7238524.142022 | 0) 0.768 us | }
7238524.142022 | 0) 5.183 us | }
7238524.142023 | 0) 0.080 us | paravirt_start_context_switch();
7238524.142024 | 0) 0.076 us | xen_read_cr0();
7238524.142024 | 0) | xen_write_cr0() {
7238524.142025 | 0) 0.088 us | paravirt_get_lazy_mode();
7238524.142025 | 0) 0.096 us | __xen_mc_entry();
7238524.142026 | 0) 0.083 us | paravirt_get_lazy_mode();
7238524.142026 | 0) 1.802 us | }
7238524.142026 | 0) | xen_load_sp0() {
7238524.142027 | 0) 0.074 us | paravirt_get_lazy_mode();
7238524.142027 | 0) 0.098 us | __xen_mc_entry();
7238524.142028 | 0) 0.073 us | paravirt_get_lazy_mode();
7238524.142029 | 0) 2.289 us | }
7238524.142029 | 0) | xen_load_tls() {
7238524.142029 | 0) 0.073 us | paravirt_get_lazy_mode();
7238524.142030 | 0) 0.079 us | paravirt_get_lazy_mode();
7238524.142031 | 0) 0.135 us | load_TLS_descriptor();
7238524.142031 | 0) 0.082 us | load_TLS_descriptor();
7238524.142032 | 0) 0.091 us | load_TLS_descriptor();
7238524.142032 | 0) 0.081 us | paravirt_get_lazy_mode();
7238524.142033 | 0) 3.306 us | }
7238524.142033 | 0) | xen_end_context_switch() {
7238524.142033 | 0) 0.697 us | xen_mc_flush();
7238524.142034 | 0) 0.083 us | paravirt_end_context_switch();
7238524.142035 | 0) 1.876 us | }
7238524.269404 | 0) | finish_task_switch() {
7238524.269408 | 0) | xen_evtchn_do_upcall() {
7238524.269408 | 0) | irq_enter() {
7238524.269408 | 0) 0.132 us | rcu_irq_enter();
7238524.269409 | 0) 0.948 us | }
7238524.269409 | 0) 0.063 us | exit_idle();
7238524.269410 | 0) | __xen_evtchn_do_upcall() {
7238524.269410 | 0) | evtchn_2l_handle_events() {
7238524.269410 | 0) 0.057 us | irq_from_virq();
7238524.269411 | 0) | evtchn_from_irq() {
7238524.269411 | 0) | irq_get_irq_data() {
7238524.269411 | 0) 0.058 us | irq_to_desc();
7238524.269412 | 0) 0.579 us | }
7238524.269412 | 0) 0.898 us | }
7238524.269412 | 0) | get_evtchn_to_irq() {
7238524.269412 | 0) 0.049 us | evtchn_2l_max_channels();
7238524.269412 | 0) 0.390 us | }
7238524.269413 | 0) | generic_handle_irq() {
7238524.269413 | 0) 0.051 us | irq_to_desc();
7238524.269413 | 0) | handle_percpu_irq() {
7238524.269413 | 0) | ack_dynirq() {
7238524.269413 | 0) | evtchn_from_irq() {
7238524.269414 | 0) | irq_get_irq_data() {
7238524.269414 | 0) 0.057 us | irq_to_desc();
7238524.269414 | 0) 0.446 us | }
7238524.269414 | 0) 0.754 us | }
7238524.269414 | 0) 0.057 us | irq_move_irq();
7238524.269415 | 0) 0.057 us | evtchn_2l_clear_pending();
7238524.269415 | 0) 1.718 us | }
7238524.269415 | 0) | handle_irq_event_percpu() {
7238524.269416 | 0) | xen_irq_work_interrupt() {
7238524.269416 | 0) | irq_enter() {
7238524.269416 | 0) 0.059 us | rcu_irq_enter();
7238524.269416 | 0) 0.380 us | }
7238524.269417 | 0) | __wake_up() {
7238524.269417 | 0) 0.051 us | _raw_spin_lock_irqsave();
7238524.269417 | 0) | __wake_up_common() {
7238524.269417 | 0) | autoremove_wake_function() {
7238524.269418 | 0) | default_wake_function() {
7238524.269418 | 0) | try_to_wake_up() {
7238524.269418 | 0) 0.230 us | _raw_spin_lock_irqsave();
7238524.269419 | 0) 0.197 us | task_waking_fair();
7238524.269419 | 0) | select_task_rq_fair() {
7238524.269419 | 0) 0.050 us | source_load();
7238524.269420 | 0) 0.057 us | target_load();
7238524.269420 | 0) 0.065 us | idle_cpu();
7238524.269421 | 0) 0.055 us | cpus_share_cache();
7238524.269421 | 0) 0.076 us | idle_cpu();
7238524.269421 | 0) 2.041 us | }
7238524.269422 | 0) 0.050 us | _raw_spin_lock();
7238524.269422 | 0) | ttwu_do_activate.constprop.124() {
7238524.269422 | 0) | activate_task() {
7238524.269422 | 0) | enqueue_task() {
7238524.269422 | 0) 0.175 us | update_rq_clock();
7238524.269423 | 0) | enqueue_task_fair() {
7238524.269423 | 0) | enqueue_entity() {
7238524.269423 | 0) 0.065 us | update_curr();
7238524.269424 | 0) 0.070 us | __compute_runnable_contrib.part.51();
7238524.269424 | 0) 0.052 us | __update_entity_load_avg_contrib();
7238524.269424 | 0) 0.050 us | update_cfs_rq_blocked_load();
7238524.269425 | 0) 0.059 us | account_entity_enqueue();
7238524.269426 | 0) 0.134 us | update_cfs_shares();
7238524.269426 | 0) 0.055 us | place_entity();
7238524.269427 | 0) 0.083 us | __enqueue_entity();
7238524.269427 | 0) 4.026 us | }
7238524.269427 | 0) | enqueue_entity() {
7238524.269428 | 0) 0.065 us | update_curr();
7238524.269428 | 0) 0.051 us | update_cfs_rq_blocked_load();
7238524.269428 | 0) 0.058 us | account_entity_enqueue();
7238524.269429 | 0) 0.082 us | update_cfs_shares();
7238524.269429 | 0) 0.105 us | place_entity();
7238524.269429 | 0) 0.049 us | __enqueue_entity();
7238524.269430 | 0) 2.247 us | }
7238524.269430 | 0) 0.050 us | hrtick_update();
7238524.269430 | 0) 7.310 us | }
7238524.269430 | 0) 8.101 us | }
7238524.269431 | 0) 8.449 us | }
7238524.269431 | 0) | ttwu_do_wakeup() {
7238524.269431 | 0) | check_preempt_curr() {
7238524.269431 | 0) | resched_task() {
7238524.269431 | 0) | xen_smp_send_reschedule() {
7238524.269432 | 0) | xen_send_IPI_one() {
7238524.269432 | 0) | notify_remote_via_irq() {
7238524.269432 | 0) | evtchn_from_irq() {
7238524.269432 | 0) | irq_get_irq_data() {
7238524.269432 | 0) 0.051 us | irq_to_desc();
7238524.269433 | 0) 0.493 us | }
7238524.269433 | 0) 0.857 us | }
7238524.269434 | 0) 1.909 us | } /* notify_remote_via_irq */
7238524.269434 | 0) 2.288 us | }
7238524.269434 | 0) 2.655 us | }
7238524.269434 | 0) 3.127 us | }
7238524.269435 | 0) 3.590 us | }
7238524.269435 | 0) 4.506 us | }
7238524.269436 | 0) 13.594 us | }
7238524.269436 | 0) 0.070 us | _raw_spin_unlock();
7238524.269436 | 0) 0.163 us | ttwu_stat();
7238524.269437 | 0) 0.080 us | _raw_spin_unlock_irqrestore();
7238524.269438 | 0) 19.508 us | }
7238524.269438 | 0) 19.991 us | }
7238524.269438 | 0) 20.486 us | }
7238524.269438 | 0) 21.024 us | }
7238524.269438 | 0) 0.076 us | _raw_spin_unlock_irqrestore();
7238524.269439 | 0) 22.247 us | }
7238524.269439 | 0) | irq_exit() {
7238524.269439 | 0) 0.101 us | idle_cpu();
7238524.269440 | 0) 0.099 us | rcu_irq_exit();
7238524.269441 | 0) 1.207 us | }
7238524.269441 | 0) 25.035 us | }
7238524.269441 | 0) 0.131 us | add_interrupt_randomness();
7238524.269442 | 0) 0.076 us | note_interrupt();
7238524.269442 | 0) 26.909 us | }
7238524.269443 | 0) 29.377 us | }
7238524.269443 | 0) 30.139 us | }
7238524.269443 | 0) 32.759 us | }
7238524.269443 | 0) 33.204 us | }
7238524.269444 | 0) | irq_exit() {
7238524.269444 | 0) | __do_softirq() {
7238524.269444 | 0) 0.068 us | msecs_to_jiffies();
7238524.269445 | 0) | rcu_process_callbacks() {
7238524.269445 | 0) 0.070 us | note_gp_changes();
7238524.269445 | 0) 0.064 us | _raw_spin_lock_irqsave();
7238524.269446 | 0) 0.135 us | rcu_accelerate_cbs();
7238524.269447 | 0) | rcu_report_qs_rnp() {
7238524.269447 | 0) 0.061 us | _raw_spin_unlock_irqrestore();
7238524.269448 | 0) 0.779 us | }
7238524.269448 | 0) 0.081 us | cpu_needs_another_gp();
7238524.269449 | 0) | file_free_rcu() {
7238524.269449 | 0) 0.291 us | kmem_cache_free();
7238524.269450 | 0) 1.139 us | }
7238524.269451 | 0) | put_cred_rcu() {
7238524.269451 | 0) | security_cred_free() {
7238524.269452 | 0) | apparmor_cred_free() {
7238524.269453 | 0) | aa_free_task_context() {
7238524.269453 | 0) | kzfree() {
7238524.269454 | 0) 0.380 us | ksize();
7238524.269455 | 0) 0.147 us | kfree();
7238524.269455 | 0) 1.602 us | }
7238524.269455 | 0) 2.631 us | }
7238524.269456 | 0) 3.611 us | } /* apparmor_cred_free */
7238524.269456 | 0) 4.927 us | }
7238524.269457 | 0) 0.071 us | key_put();
7238524.269457 | 0) 0.071 us | key_put();
7238524.269458 | 0) 0.065 us | key_put();
7238524.269458 | 0) 0.066 us | key_put();
7238524.269459 | 0) 0.390 us | free_uid();
7238524.269460 | 0) 0.178 us | kmem_cache_free();
7238524.269460 | 0) 9.429 us | }
7238524.269461 | 0) 0.099 us | note_gp_changes();
7238524.269461 | 0) 0.080 us | cpu_needs_another_gp();
7238524.269462 | 0) 16.796 us | }
7238524.269462 | 0) 0.068 us | rcu_bh_qs();
7238524.269462 | 0) 0.066 us | __local_bh_enable();
7238524.269463 | 0) 18.770 us | }
7238524.269463 | 0) 0.073 us | idle_cpu();
7238524.269464 | 0) 0.088 us | rcu_irq_exit();
7238524.269464 | 0) 20.487 us | }
7238524.269465 | 0) 56.365 us | }
7238524.269465 | 0) 58.028 us | }
7238524.269466 | 0) 127463.5 us | }
7238524.269466 | 0) 127464.2 us | }
7238524.269467 | 0) 127465.0 us | }
7238524.269467 | 0) 0.095 us | down_read();
7238524.269468 | 0) | copy_from_read_buf() {
7238524.269469 | 0) | tty_audit_add_data() {
7238524.269469 | 0) 0.228 us | _raw_spin_lock_irqsave();
7238524.269470 | 0) 0.070 us | _raw_spin_unlock_irqrestore();
7238524.269471 | 0) 0.074 us | _raw_spin_lock_irqsave();
7238524.269471 | 0) 0.079 us | _raw_spin_unlock_irqrestore();
7238524.269472 | 0) 2.616 us | }
7238524.269472 | 0) 3.878 us | }
7238524.269473 | 0) 0.104 us | copy_from_read_buf();
7238524.269473 | 0) 0.074 us | n_tty_set_room();
7238524.269474 | 0) 0.067 us | n_tty_write_wakeup();
7238524.269474 | 0) | __wake_up() {
7238524.269475 | 0) 0.077 us | _raw_spin_lock_irqsave();
7238524.269475 | 0) | __wake_up_common() {
7238524.269476 | 0) 0.095 us | pollwake();
7238524.269476 | 0) 0.694 us | }
7238524.269476 | 0) 0.064 us | _raw_spin_unlock_irqrestore();
7238524.269477 | 0) 2.128 us | }
7238524.269477 | 0) 0.062 us | n_tty_set_room();
7238524.269477 | 0) 0.066 us | up_read();
7238524.269478 | 0) | remove_wait_queue() {
7238524.269478 | 0) 0.080 us | _raw_spin_lock_irqsave();
7238524.269479 | 0) 0.081 us | _raw_spin_unlock_irqrestore();
7238524.269480 | 0) 1.225 us | }
7238524.269480 | 0) 0.152 us | mutex_unlock();
7238524.269480 | 0) 127485.3 us | }
7238524.269481 | 0) | tty_ldisc_deref() {
7238524.269481 | 0) 0.081 us | ldsem_up_read();
7238524.269482 | 0) 0.655 us | }
7238524.269482 | 0) 0.089 us | get_seconds();
7238524.269483 | 0) 127490.1 us | }
7238524.269484 | 0) 0.287 us | __fsnotify_parent();
7238524.269484 | 0) 0.183 us | fsnotify();
7238524.269485 | 0) 127496.2 us | }
7238524.269559 | 0) | vfs_read() {
7238524.269559 | 0) | rw_verify_area() {
7238524.269560 | 0) | security_file_permission() {
7238524.269560 | 0) | apparmor_file_permission() {
7238524.269561 | 0) 0.164 us | common_file_perm();
7238524.269561 | 0) 0.831 us | }
7238524.269562 | 0) 0.078 us | __fsnotify_parent();
7238524.269562 | 0) 0.080 us | fsnotify();
7238524.269563 | 0) 2.765 us | }
7238524.269563 | 0) 3.490 us | }
7238524.269564 | 0) | tty_read() {
7238524.269564 | 0) 0.066 us | tty_paranoia_check();
7238524.269564 | 0) | tty_ldisc_ref_wait() {
7238524.269565 | 0) 0.085 us | ldsem_down_read();
7238524.269565 | 0) 0.656 us | }
7238524.269566 | 0) | n_tty_read() {
7238524.269566 | 0) 0.078 us | _raw_spin_lock_irq();
7238524.269567 | 0) 0.118 us | mutex_lock_interruptible();
7238524.269567 | 0) 0.078 us | down_read();
7238524.269568 | 0) | add_wait_queue() {
7238524.269568 | 0) 0.089 us | _raw_spin_lock_irqsave();
7238524.269569 | 0) 0.082 us | _raw_spin_unlock_irqrestore();
7238524.269569 | 0) 1.164 us | }
7238524.269570 | 0) 0.073 us | tty_hung_up_p();
7238524.269570 | 0) 0.076 us | n_tty_set_room();
7238524.269571 | 0) 0.078 us | up_read();
7238524.269571 | 0) | schedule_timeout() {
7238524.269572 | 0) | schedule() {
7238524.269572 | 0) | __schedule() {
7238524.269572 | 0) 0.078 us | rcu_note_context_switch();
7238524.269573 | 0) 0.085 us | _raw_spin_lock_irq();
7238524.269574 | 0) | deactivate_task() {
7238524.269574 | 0) | dequeue_task() {
7238524.269574 | 0) 0.185 us | update_rq_clock();
7238524.269575 | 0) | dequeue_task_fair() {
7238524.269575 | 0) | dequeue_entity() {
7238524.269575 | 0) | update_curr() {
7238524.269576 | 0) 0.206 us | cpuacct_charge();
7238524.269577 | 0) 0.937 us | }
7238524.269577 | 0) 0.084 us | __update_entity_load_avg_contrib();
7238524.269577 | 0) 0.077 us | update_cfs_rq_blocked_load();
7238524.269578 | 0) 0.075 us | clear_buddies();
7238524.269579 | 0) 0.096 us | account_entity_dequeue();
7238524.269579 | 0) | update_cfs_shares() {
7238524.269580 | 0) 0.095 us | update_curr();
7238524.269580 | 0) 0.104 us | account_entity_dequeue();
7238524.269581 | 0) 0.076 us | account_entity_enqueue();
7238524.269581 | 0) 1.898 us | }
7238524.269582 | 0) 6.120 us | }
7238524.269582 | 0) | dequeue_entity() {
7238524.269582 | 0) 0.093 us | update_curr();
7238524.269583 | 0) 0.116 us | __update_entity_load_avg_contrib();
7238524.269583 | 0) 0.085 us | update_cfs_rq_blocked_load();
7238524.269584 | 0) 0.067 us | clear_buddies();
7238524.269585 | 0) 0.082 us | account_entity_dequeue();
7238524.269585 | 0) 0.097 us | update_cfs_shares();
7238524.269586 | 0) 3.833 us | }
7238524.269586 | 0) 0.070 us | hrtick_update();
7238524.269587 | 0) 11.677 us | }
7238524.269587 | 0) 13.001 us | }
7238524.269587 | 0) 13.516 us | }
7238524.269588 | 0) | pick_next_task_fair() {
7238524.269588 | 0) 0.072 us | check_cfs_rq_runtime();
7238524.269588 | 0) | pick_next_entity() {
7238524.269589 | 0) 0.080 us | clear_buddies();
7238524.269589 | 0) 0.675 us | }
7238524.269590 | 0) | put_prev_entity() {
7238524.269590 | 0) 0.071 us | check_cfs_rq_runtime();
7238524.269591 | 0) 0.543 us | }
7238524.269591 | 0) | put_prev_entity() {
7238524.269591 | 0) 0.066 us | check_cfs_rq_runtime();
7238524.269592 | 0) 0.658 us | }
7238524.269592 | 0) | set_next_entity() {
7238524.269593 | 0) 0.082 us | update_stats_wait_end();
7238524.269593 | 0) 0.844 us | }
7238524.269594 | 0) 5.970 us | }
7238524.269594 | 0) 0.076 us | paravirt_start_context_switch();
7238524.269595 | 0) 0.074 us | xen_read_cr0();
7238524.269596 | 0) | xen_write_cr0() {
7238524.269597 | 0) 0.081 us | paravirt_get_lazy_mode();
7238524.269597 | 0) 0.086 us | __xen_mc_entry();
7238524.269598 | 0) 0.070 us | paravirt_get_lazy_mode();
7238524.269598 | 0) 1.739 us | }
7238524.269598 | 0) | xen_load_sp0() {
7238524.269599 | 0) 0.078 us | paravirt_get_lazy_mode();
7238524.269599 | 0) 0.078 us | __xen_mc_entry();
7238524.269600 | 0) 0.069 us | paravirt_get_lazy_mode();
7238524.269600 | 0) 1.568 us | }
7238524.269601 | 0) | xen_load_tls() {
7238524.269601 | 0) 0.068 us | paravirt_get_lazy_mode();
7238524.269601 | 0) 0.068 us | paravirt_get_lazy_mode();
7238524.269602 | 0) 0.078 us | load_TLS_descriptor();
7238524.269602 | 0) 0.071 us | load_TLS_descriptor();
7238524.269603 | 0) 0.073 us | load_TLS_descriptor();
7238524.269603 | 0) 0.063 us | paravirt_get_lazy_mode();
7238524.269604 | 0) 3.025 us | }
7238524.269604 | 0) | xen_end_context_switch() {
7238524.269604 | 0) 0.646 us | xen_mc_flush();
7238524.269605 | 0) 0.087 us | paravirt_end_context_switch();
7238524.269606 | 0) 1.604 us | }
^C
Ending tracing...
If you read through the durations carefully, you can see that the shell begins
by completing a 19 second read (time between commands), then has a series of
100 to 200 ms reads (inter-keystroke latency).
The function times printed are inclusive of their children.
The -C option will print on-CPU times only, excluding sleeping or blocking
events from the function duration times. Eg:
# ./funcgraph -Ctp 25285 vfs_read
Tracing "vfs_read" for PID 25285... Ctrl-C to end.
7338520.591816 | 0) | finish_task_switch() {
7338520.591820 | 0) | xen_evtchn_do_upcall() {
7338520.591821 | 0) | irq_enter() {
7338520.591821 | 0) 0.134 us | rcu_irq_enter();
7338520.591822 | 0) 0.823 us | }
7338520.591822 | 0) 0.055 us | exit_idle();
7338520.591822 | 0) | __xen_evtchn_do_upcall() {
7338520.591823 | 0) | evtchn_2l_handle_events() {
7338520.591823 | 0) 0.051 us | irq_from_virq();
7338520.591823 | 0) | evtchn_from_irq() {
7338520.591823 | 0) | irq_get_irq_data() {
7338520.591824 | 0) 0.064 us | irq_to_desc();
7338520.591824 | 0) 0.572 us | }
7338520.591824 | 0) 0.973 us | }
7338520.591825 | 0) | get_evtchn_to_irq() {
7338520.591825 | 0) 0.049 us | evtchn_2l_max_channels();
7338520.591825 | 0) 0.386 us | }
7338520.591825 | 0) | generic_handle_irq() {
7338520.591825 | 0) 0.061 us | irq_to_desc();
7338520.591826 | 0) | handle_percpu_irq() {
7338520.591826 | 0) | ack_dynirq() {
7338520.591826 | 0) | evtchn_from_irq() {
7338520.591826 | 0) | irq_get_irq_data() {
7338520.591827 | 0) 0.050 us | irq_to_desc();
7338520.591827 | 0) 0.441 us | }
7338520.591827 | 0) 0.748 us | }
7338520.591827 | 0) 0.048 us | irq_move_irq();
7338520.591828 | 0) 0.053 us | evtchn_2l_clear_pending();
7338520.591828 | 0) 1.810 us | }
7338520.591828 | 0) | handle_irq_event_percpu() {
7338520.591828 | 0) | xen_irq_work_interrupt() {
7338520.591829 | 0) | irq_enter() {
7338520.591829 | 0) 0.069 us | rcu_irq_enter();
7338520.591829 | 0) 0.386 us | }
7338520.591830 | 0) | __wake_up() {
7338520.591830 | 0) 0.060 us | _raw_spin_lock_irqsave();
7338520.591830 | 0) | __wake_up_common() {
7338520.591830 | 0) | autoremove_wake_function() {
7338520.591831 | 0) | default_wake_function() {
7338520.591831 | 0) | try_to_wake_up() {
7338520.591831 | 0) 0.223 us | _raw_spin_lock_irqsave();
7338520.591832 | 0) 0.243 us | task_waking_fair();
7338520.591832 | 0) | select_task_rq_fair() {
7338520.591833 | 0) 0.063 us | source_load();
7338520.591833 | 0) 0.059 us | target_load();
7338520.591834 | 0) 0.060 us | idle_cpu();
7338520.591834 | 0) 0.059 us | cpus_share_cache();
7338520.591834 | 0) 0.085 us | idle_cpu();
7338520.591835 | 0) 2.176 us | }
7338520.591835 | 0) 0.050 us | _raw_spin_lock();
7338520.591835 | 0) | ttwu_do_activate.constprop.124() {
7338520.591835 | 0) | activate_task() {
7338520.591836 | 0) | enqueue_task() {
7338520.591836 | 0) 0.197 us | update_rq_clock();
7338520.591836 | 0) | enqueue_task_fair() {
7338520.591836 | 0) | enqueue_entity() {
7338520.591837 | 0) 0.118 us | update_curr();
7338520.591837 | 0) 0.060 us | __compute_runnable_contrib.part.51();
7338520.591838 | 0) 0.052 us | __update_entity_load_avg_contrib();
7338520.591838 | 0) 0.132 us | update_cfs_rq_blocked_load();
7338520.591838 | 0) 0.068 us | account_entity_enqueue();
7338520.591839 | 0) 0.327 us | update_cfs_shares();
7338520.591839 | 0) 0.055 us | place_entity();
7338520.591840 | 0) 0.086 us | __enqueue_entity();
7338520.591840 | 0) 0.069 us | update_cfs_rq_blocked_load();
7338520.591840 | 0) 3.870 us | }
7338520.591841 | 0) | enqueue_entity() {
7338520.591841 | 0) 0.050 us | update_curr();
7338520.591841 | 0) 0.048 us | __compute_runnable_contrib.part.51();
7338520.591842 | 0) 0.079 us | __update_entity_load_avg_contrib();
7338520.591842 | 0) 0.068 us | update_cfs_rq_blocked_load();
7338520.591842 | 0) 0.072 us | account_entity_enqueue();
7338520.591843 | 0) 0.068 us | update_cfs_shares();
7338520.591844 | 0) 0.123 us | place_entity();
7338520.591844 | 0) 0.051 us | __enqueue_entity();
7338520.591845 | 0) 3.919 us | }
7338520.591845 | 0) 0.059 us | hrtick_update();
7338520.591845 | 0) 8.895 us | }
7338520.591846 | 0) 9.770 us | }
7338520.591846 | 0) 10.197 us | }
7338520.591846 | 0) | ttwu_do_wakeup() {
7338520.591846 | 0) | check_preempt_curr() {
7338520.591846 | 0) | resched_task() {
7338520.591847 | 0) | xen_smp_send_reschedule() {
7338520.591847 | 0) | xen_send_IPI_one() {
7338520.591847 | 0) | notify_remote_via_irq() {
7338520.591847 | 0) | evtchn_from_irq() {
7338520.591848 | 0) | irq_get_irq_data() {
7338520.591848 | 0) 0.051 us | irq_to_desc();
7338520.591848 | 0) 0.503 us | }
7338520.591848 | 0) 1.031 us | }
7338520.591849 | 0) 2.112 us | }
7338520.591849 | 0) 2.484 us | }
7338520.591850 | 0) 2.851 us | }
7338520.591850 | 0) 3.311 us | }
7338520.591850 | 0) 3.828 us | }
7338520.591851 | 0) 4.788 us | }
7338520.591851 | 0) 15.731 us | }
7338520.591851 | 0) 0.074 us | _raw_spin_unlock();
7338520.591852 | 0) 0.156 us | ttwu_stat();
7338520.591852 | 0) 0.080 us | _raw_spin_unlock_irqrestore();
7338520.591853 | 0) 21.807 us | }
7338520.591853 | 0) 22.286 us | }
7338520.591853 | 0) 22.738 us | }
7338520.591854 | 0) 23.387 us | }
7338520.591854 | 0) 0.105 us | _raw_spin_unlock_irqrestore();
7338520.591854 | 0) 24.698 us | }
7338520.591855 | 0) | irq_exit() {
7338520.591855 | 0) 0.086 us | idle_cpu();
7338520.591856 | 0) 0.105 us | rcu_irq_exit();
7338520.591856 | 0) 1.272 us | }
7338520.591856 | 0) 27.818 us | }
7338520.591857 | 0) 0.140 us | add_interrupt_randomness();
7338520.591857 | 0) 0.084 us | note_interrupt();
7338520.591858 | 0) 29.866 us | }
7338520.591858 | 0) 32.390 us | }
7338520.591859 | 0) 33.204 us | }
7338520.591859 | 0) 36.137 us | }
7338520.591859 | 0) 36.574 us | }
7338520.591859 | 0) | irq_exit() {
7338520.591860 | 0) 0.073 us | idle_cpu();
7338520.591860 | 0) 0.076 us | rcu_irq_exit();
7338520.591861 | 0) 1.091 us | }
7338520.591861 | 0) 40.156 us | }
7338520.591862 | 0) 41.874 us | }
7338520.591862 | 0) 75.633 us | } /* __schedule */
7338520.591862 | 0) 76.182 us | } /* schedule */
7338520.591863 | 0) 76.965 us | } /* schedule_timeout */
7338520.591863 | 0) 0.070 us | down_read();
7338520.591864 | 0) | copy_from_read_buf() {
7338520.591865 | 0) | tty_audit_add_data() {
7338520.591865 | 0) 0.232 us | _raw_spin_lock_irqsave();
7338520.591866 | 0) 0.079 us | _raw_spin_unlock_irqrestore();
7338520.591867 | 0) 0.122 us | _raw_spin_lock_irqsave();
7338520.591867 | 0) 0.066 us | _raw_spin_unlock_irqrestore();
7338520.591868 | 0) 2.642 us | }
7338520.591868 | 0) 3.886 us | }
7338520.591868 | 0) 0.149 us | copy_from_read_buf();
7338520.591869 | 0) 0.072 us | n_tty_set_room();
7338520.591870 | 0) 0.071 us | n_tty_write_wakeup();
7338520.591870 | 0) | __wake_up() {
7338520.591871 | 0) 0.071 us | _raw_spin_lock_irqsave();
7338520.591872 | 0) | __wake_up_common() {
7338520.591872 | 0) 0.097 us | pollwake();
7338520.591873 | 0) 0.739 us | }
7338520.591873 | 0) 0.066 us | _raw_spin_unlock_irqrestore();
7338520.591874 | 0) 3.043 us | }
7338520.591874 | 0) 0.075 us | n_tty_set_room();
7338520.591875 | 0) 0.106 us | up_read();
7338520.591875 | 0) | remove_wait_queue() {
7338520.591875 | 0) 0.078 us | _raw_spin_lock_irqsave();
7338520.591876 | 0) 0.075 us | _raw_spin_unlock_irqrestore();
7338520.591877 | 0) 1.165 us | }
7338520.591877 | 0) 0.137 us | mutex_unlock();
7338520.591877 | 0) 98.321 us | } /* n_tty_read */
7338520.591878 | 0) | tty_ldisc_deref() {
7338520.591878 | 0) 0.072 us | ldsem_up_read();
7338520.591879 | 0) 0.561 us | }
7338520.591879 | 0) 0.090 us | get_seconds();
7338520.591880 | 0) 102.599 us | } /* tty_read */
7338520.591880 | 0) 0.362 us | __fsnotify_parent();
7338520.591881 | 0) 0.171 us | fsnotify();
7338520.591882 | 0) 109.640 us | } /* vfs_read */
7338520.591951 | 0) | vfs_read() {
7338520.591951 | 0) | rw_verify_area() {
7338520.591952 | 0) | security_file_permission() {
7338520.591952 | 0) | apparmor_file_permission() {
7338520.591952 | 0) 0.174 us | common_file_perm();
7338520.591953 | 0) 0.762 us | }
7338520.591953 | 0) 0.126 us | __fsnotify_parent();
7338520.591954 | 0) 0.088 us | fsnotify();
7338520.591954 | 0) 2.609 us | }
7338520.591955 | 0) 3.351 us | }
7338520.591955 | 0) | tty_read() {
7338520.591956 | 0) 0.081 us | tty_paranoia_check();
7338520.591956 | 0) | tty_ldisc_ref_wait() {
7338520.591956 | 0) 0.090 us | ldsem_down_read();
7338520.591957 | 0) 0.633 us | }
7338520.591957 | 0) | n_tty_read() {
7338520.591958 | 0) 0.073 us | _raw_spin_lock_irq();
7338520.591958 | 0) 0.089 us | mutex_lock_interruptible();
7338520.591959 | 0) 0.080 us | down_read();
7338520.591960 | 0) | add_wait_queue() {
7338520.591960 | 0) 0.084 us | _raw_spin_lock_irqsave();
7338520.591960 | 0) 0.087 us | _raw_spin_unlock_irqrestore();
7338520.591961 | 0) 1.215 us | }
7338520.591961 | 0) 0.078 us | tty_hung_up_p();
7338520.591962 | 0) 0.084 us | n_tty_set_room();
7338520.591962 | 0) 0.072 us | up_read();
7338520.591963 | 0) | schedule_timeout() {
7338520.591963 | 0) | schedule() {
7338520.591964 | 0) | __schedule() {
7338520.591964 | 0) 0.084 us | rcu_note_context_switch();
7338520.591965 | 0) 0.086 us | _raw_spin_lock_irq();
7338520.591965 | 0) | deactivate_task() {
7338520.591966 | 0) | dequeue_task() {
7338520.591966 | 0) 0.171 us | update_rq_clock();
7338520.591966 | 0) | dequeue_task_fair() {
7338520.591967 | 0) | dequeue_entity() {
7338520.591967 | 0) | update_curr() {
7338520.591967 | 0) 0.248 us | cpuacct_charge();
7338520.591968 | 0) 0.974 us | }
7338520.591969 | 0) 0.074 us | update_cfs_rq_blocked_load();
7338520.591969 | 0) 0.081 us | clear_buddies();
7338520.591970 | 0) 0.094 us | account_entity_dequeue();
7338520.591971 | 0) | update_cfs_shares() {
7338520.591971 | 0) 0.096 us | update_curr();
7338520.591971 | 0) 0.093 us | account_entity_dequeue();
7338520.591972 | 0) 0.079 us | account_entity_enqueue();
7338520.591972 | 0) 1.743 us | }
7338520.591972 | 0) 5.515 us | }
7338520.591973 | 0) | dequeue_entity() {
7338520.591973 | 0) 0.088 us | update_curr();
7338520.591974 | 0) 0.106 us | update_cfs_rq_blocked_load();
7338520.591975 | 0) 0.078 us | clear_buddies();
7338520.591975 | 0) 0.088 us | account_entity_dequeue();
7338520.591976 | 0) 0.091 us | update_cfs_shares();
7338520.591977 | 0) 3.639 us | }
7338520.591977 | 0) 0.078 us | hrtick_update();
7338520.591978 | 0) 10.851 us | }
7338520.591978 | 0) 11.992 us | }
7338520.591978 | 0) 12.496 us | }
7338520.591978 | 0) | pick_next_task_fair() {
7338520.591979 | 0) 0.079 us | check_cfs_rq_runtime();
7338520.591979 | 0) | pick_next_entity() {
7338520.591979 | 0) 0.080 us | clear_buddies();
7338520.591980 | 0) 0.594 us | }
7338520.591980 | 0) | put_prev_entity() {
7338520.591980 | 0) 0.078 us | check_cfs_rq_runtime();
7338520.591981 | 0) 0.641 us | }
7338520.591981 | 0) | put_prev_entity() {
7338520.591982 | 0) 0.076 us | check_cfs_rq_runtime();
7338520.591982 | 0) 0.610 us | }
7338520.591982 | 0) | set_next_entity() {
7338520.591983 | 0) 0.097 us | update_stats_wait_end();
7338520.591983 | 0) 0.744 us | }
7338520.591984 | 0) 5.115 us | }
7338520.591984 | 0) 0.076 us | paravirt_start_context_switch();
7338520.591985 | 0) 0.086 us | xen_read_cr0();
7338520.591986 | 0) | xen_write_cr0() {
7338520.591986 | 0) 0.078 us | paravirt_get_lazy_mode();
7338520.591987 | 0) 0.086 us | __xen_mc_entry();
7338520.591987 | 0) 0.078 us | paravirt_get_lazy_mode();
7338520.591988 | 0) 1.698 us | }
7338520.591988 | 0) | xen_load_sp0() {
7338520.591988 | 0) 0.074 us | paravirt_get_lazy_mode();
7338520.591989 | 0) 0.084 us | __xen_mc_entry();
7338520.591989 | 0) 0.084 us | paravirt_get_lazy_mode();
7338520.591990 | 0) 1.724 us | }
7338520.591990 | 0) | xen_load_tls() {
7338520.591991 | 0) 0.080 us | paravirt_get_lazy_mode();
7338520.591991 | 0) 0.088 us | paravirt_get_lazy_mode();
7338520.591992 | 0) 0.140 us | load_TLS_descriptor();
7338520.591992 | 0) 0.079 us | load_TLS_descriptor();
7338520.591993 | 0) 0.087 us | load_TLS_descriptor();
7338520.591994 | 0) 0.078 us | paravirt_get_lazy_mode();
7338520.591994 | 0) 3.666 us | }
7338520.591995 | 0) | xen_end_context_switch() {
7338520.591995 | 0) 0.644 us | xen_mc_flush();
7338520.591996 | 0) 0.080 us | paravirt_end_context_switch();
7338520.591997 | 0) 1.813 us | }
7338520.855105 | 0) | finish_task_switch() {
7338520.855110 | 0) | xen_evtchn_do_upcall() {
7338520.855110 | 0) | irq_enter() {
7338520.855110 | 0) 0.137 us | rcu_irq_enter();
7338520.855111 | 0) 0.673 us | }
7338520.855111 | 0) 0.063 us | exit_idle();
7338520.855111 | 0) | __xen_evtchn_do_upcall() {
7338520.855112 | 0) | evtchn_2l_handle_events() {
7338520.855112 | 0) 0.050 us | irq_from_virq();
7338520.855112 | 0) | evtchn_from_irq() {
7338520.855112 | 0) | irq_get_irq_data() {
7338520.855113 | 0) 0.050 us | irq_to_desc();
7338520.855113 | 0) 0.568 us | }
7338520.855113 | 0) 0.895 us | }
7338520.855114 | 0) | get_evtchn_to_irq() {
7338520.855114 | 0) 0.048 us | evtchn_2l_max_channels();
7338520.855114 | 0) 0.386 us | }
7338520.855114 | 0) | generic_handle_irq() {
7338520.855114 | 0) 0.051 us | irq_to_desc();
7338520.855115 | 0) | handle_percpu_irq() {
7338520.855115 | 0) | ack_dynirq() {
7338520.855115 | 0) | evtchn_from_irq() {
7338520.855115 | 0) | irq_get_irq_data() {
7338520.855116 | 0) 0.058 us | irq_to_desc();
7338520.855117 | 0) 1.264 us | }
7338520.855117 | 0) 1.644 us | }
7338520.855117 | 0) 0.048 us | irq_move_irq();
7338520.855118 | 0) 0.050 us | evtchn_2l_clear_pending();
7338520.855118 | 0) 2.876 us | }
7338520.855118 | 0) | handle_irq_event_percpu() {
7338520.855119 | 0) | xen_irq_work_interrupt() {
7338520.855119 | 0) | irq_enter() {
7338520.855119 | 0) 0.055 us | rcu_irq_enter();
7338520.855119 | 0) 0.460 us | }
7338520.855120 | 0) | __wake_up() {
7338520.855120 | 0) 0.057 us | _raw_spin_lock_irqsave();
7338520.855120 | 0) | __wake_up_common() {
7338520.855121 | 0) | autoremove_wake_function() {
7338520.855121 | 0) | default_wake_function() {
7338520.855121 | 0)
gitextract_cv8l2gg4/
├── LICENSE
├── README.md
├── deprecated/
│ ├── README.md
│ ├── execsnoop-proc
│ ├── execsnoop-proc.8
│ └── execsnoop-proc_example.txt
├── disk/
│ └── bitesize
├── examples/
│ ├── bitesize_example.txt
│ ├── cachestat_example.txt
│ ├── execsnoop_example.txt
│ ├── funccount_example.txt
│ ├── funcgraph_example.txt
│ ├── funcslower_example.txt
│ ├── functrace_example.txt
│ ├── iolatency_example.txt
│ ├── iosnoop_example.txt
│ ├── killsnoop_example.txt
│ ├── kprobe_example.txt
│ ├── opensnoop_example.txt
│ ├── perf-stat-hist_example.txt
│ ├── reset-ftrace_example.txt
│ ├── syscount_example.txt
│ ├── tcpretrans_example.txt
│ ├── tpoint_example.txt
│ └── uprobe_example.txt
├── execsnoop
├── fs/
│ └── cachestat
├── iolatency
├── iosnoop
├── kernel/
│ ├── funccount
│ ├── funcgraph
│ ├── funcslower
│ ├── functrace
│ └── kprobe
├── killsnoop
├── man/
│ └── man8/
│ ├── bitesize.8
│ ├── cachestat.8
│ ├── execsnoop.8
│ ├── funccount.8
│ ├── funcgraph.8
│ ├── funcslower.8
│ ├── functrace.8
│ ├── iolatency.8
│ ├── iosnoop.8
│ ├── killsnoop.8
│ ├── kprobe.8
│ ├── opensnoop.8
│ ├── perf-stat-hist.8
│ ├── reset-ftrace.8
│ ├── syscount.8
│ ├── tcpretrans.8
│ ├── tpoint.8
│ └── uprobe.8
├── misc/
│ └── perf-stat-hist
├── net/
│ └── tcpretrans
├── opensnoop
├── syscount
├── system/
│ └── tpoint
├── tools/
│ └── reset-ftrace
└── user/
└── uprobe
Condensed preview — 60 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (557K chars).
[
{
"path": "LICENSE",
"chars": 18025,
"preview": "GNU GENERAL PUBLIC LICENSE\n Version 2, June 1991\n\n Copyright (C) 1989, 1991 Free Software Foundati"
},
{
"path": "README.md",
"chars": 12192,
"preview": "perf-tools\n==========\n\nA miscellaneous collection of in-development and unsupported performance analysis tools for Linux"
},
{
"path": "deprecated/README.md",
"chars": 30,
"preview": "Deprecated versions of tools.\n"
},
{
"path": "deprecated/execsnoop-proc",
"chars": 4625,
"preview": "#!/usr/bin/perl\n#\n# execsnoop - trace process exec() with arguments. /proc version.\n# Written using Linux ft"
},
{
"path": "deprecated/execsnoop-proc.8",
"chars": 2618,
"preview": ".TH execsnoop\\-proc 8 \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nexecsnoop\\-proc \\- trace process exec() with arguments. Use"
},
{
"path": "deprecated/execsnoop-proc_example.txt",
"chars": 1564,
"preview": "Demonstrations of execsnoop-proc, the Linux ftrace version.\n\nHere's execsnoop showing what's really executed by \"man ls\""
},
{
"path": "disk/bitesize",
"chars": 4974,
"preview": "#!/bin/bash\n#\n# bitesize - show disk I/O size as a histogram.\n# Written using Linux perf_events (aka \"perf\")."
},
{
"path": "examples/bitesize_example.txt",
"chars": 2495,
"preview": "Demonstrations of bitesize, the Linux perf_events version.\n\n\nbitesize traces block I/O issued, and reports a histogram o"
},
{
"path": "examples/cachestat_example.txt",
"chars": 2594,
"preview": "Demonstrations of cachestat, the Linux ftrace version.\n\n\nHere is some sample output showing file system cache statistics"
},
{
"path": "examples/execsnoop_example.txt",
"chars": 6860,
"preview": "Demonstrations of execsnoop, the Linux ftrace version.\n\n\nHere's execsnoop showing what's really executed by \"man ls\":\n\n#"
},
{
"path": "examples/funccount_example.txt",
"chars": 4452,
"preview": "Demonstrations of funccount, the Linux ftrace version.\n\n\nTracing all kernel functions that start with \"bio_\" (which woul"
},
{
"path": "examples/funcgraph_example.txt",
"chars": 156111,
"preview": "Demonstrations of funcgraph, the Linux ftrace version.\n\n\nI'll start by showing do_nanosleep(), since it's usually a low "
},
{
"path": "examples/funcslower_example.txt",
"chars": 4366,
"preview": "Demonstrations of funcslower, the Linux ftrace version.\n\n\nShow me ext3_readpages() calls slower than 1000 microseconds ("
},
{
"path": "examples/functrace_example.txt",
"chars": 14148,
"preview": "Demonstrations of functrace, the Linux ftrace version.\n\n\nA (usually) good example to start with is do_nanosleep(), since"
},
{
"path": "examples/iolatency_example.txt",
"chars": 18318,
"preview": "Demonstrations of iolatency, the Linux ftrace version.\n\n\nHere's a busy system doing over 4k disk IOPS:\n\n# ./iolatency\nTr"
},
{
"path": "examples/iosnoop_example.txt",
"chars": 15331,
"preview": "Demonstrations of iosnoop, the Linux ftrace version.\n\n\nHere's Linux 3.16, tracing tar archiving a filesystem:\n\n# ./iosno"
},
{
"path": "examples/killsnoop_example.txt",
"chars": 2482,
"preview": "Demonstrations of killsnoop, the Linux ftrace version.\n\n\nWhat signals are happening on my system?\n\n# ./killsnoop \nTracin"
},
{
"path": "examples/kprobe_example.txt",
"chars": 21909,
"preview": "Demonstrations of kprobe, the Linux ftrace version.\n\n\nThis traces the kernel do_sys_open() function, when it is called:\n"
},
{
"path": "examples/opensnoop_example.txt",
"chars": 2015,
"preview": "Demonstrations of opensnoop, the Linux ftrace version.\n\n\n# ./opensnoop\nTracing open()s. Ctrl-C to end.\nCOMM "
},
{
"path": "examples/perf-stat-hist_example.txt",
"chars": 7412,
"preview": "Demonstrations of perf-stat-hist, the Linux perf_events version.\n\n\nTracing the net:net_dev_xmit tracepoint, and building"
},
{
"path": "examples/reset-ftrace_example.txt",
"chars": 2631,
"preview": "Demonstrations of reset-ftrace, the Linux ftrace tool.\n\n\nYou will probably never need this tool. If you kill -9 an ftrac"
},
{
"path": "examples/syscount_example.txt",
"chars": 9259,
"preview": "Demonstrations of syscount, the Linux perf_events version.\n\n\nThe first mode I use is \"-c\", where it behaves like \"strace"
},
{
"path": "examples/tcpretrans_example.txt",
"chars": 3519,
"preview": "Demonstrations of tcpretrans, the Linux ftrace version.\n\n\nTracing TCP retransmits on a busy server:\n\n# ./tcpretrans \nTIM"
},
{
"path": "examples/tpoint_example.txt",
"chars": 7949,
"preview": "Demonstrations of tpoint, the Linux ftrace version.\n\n\nLet's trace block:block_rq_issue, to see block device (disk) I/O r"
},
{
"path": "examples/uprobe_example.txt",
"chars": 14838,
"preview": "Demonstrations of uprobe, the Linux ftrace version.\n\nTrace the readline() function from all processes named \"bash\":\n\n# ."
},
{
"path": "execsnoop",
"chars": 8689,
"preview": "#!/bin/bash\n#\n# execsnoop - trace process exec() with arguments.\n# Written using Linux ftrace.\n#\n# This show"
},
{
"path": "fs/cachestat",
"chars": 5435,
"preview": "#!/bin/bash\n#\n# cachestat - show Linux page cache hit/miss statistics.\n# Uses Linux ftrace.\n#\n# This is a pr"
},
{
"path": "iolatency",
"chars": 7683,
"preview": "#!/bin/bash\n#\n# iolatency - summarize block device I/O latency as a histogram.\n# Written using Linux ftrace."
},
{
"path": "iosnoop",
"chars": 9112,
"preview": "#!/bin/bash\n#\n# iosnoop - trace block device I/O.\n# Written using Linux ftrace.\n#\n# This traces disk I/O at th"
},
{
"path": "kernel/funccount",
"chars": 4444,
"preview": "#!/bin/bash\n#\n# funccount - count kernel function calls matching specified wildcards.\n# Uses Linux ftrace.\n#"
},
{
"path": "kernel/funcgraph",
"chars": 9067,
"preview": "#!/bin/bash\n#\n# funcgraph - trace kernel function graph, showing child function calls.\n# Uses Linux ftrace.\n"
},
{
"path": "kernel/funcslower",
"chars": 7579,
"preview": "#!/bin/bash\n#\n# funcslower - trace kernel functions slower than a threshold (microseconds).\n# Uses Linux ft"
},
{
"path": "kernel/functrace",
"chars": 6245,
"preview": "#!/bin/bash\n#\n# functrace - trace kernel function calls matching specified wildcards.\n# Uses Linux ftrace.\n#"
},
{
"path": "kernel/kprobe",
"chars": 8041,
"preview": "#!/bin/bash\n#\n# kprobe - trace a given kprobe definition. Kernel dynamic tracing.\n# Written using Linux ftrace."
},
{
"path": "killsnoop",
"chars": 8051,
"preview": "#!/bin/bash\n#\n# killsnoop - trace kill() syscalls with signal/process details.\n# Written using Linux ftrace."
},
{
"path": "man/man8/bitesize.8",
"chars": 2457,
"preview": ".TH bitesize 8 \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nbitesize \\- show disk I/O size as a histogram. Uses Linux perf_eve"
},
{
"path": "man/man8/cachestat.8",
"chars": 3839,
"preview": ".TH cachestat 8 \"2014-12-28\" \"USER COMMANDS\"\n.SH NAME\ncachestat \\- Measure page cache hits/misses. Uses Linux ftrace.\n."
},
{
"path": "man/man8/execsnoop.8",
"chars": 3497,
"preview": ".TH execsnoop 8 \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nexecsnoop \\- trace process exec() with arguments. Uses Linux ftra"
},
{
"path": "man/man8/funccount.8",
"chars": 2094,
"preview": ".TH funccount 8 \"2014-07-19\" \"USER COMMANDS\"\n.SH NAME\nfunccount \\- count kernel function calls matching specified wildc"
},
{
"path": "man/man8/funcgraph.8",
"chars": 6010,
"preview": ".TH funcgraph 8 \"2014-07-29\" \"USER COMMANDS\"\n.SH NAME\nfuncgraph \\- trace kernel function graph, showing child function "
},
{
"path": "man/man8/funcslower.8",
"chars": 4190,
"preview": ".TH funcslower 8 \"2014-07-30\" \"USER COMMANDS\"\n.SH NAME\nfuncslower \\- trace kernel functions slower than a threshold (mi"
},
{
"path": "man/man8/functrace.8",
"chars": 3922,
"preview": ".TH functrace 8 \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nfunctrace \\- trace kernel function calls matching specified wildc"
},
{
"path": "man/man8/iolatency.8",
"chars": 3656,
"preview": ".TH iolatency 8 \"2014-07-12\" \"USER COMMANDS\"\n.SH NAME\niolatency \\- summarize block device I/O latency as a histogram. U"
},
{
"path": "man/man8/iosnoop.8",
"chars": 5856,
"preview": ".TH iosnoop 8 \"2014-07-12\" \"USER COMMANDS\"\n.SH NAME\niosnoop \\- trace block I/O events as they occur. Uses Linux ftrace."
},
{
"path": "man/man8/killsnoop.8",
"chars": 2944,
"preview": ".TH killsnoop 8 \"2014-09-15\" \"USER COMMANDS\"\n.SH NAME\nkillsnoop \\- trace kill() syscalls with process and signal detail"
},
{
"path": "man/man8/kprobe.8",
"chars": 5394,
"preview": ".TH kprobe 8 \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nkprobe \\- trace a given kprobe definition. Kernel dynamic tracing. U"
},
{
"path": "man/man8/opensnoop.8",
"chars": 3194,
"preview": ".TH opensnoop 8 \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nopensnoop \\- trace open() syscalls with file details. Uses Linux "
},
{
"path": "man/man8/perf-stat-hist.8",
"chars": 3920,
"preview": ".TH perf-stat-hist 8 \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nperf-stat-hist \\- histogram summary of tracepoint values. Us"
},
{
"path": "man/man8/reset-ftrace.8",
"chars": 1292,
"preview": ".TH reset-ftrace 8 \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nreset-ftrace \\- reset state of ftrace, disabling all tracing. "
},
{
"path": "man/man8/syscount.8",
"chars": 2599,
"preview": ".TH syscount 8 \"2014-07-07\" \"USER COMMANDS\"\n.SH NAME\nsyscount \\- count system calls. Uses Linux perf_events.\n.SH SYNOPS"
},
{
"path": "man/man8/tcpretrans.8",
"chars": 2773,
"preview": ".TH tcpretrans 8 \"2014-07-31\" \"USER COMMANDS\"\n.SH NAME\ntcpretrans \\- show TCP retransmits, with address and other detai"
},
{
"path": "man/man8/tpoint.8",
"chars": 4136,
"preview": ".TH tpoint 8 \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\ntpoint \\- trace a given tracepoint. Static tracing. Uses Linux ftrac"
},
{
"path": "man/man8/uprobe.8",
"chars": 6048,
"preview": ".TH uprobe 8 \"2014-07-20\" \"USER COMMANDS\"\n.SH NAME\nuprobe \\- trace a given uprobe definition. User-level dynamic tracin"
},
{
"path": "misc/perf-stat-hist",
"chars": 6781,
"preview": "#!/bin/bash\n#\n# perf-stat-hist - perf_events stat histogram hack.\n# Written using Linux perf_events (ak"
},
{
"path": "net/tcpretrans",
"chars": 8988,
"preview": "#!/usr/bin/perl\n#\n# tcpretrans - show TCP retransmts, with address and other details.\n# Written using Linux"
},
{
"path": "opensnoop",
"chars": 8708,
"preview": "#!/bin/bash\n#\n# opensnoop - trace open() syscalls with file details.\n# Written using Linux ftrace.\n#\n# This "
},
{
"path": "syscount",
"chars": 6487,
"preview": "#!/bin/bash\n#\n# syscount - count system calls.\n# Written using Linux perf_events (aka \"perf\").\n#\n# This is a "
},
{
"path": "system/tpoint",
"chars": 6363,
"preview": "#!/bin/bash\n#\n# tpoint - trace a given tracepoint. Static tracing.\n# Written using Linux ftrace.\n#\n# This will "
},
{
"path": "tools/reset-ftrace",
"chars": 3253,
"preview": "#!/bin/bash\n#\n# reset-ftrace - reset state of ftrace, disabling all tracing.\n# Written for Linux ftrace.\n"
},
{
"path": "user/uprobe",
"chars": 12191,
"preview": "#!/bin/bash\n#\n# uprobe - trace a given uprobe definition. User-level dynamic tracing.\n# Written using Linux ftr"
}
]
About this extraction
This page contains the full source code of the brendangregg/perf-tools GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 60 files (523.1 KB), approximately 156.4k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.